2024-05-08T21:06:36.395 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-08T21:06:36.395 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} --owner scheduled_pdonnell@teuthology --name pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi --verbose --archive /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7698047 -- /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7698047/orig.config.yaml 2024-05-08T21:06:36.442 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-08T21:06:36.529 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7698047 branch: wip-pdonnell-testing-20240508.150423-reef description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} email: pdonnell@redhat.com first_in_suite: false job_id: '7698047' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20240508.150423-reef ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - bad backtrace - object missing on disk - error reading table object - error reading sessionmap - unmatched fragstat - unmatched rstat - was unreadable, recreating it now - Scrub error on inode - Metadata damage detected - MDS_FAILED - MDS_DAMAGE sha1: d21e4f961318b56f0d915caa2490c3711bf13c34 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d21e4f961318b56f0d915caa2490c3711bf13c34 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240508.150423-reef sha1: d21e4f961318b56f0d915caa2490c3711bf13c34 owner: scheduled_pdonnell@teuthology priority: 65 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 2738 sha1: d21e4f961318b56f0d915caa2490c3711bf13c34 sleep_before_teardown: 0 subset: 17/512 suite: fs suite_branch: wip-pdonnell-testing-20240508.150423-reef suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d21e4f961318b56f0d915caa2490c3711bf13c34/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d21e4f961318b56f0d915caa2490c3711bf13c34 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4gFNQ1ey7Dp5Fx416rLYsi8Dgi1Kok8hXob6AtIr+XKEJVSs/karzoZzdqjR4wbzZbCvO4bFrFgFY6Pr/A5A0= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNCJ9V9wH96GjNq0MhtgUxT+2pzTTqQNcfJr+gi2hpna4lGM81wHR8Gmgv+/DuVpEXBdopANQ7gkmGZ8nL0zhlE= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_recovery_pool teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-08_19:22:21 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-08T21:06:36.529 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d21e4f961318b56f0d915caa2490c3711bf13c34/qa; will attempt to use it 2024-05-08T21:06:36.530 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d21e4f961318b56f0d915caa2490c3711bf13c34/qa/tasks 2024-05-08T21:06:36.530 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-08T21:06:36.532 INFO:teuthology.task.internal:Checking packages... 2024-05-08T21:06:36.554 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'd21e4f961318b56f0d915caa2490c3711bf13c34' 2024-05-08T21:06:36.555 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-08T21:06:36.555 INFO:teuthology.packaging:ref: None 2024-05-08T21:06:36.555 INFO:teuthology.packaging:tag: None 2024-05-08T21:06:36.555 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240508.150423-reef 2024-05-08T21:06:36.555 INFO:teuthology.packaging:sha1: d21e4f961318b56f0d915caa2490c3711bf13c34 2024-05-08T21:06:36.555 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20240508.150423-reef 2024-05-08T21:06:36.784 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1322.gd21e4f96 2024-05-08T21:06:36.785 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-08T21:06:36.791 INFO:teuthology.task.internal:no buildpackages task found 2024-05-08T21:06:36.791 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-08T21:06:36.800 INFO:teuthology.task.internal:Saving configuration 2024-05-08T21:06:36.819 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-08T21:06:36.825 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-08T21:06:36.850 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7698047', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-08 21:01:10.433473', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4gFNQ1ey7Dp5Fx416rLYsi8Dgi1Kok8hXob6AtIr+XKEJVSs/karzoZzdqjR4wbzZbCvO4bFrFgFY6Pr/A5A0='} 2024-05-08T21:06:36.872 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7698047', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-08 21:01:10.431731', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNCJ9V9wH96GjNq0MhtgUxT+2pzTTqQNcfJr+gi2hpna4lGM81wHR8Gmgv+/DuVpEXBdopANQ7gkmGZ8nL0zhlE='} 2024-05-08T21:06:36.872 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-08T21:06:36.879 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-05-08T21:06:36.879 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-05-08T21:06:36.879 INFO:teuthology.run_tasks:Running task console_log... 2024-05-08T21:06:36.948 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbcb6995160>, signals=[15]) 2024-05-08T21:06:36.949 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-08T21:06:36.955 INFO:teuthology.task.internal:Opening connections... 2024-05-08T21:06:36.955 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2024-05-08T21:06:36.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T21:06:37.036 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-05-08T21:06:37.037 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T21:06:37.110 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-08T21:06:37.117 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2024-05-08T21:06:37.139 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2024-05-08T21:06:37.139 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2024-05-08T21:06:37.201 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="Red Hat Enterprise Linux" 2024-05-08T21:06:37.201 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="8.6 (Ootpa)" 2024-05-08T21:06:37.201 INFO:teuthology.orchestra.run.smithi003.stdout:ID="rhel" 2024-05-08T21:06:37.201 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="fedora" 2024-05-08T21:06:37.201 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="8.6" 2024-05-08T21:06:37.201 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el8" 2024-05-08T21:06:37.201 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-05-08T21:06:37.201 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2024-05-08T21:06:37.201 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-05-08T21:06:37.201 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://www.redhat.com/" 2024-05-08T21:06:37.201 INFO:teuthology.orchestra.run.smithi003.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-05-08T21:06:37.201 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-08T21:06:37.201 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:06:37.201 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-05-08T21:06:37.202 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-05-08T21:06:37.202 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-05-08T21:06:37.202 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-05-08T21:06:37.202 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2024-05-08T21:06:37.227 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-05-08T21:06:37.248 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-05-08T21:06:37.248 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-05-08T21:06:37.311 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="Red Hat Enterprise Linux" 2024-05-08T21:06:37.311 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="8.6 (Ootpa)" 2024-05-08T21:06:37.311 INFO:teuthology.orchestra.run.smithi186.stdout:ID="rhel" 2024-05-08T21:06:37.311 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="fedora" 2024-05-08T21:06:37.311 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="8.6" 2024-05-08T21:06:37.311 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el8" 2024-05-08T21:06:37.312 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-05-08T21:06:37.312 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-05-08T21:06:37.312 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-05-08T21:06:37.312 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://www.redhat.com/" 2024-05-08T21:06:37.312 INFO:teuthology.orchestra.run.smithi186.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-05-08T21:06:37.312 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-08T21:06:37.312 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:06:37.312 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-05-08T21:06:37.312 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-05-08T21:06:37.312 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-05-08T21:06:37.312 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-05-08T21:06:37.312 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-05-08T21:06:37.335 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-08T21:06:37.345 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-08T21:06:37.351 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-08T21:06:37.351 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2024-05-08T21:06:37.354 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-05-08T21:06:37.373 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-08T21:06:37.379 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-08T21:06:37.379 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2024-05-08T21:06:37.414 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-05-08T21:06:37.448 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-08T21:06:37.505 INFO:teuthology.run_tasks:Running task kernel... 2024-05-08T21:06:37.519 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-05-08T21:06:37.519 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2024-05-08T21:06:37.519 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-05-08T21:06:37.520 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-08T21:06:37.520 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-08T21:06:37.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T21:06:37.539 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-05-08T21:06:37.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T21:06:37.540 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-05-08T21:06:37.601 INFO:teuthology.orchestra.run.smithi186.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-05-08T21:06:37.601 INFO:teuthology.task.kernel:Running kernel on smithi186: 4.18.0-513.9.1.el8_9.x86_64 2024-05-08T21:06:37.602 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum install -y kernel 2024-05-08T21:06:37.603 INFO:teuthology.orchestra.run.smithi003.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-05-08T21:06:37.603 INFO:teuthology.task.kernel:Running kernel on smithi003: 4.18.0-513.9.1.el8_9.x86_64 2024-05-08T21:06:37.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2024-05-08T21:06:39.138 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2024-05-08T21:06:39.138 INFO:teuthology.orchestra.run.smithi003.stdout:Unable to read consumer identity 2024-05-08T21:06:39.203 INFO:teuthology.orchestra.run.smithi186.stdout:Updating Subscription Management repositories. 2024-05-08T21:06:39.204 INFO:teuthology.orchestra.run.smithi186.stdout:Unable to read consumer identity 2024-05-08T21:06:39.473 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:06:39.473 INFO:teuthology.orchestra.run.smithi003.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-05-08T21:06:39.473 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:06:39.536 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:06:39.555 INFO:teuthology.orchestra.run.smithi186.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-05-08T21:06:39.555 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:06:41.231 INFO:teuthology.orchestra.run.smithi003.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2024-05-08T21:06:41.342 INFO:teuthology.orchestra.run.smithi186.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-05-08T21:06:42.373 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2024-05-08T21:06:43.786 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 16 MB 00:02 2024-05-08T21:06:46.457 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 625 kB/s | 24 kB 00:00 2024-05-08T21:06:47.326 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-05-08T21:06:47.327 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-05-08T21:06:47.411 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-05-08T21:06:47.412 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-05-08T21:06:47.412 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-05-08T21:06:47.501 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 595 kB/s | 24 kB 00:00 2024-05-08T21:06:47.558 DEBUG:teuthology.orchestra.run.smithi003:> echo no | sudo yum reinstall kernel || true 2024-05-08T21:06:47.947 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2024-05-08T21:06:47.948 INFO:teuthology.orchestra.run.smithi003.stdout:Unable to read consumer identity 2024-05-08T21:06:48.110 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:06:48.110 INFO:teuthology.orchestra.run.smithi003.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-05-08T21:06:48.110 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:06:48.303 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-05-08T21:06:48.303 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-05-08T21:06:48.378 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-08T21:06:48.379 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-05-08T21:06:48.379 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-08T21:06:48.467 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:02 ago on Wed 08 May 2024 09:06:46 PM UTC. 2024-05-08T21:06:48.522 DEBUG:teuthology.orchestra.run.smithi186:> echo no | sudo yum reinstall kernel || true 2024-05-08T21:06:48.554 INFO:teuthology.orchestra.run.smithi003.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-05-08T21:06:48.555 INFO:teuthology.orchestra.run.smithi003.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-05-08T21:06:48.594 INFO:teuthology.orchestra.run.smithi003.stderr:Error: No packages marked for reinstall. 2024-05-08T21:06:48.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum reinstall -y kernel || true 2024-05-08T21:06:48.861 INFO:teuthology.orchestra.run.smithi186.stdout:Updating Subscription Management repositories. 2024-05-08T21:06:48.861 INFO:teuthology.orchestra.run.smithi186.stdout:Unable to read consumer identity 2024-05-08T21:06:49.036 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2024-05-08T21:06:49.036 INFO:teuthology.orchestra.run.smithi003.stdout:Unable to read consumer identity 2024-05-08T21:06:49.110 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:06:49.111 INFO:teuthology.orchestra.run.smithi186.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-05-08T21:06:49.111 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:06:49.277 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:06:49.277 INFO:teuthology.orchestra.run.smithi003.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-05-08T21:06:49.278 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:06:49.419 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:02 ago on Wed 08 May 2024 09:06:47 PM UTC. 2024-05-08T21:06:49.492 INFO:teuthology.orchestra.run.smithi186.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-05-08T21:06:49.497 INFO:teuthology.orchestra.run.smithi186.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-05-08T21:06:49.538 INFO:teuthology.orchestra.run.smithi186.stderr:Error: No packages marked for reinstall. 2024-05-08T21:06:49.585 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum reinstall -y kernel || true 2024-05-08T21:06:49.632 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:03 ago on Wed 08 May 2024 09:06:46 PM UTC. 2024-05-08T21:06:49.700 INFO:teuthology.orchestra.run.smithi003.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-05-08T21:06:49.701 INFO:teuthology.orchestra.run.smithi003.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-05-08T21:06:49.730 INFO:teuthology.orchestra.run.smithi003.stderr:Error: No packages marked for reinstall. 2024-05-08T21:06:49.784 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-05-08T21:06:49.924 INFO:teuthology.orchestra.run.smithi186.stdout:Updating Subscription Management repositories. 2024-05-08T21:06:49.925 INFO:teuthology.orchestra.run.smithi186.stdout:Unable to read consumer identity 2024-05-08T21:06:49.957 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-05-08T21:06:49.958 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-05-08T21:06:49.958 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-08T21:06:49.959 DEBUG:teuthology.parallel:result is None 2024-05-08T21:06:50.121 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:06:50.121 INFO:teuthology.orchestra.run.smithi186.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-05-08T21:06:50.121 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:06:50.433 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:03 ago on Wed 08 May 2024 09:06:47 PM UTC. 2024-05-08T21:06:50.501 INFO:teuthology.orchestra.run.smithi186.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-05-08T21:06:50.501 INFO:teuthology.orchestra.run.smithi186.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-05-08T21:06:50.534 INFO:teuthology.orchestra.run.smithi186.stderr:Error: No packages marked for reinstall. 2024-05-08T21:06:50.587 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-05-08T21:06:50.752 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-05-08T21:06:50.752 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-05-08T21:06:50.752 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-08T21:06:50.752 DEBUG:teuthology.parallel:result is None 2024-05-08T21:06:50.752 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-08T21:06:50.759 INFO:teuthology.task.internal:Creating test directory... 2024-05-08T21:06:50.759 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-08T21:06:50.761 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-08T21:06:50.781 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-08T21:06:50.788 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-08T21:06:50.795 INFO:teuthology.task.internal:Creating archive directory... 2024-05-08T21:06:50.795 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-08T21:06:50.824 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-08T21:06:50.857 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-08T21:06:50.863 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-08T21:06:50.863 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-08T21:06:50.895 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-08T21:06:50.929 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T21:06:50.936 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T21:06:50.942 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T21:06:50.957 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T21:06:50.959 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-08T21:06:50.966 INFO:teuthology.task.internal:Configuring sudo... 2024-05-08T21:06:50.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-08T21:06:51.001 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-08T21:06:51.034 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-08T21:06:51.043 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-08T21:06:51.043 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-08T21:06:51.076 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-08T21:06:51.095 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-08T21:06:51.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-08T21:06:51.241 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-08T21:06:51.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-08T21:06:51.379 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:06:51.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-08T21:06:51.453 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-08T21:06:51.498 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-08T21:06:51.550 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-08T21:06:51.638 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-08T21:06:51.679 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:06:51.679 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-08T21:06:51.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2024-05-08T21:06:51.749 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-05-08T21:06:51.801 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-08T21:06:51.834 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-08T21:06:52.189 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-08T21:06:52.196 INFO:teuthology.task.internal:Starting timer... 2024-05-08T21:06:52.196 INFO:teuthology.run_tasks:Running task pcp... 2024-05-08T21:06:52.206 INFO:teuthology.run_tasks:Running task selinux... 2024-05-08T21:06:52.215 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2024-05-08T21:06:52.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2024-05-08T21:06:52.296 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2024-05-08T21:06:52.298 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-05-08T21:06:52.374 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-05-08T21:06:52.376 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-08T21:06:52.376 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2024-05-08T21:06:52.403 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2024-05-08T21:06:52.403 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-05-08T21:06:52.427 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-05-08T21:06:52.428 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2024-05-08T21:06:52.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-08T21:06:52.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T21:06:52.478 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-08T21:06:52.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T21:06:52.507 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-08T21:06:52.507 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2024-05-08T21:06:52.557 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-05-08T21:06:52.584 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-08T21:06:52.597 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-08T21:06:52.614 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-08T21:06:52.615 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-08T21:17:42.504 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-05-08T21:17:42.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-05-08T21:17:42.507 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T21:17:42.589 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-05-08T21:17:42.664 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-05-08T21:17:42.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-08T21:17:42.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T21:17:42.739 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-05-08T21:17:42.813 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-08T21:17:42.814 INFO:teuthology.run_tasks:Running task clock... 2024-05-08T21:17:42.825 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-08T21:17:42.825 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-08T21:17:42.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-08T21:17:42.828 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-08T21:17:42.828 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-08T21:17:42.865 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-08T21:17:42.884 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-08T21:17:42.887 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-08T21:17:42.903 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-08T21:17:42.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2024-05-08T21:17:42.968 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2024-05-08T21:17:42.982 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-05-08T21:17:42.986 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-08T21:17:42.994 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-05-08T21:17:43.003 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-08T21:17:43.010 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-08T21:17:43.024 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-08T21:17:43.091 INFO:teuthology.orchestra.run.smithi003.stderr:bash: ntpq: command not found 2024-05-08T21:17:43.095 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-08T21:17:43.095 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-05-08T21:17:43.095 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T21:17:43.096 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T21:17:43.096 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T21:17:43.096 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T21:17:43.114 INFO:teuthology.orchestra.run.smithi186.stderr:bash: ntpq: command not found 2024-05-08T21:17:43.118 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-08T21:17:43.118 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-05-08T21:17:43.118 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T21:17:43.118 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T21:17:43.118 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T21:17:43.118 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T21:17:43.119 INFO:teuthology.run_tasks:Running task install... 2024-05-08T21:17:43.128 DEBUG:teuthology.task.install:project ceph 2024-05-08T21:17:43.128 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34'}} 2024-05-08T21:17:43.128 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34'} 2024-05-08T21:17:43.128 INFO:teuthology.task.install:Using flavor: default 2024-05-08T21:17:43.134 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-08T21:17:43.134 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-05-08T21:17:43.135 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34', 'tag': None, 'wait_for_package': False} 2024-05-08T21:17:43.135 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d21e4f961318b56f0d915caa2490c3711bf13c34 2024-05-08T21:17:43.137 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34', 'tag': None, 'wait_for_package': False} 2024-05-08T21:17:43.137 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d21e4f961318b56f0d915caa2490c3711bf13c34 2024-05-08T21:17:43.281 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/ 2024-05-08T21:17:43.282 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1322.gd21e4f96 2024-05-08T21:17:43.361 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/ 2024-05-08T21:17:43.361 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1322.gd21e4f96 2024-05-08T21:17:43.684 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-08T21:17:43.684 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:17:43.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-08T21:17:43.719 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-08T21:17:43.719 DEBUG:teuthology.orchestra.run.smithi003:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d21e4f961318b56f0d915caa2490c3711bf13c34/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-08T21:17:43.765 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-08T21:17:43.765 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:17:43.765 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-08T21:17:43.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-08T21:17:43.796 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-08T21:17:43.796 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d21e4f961318b56f0d915caa2490c3711bf13c34/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-08T21:17:43.868 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-08T21:17:43.883 DEBUG:teuthology.orchestra.run.smithi003:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-08T21:17:43.914 INFO:teuthology.orchestra.run.smithi003.stdout:check_obsoletes = 1 2024-05-08T21:17:43.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum clean all 2024-05-08T21:17:43.949 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-08T21:17:44.018 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2024-05-08T21:17:44.019 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2024-05-08T21:17:44.351 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2024-05-08T21:17:44.398 INFO:teuthology.orchestra.run.smithi186.stdout:Updating Subscription Management repositories. 2024-05-08T21:17:45.635 INFO:teuthology.orchestra.run.smithi003.stdout:63 files removed 2024-05-08T21:17:45.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-08T21:17:45.716 INFO:teuthology.orchestra.run.smithi186.stdout:63 files removed 2024-05-08T21:17:45.768 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-08T21:17:46.080 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2024-05-08T21:17:46.106 INFO:teuthology.orchestra.run.smithi186.stdout:Updating Subscription Management repositories. 2024-05-08T21:17:48.396 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 98 kB/s | 80 kB 00:00 2024-05-08T21:17:48.560 INFO:teuthology.orchestra.run.smithi003.stdout:ceph packages for x86_64 89 kB/s | 80 kB 00:00 2024-05-08T21:17:49.267 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 14 kB/s | 11 kB 00:00 2024-05-08T21:17:49.386 INFO:teuthology.orchestra.run.smithi003.stdout:ceph noarch packages 14 kB/s | 11 kB 00:00 2024-05-08T21:17:50.047 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 2.0 kB/s | 1.5 kB 00:00 2024-05-08T21:17:50.128 INFO:teuthology.orchestra.run.smithi003.stdout:ceph source packages 2.1 kB/s | 1.5 kB 00:00 2024-05-08T21:17:50.197 INFO:teuthology.orchestra.run.smithi186.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2024-05-08T21:17:50.287 INFO:teuthology.orchestra.run.smithi003.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-05-08T21:17:51.882 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2024-05-08T21:17:52.331 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 7.7 MB/s | 16 MB 00:02 2024-05-08T21:17:55.940 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 406 kB/s | 24 kB 00:00 2024-05-08T21:17:56.080 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 456 kB/s | 24 kB 00:00 2024-05-08T21:17:57.292 INFO:teuthology.orchestra.run.smithi186.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 48 MB/s | 61 MB 00:01 2024-05-08T21:17:57.665 INFO:teuthology.orchestra.run.smithi003.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 40 MB/s | 61 MB 00:01 2024-05-08T21:18:08.884 INFO:teuthology.orchestra.run.smithi186.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 50 MB/s | 67 MB 00:01 2024-05-08T21:18:10.181 INFO:teuthology.orchestra.run.smithi003.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 13 MB/s | 8.8 MB 00:00 2024-05-08T21:18:13.541 INFO:teuthology.orchestra.run.smithi003.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 45 MB/s | 67 MB 00:01 2024-05-08T21:18:19.509 INFO:teuthology.orchestra.run.smithi186.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 15 MB/s | 8.8 MB 00:00 2024-05-08T21:18:26.549 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-05-08T21:18:26.550 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-05-08T21:18:26.568 INFO:teuthology.orchestra.run.smithi186.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-05-08T21:18:26.569 INFO:teuthology.orchestra.run.smithi186.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-05-08T21:18:26.571 INFO:teuthology.orchestra.run.smithi186.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-05-08T21:18:26.572 INFO:teuthology.orchestra.run.smithi186.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-05-08T21:18:26.575 INFO:teuthology.orchestra.run.smithi186.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-05-08T21:18:26.578 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-05-08T21:18:26.580 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-05-08T21:18:26.794 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-08T21:18:26.809 INFO:teuthology.orchestra.run.smithi186.stdout:=================================================================================================================================== 2024-05-08T21:18:26.809 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-05-08T21:18:26.809 INFO:teuthology.orchestra.run.smithi186.stdout:=================================================================================================================================== 2024-05-08T21:18:26.809 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-08T21:18:26.809 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-15.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-05-08T21:18:26.809 INFO:teuthology.orchestra.run.smithi186.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-05-08T21:18:26.809 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 6.4 k 2024-05-08T21:18:26.809 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 5.5 M 2024-05-08T21:18:26.809 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 860 k 2024-05-08T21:18:26.809 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 166 k 2024-05-08T21:18:26.809 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 1.6 M 2024-05-08T21:18:26.809 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 138 k 2024-05-08T21:18:26.810 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 3.5 M 2024-05-08T21:18:26.810 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 7.4 M 2024-05-08T21:18:26.810 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 49 k 2024-05-08T21:18:26.810 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 13 M 2024-05-08T21:18:26.810 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 47 M 2024-05-08T21:18:26.810 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 263 k 2024-05-08T21:18:26.810 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 170 k 2024-05-08T21:18:26.810 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 261 k 2024-05-08T21:18:26.810 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 18 k 2024-05-08T21:18:26.810 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-05-08T21:18:26.810 INFO:teuthology.orchestra.run.smithi186.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-05-08T21:18:26.810 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-05-08T21:18:26.811 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 31 k 2024-05-08T21:18:26.811 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 782 k 2024-05-08T21:18:26.811 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 139 k 2024-05-08T21:18:26.811 INFO:teuthology.orchestra.run.smithi186.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-05-08T21:18:26.811 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel x86_64 2.32.1-44.el8_9.1 rhel-8-for-x86_64-baseos-rpms 100 k 2024-05-08T21:18:26.811 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-05-08T21:18:26.811 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-05-08T21:18:26.811 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-05-08T21:18:26.811 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-05-08T21:18:26.811 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 197 k 2024-05-08T21:18:26.811 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 370 k 2024-05-08T21:18:26.811 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 365 k 2024-05-08T21:18:26.811 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 107 k 2024-05-08T21:18:26.812 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 95 k 2024-05-08T21:18:26.812 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 3.5 M 2024-05-08T21:18:26.812 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 181 k 2024-05-08T21:18:26.812 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-05-08T21:18:26.812 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2024-05-08T21:18:26.812 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-05-08T21:18:26.812 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-05-08T21:18:26.812 INFO:teuthology.orchestra.run.smithi186.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-05-08T21:18:26.812 INFO:teuthology.orchestra.run.smithi186.stdout: libblkid x86_64 2.32.1-44.el8_9.1 rhel-8-for-x86_64-baseos-rpms 221 k 2024-05-08T21:18:26.812 INFO:teuthology.orchestra.run.smithi186.stdout: libfdisk x86_64 2.32.1-44.el8_9.1 rhel-8-for-x86_64-baseos-rpms 254 k 2024-05-08T21:18:26.812 INFO:teuthology.orchestra.run.smithi186.stdout: libmount x86_64 2.32.1-44.el8_9.1 rhel-8-for-x86_64-baseos-rpms 237 k 2024-05-08T21:18:26.812 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 3.7 M 2024-05-08T21:18:26.812 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 3.6 M 2024-05-08T21:18:26.813 INFO:teuthology.orchestra.run.smithi186.stdout: libsmartcols x86_64 2.32.1-44.el8_9.1 rhel-8-for-x86_64-baseos-rpms 180 k 2024-05-08T21:18:26.813 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid x86_64 2.32.1-44.el8_9.1 rhel-8-for-x86_64-baseos-rpms 99 k 2024-05-08T21:18:26.813 INFO:teuthology.orchestra.run.smithi186.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-05-08T21:18:26.813 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-05-08T21:18:26.813 INFO:teuthology.orchestra.run.smithi186.stdout: util-linux x86_64 2.32.1-44.el8_9.1 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-05-08T21:18:26.813 INFO:teuthology.orchestra.run.smithi186.stdout: util-linux-user x86_64 2.32.1-44.el8_9.1 rhel-8-for-x86_64-baseos-rpms 103 k 2024-05-08T21:18:26.813 INFO:teuthology.orchestra.run.smithi186.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-05-08T21:18:26.813 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-05-08T21:18:26.813 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-05-08T21:18:26.813 INFO:teuthology.orchestra.run.smithi186.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-05-08T21:18:26.813 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 22 M 2024-05-08T21:18:26.813 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 24 k 2024-05-08T21:18:26.813 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 2.3 M 2024-05-08T21:18:26.813 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 245 k 2024-05-08T21:18:26.814 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 4.8 M 2024-05-08T21:18:26.814 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 18 M 2024-05-08T21:18:26.814 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 16 k 2024-05-08T21:18:26.814 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 24 k 2024-05-08T21:18:26.814 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-05-08T21:18:26.814 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-05-08T21:18:26.814 INFO:teuthology.orchestra.run.smithi186.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-05-08T21:18:26.814 INFO:teuthology.orchestra.run.smithi186.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-05-08T21:18:26.814 INFO:teuthology.orchestra.run.smithi186.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-05-08T21:18:26.814 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 142 k 2024-05-08T21:18:26.814 INFO:teuthology.orchestra.run.smithi186.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-05-08T21:18:26.814 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-05-08T21:18:26.814 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-05-08T21:18:26.815 INFO:teuthology.orchestra.run.smithi186.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-05-08T21:18:26.815 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-05-08T21:18:26.815 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-05-08T21:18:26.815 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 474 k 2024-05-08T21:18:26.815 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-05-08T21:18:26.815 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 6.5 M 2024-05-08T21:18:26.815 INFO:teuthology.orchestra.run.smithi186.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-05-08T21:18:26.815 INFO:teuthology.orchestra.run.smithi186.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-05-08T21:18:26.815 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-05-08T21:18:26.815 INFO:teuthology.orchestra.run.smithi186.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-05-08T21:18:26.815 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-05-08T21:18:26.815 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-05-08T21:18:26.815 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-05-08T21:18:26.815 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-05-08T21:18:26.816 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-05-08T21:18:26.816 INFO:teuthology.orchestra.run.smithi186.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-05-08T21:18:26.816 INFO:teuthology.orchestra.run.smithi186.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-05-08T21:18:26.816 INFO:teuthology.orchestra.run.smithi186.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-05-08T21:18:26.816 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-05-08T21:18:26.816 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-05-08T21:18:26.816 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-05-08T21:18:26.816 INFO:teuthology.orchestra.run.smithi186.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-05-08T21:18:26.816 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-05-08T21:18:26.816 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 45 k 2024-05-08T21:18:26.816 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 129 k 2024-05-08T21:18:26.816 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-05-08T21:18:26.816 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-05-08T21:18:26.816 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-05-08T21:18:26.817 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-05-08T21:18:26.817 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-05-08T21:18:26.817 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-05-08T21:18:26.817 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-05-08T21:18:26.817 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-05-08T21:18:26.817 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-05-08T21:18:26.817 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-05-08T21:18:26.817 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-05-08T21:18:26.817 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-05-08T21:18:26.817 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-05-08T21:18:26.817 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-05-08T21:18:26.817 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-05-08T21:18:26.817 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-05-08T21:18:26.817 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-05-08T21:18:26.818 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-05-08T21:18:26.818 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-05-08T21:18:26.818 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-05-08T21:18:26.818 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-05-08T21:18:26.818 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-05-08T21:18:26.818 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-05-08T21:18:26.818 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-05-08T21:18:26.818 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-05-08T21:18:26.818 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-05-08T21:18:26.818 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-05-08T21:18:26.818 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-05-08T21:18:26.818 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-05-08T21:18:26.818 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-05-08T21:18:26.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-05-08T21:18:26.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-05-08T21:18:26.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-05-08T21:18:26.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-05-08T21:18:26.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-05-08T21:18:26.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-05-08T21:18:26.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-05-08T21:18:26.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-05-08T21:18:26.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-05-08T21:18:26.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-05-08T21:18:26.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-05-08T21:18:26.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-05-08T21:18:26.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-05-08T21:18:26.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-05-08T21:18:26.820 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-05-08T21:18:26.820 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-05-08T21:18:26.820 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-05-08T21:18:26.820 INFO:teuthology.orchestra.run.smithi186.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-05-08T21:18:26.820 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-05-08T21:18:26.820 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-05-08T21:18:26.820 INFO:teuthology.orchestra.run.smithi186.stdout:Enabling module streams: 2024-05-08T21:18:26.820 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql 10 2024-05-08T21:18:26.820 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:18:26.820 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-08T21:18:26.820 INFO:teuthology.orchestra.run.smithi186.stdout:=================================================================================================================================== 2024-05-08T21:18:26.820 INFO:teuthology.orchestra.run.smithi186.stdout:Install 130 Packages 2024-05-08T21:18:26.820 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 16 Packages 2024-05-08T21:18:26.820 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:18:26.821 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 215 M 2024-05-08T21:18:26.821 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-08T21:18:27.093 INFO:teuthology.orchestra.run.smithi186.stdout:(1/146): ceph-18.2.2-1322.gd21e4f96.el8.x86_64. 45 kB/s | 6.4 kB 00:00 2024-05-08T21:18:27.352 INFO:teuthology.orchestra.run.smithi186.stdout:(2/146): ceph-fuse-18.2.2-1322.gd21e4f96.el8.x8 3.3 MB/s | 860 kB 00:00 2024-05-08T21:18:27.419 INFO:teuthology.orchestra.run.smithi186.stdout:(3/146): ceph-immutable-object-cache-18.2.2-132 2.4 MB/s | 166 kB 00:00 2024-05-08T21:18:27.619 INFO:teuthology.orchestra.run.smithi186.stdout:(4/146): ceph-base-18.2.2-1322.gd21e4f96.el8.x8 8.3 MB/s | 5.5 MB 00:00 2024-05-08T21:18:27.811 INFO:teuthology.orchestra.run.smithi186.stdout:(5/146): ceph-mgr-18.2.2-1322.gd21e4f96.el8.x86 8.1 MB/s | 1.6 MB 00:00 2024-05-08T21:18:27.895 INFO:teuthology.orchestra.run.smithi186.stdout:(6/146): ceph-mds-18.2.2-1322.gd21e4f96.el8.x86 4.8 MB/s | 2.3 MB 00:00 2024-05-08T21:18:28.353 INFO:teuthology.orchestra.run.smithi186.stdout:(7/146): ceph-mon-18.2.2-1322.gd21e4f96.el8.x86 8.9 MB/s | 4.8 MB 00:00 2024-05-08T21:18:28.879 INFO:teuthology.orchestra.run.smithi186.stdout:(8/146): ceph-common-18.2.2-1322.gd21e4f96.el8. 11 MB/s | 22 MB 00:01 2024-05-08T21:18:28.929 INFO:teuthology.orchestra.run.smithi186.stdout:(9/146): ceph-selinux-18.2.2-1322.gd21e4f96.el8 489 kB/s | 24 kB 00:00 2024-05-08T21:18:30.096 INFO:teuthology.orchestra.run.smithi186.stdout:(10/146): ceph-radosgw-18.2.2-1322.gd21e4f96.el 7.5 MB/s | 13 MB 00:01 2024-05-08T21:18:30.171 INFO:teuthology.orchestra.run.smithi186.stdout:(11/146): cephfs-mirror-18.2.2-1322.gd21e4f96.e 3.4 MB/s | 261 kB 00:00 2024-05-08T21:18:30.230 INFO:teuthology.orchestra.run.smithi186.stdout:(12/146): libcephfs-devel-18.2.2-1322.gd21e4f96 531 kB/s | 31 kB 00:00 2024-05-08T21:18:30.380 INFO:teuthology.orchestra.run.smithi186.stdout:(13/146): libcephfs2-18.2.2-1322.gd21e4f96.el8. 5.1 MB/s | 782 kB 00:00 2024-05-08T21:18:30.447 INFO:teuthology.orchestra.run.smithi186.stdout:(14/146): libcephsqlite-18.2.2-1322.gd21e4f96.e 2.1 MB/s | 142 kB 00:00 2024-05-08T21:18:30.539 INFO:teuthology.orchestra.run.smithi186.stdout:(15/146): librados-devel-18.2.2-1322.gd21e4f96. 1.5 MB/s | 139 kB 00:00 2024-05-08T21:18:30.631 INFO:teuthology.orchestra.run.smithi186.stdout:(16/146): libradosstriper1-18.2.2-1322.gd21e4f9 5.1 MB/s | 474 kB 00:00 2024-05-08T21:18:30.785 INFO:teuthology.orchestra.run.smithi003.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-05-08T21:18:30.786 INFO:teuthology.orchestra.run.smithi003.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-05-08T21:18:30.807 INFO:teuthology.orchestra.run.smithi003.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-05-08T21:18:30.808 INFO:teuthology.orchestra.run.smithi003.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-05-08T21:18:30.810 INFO:teuthology.orchestra.run.smithi003.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-05-08T21:18:30.812 INFO:teuthology.orchestra.run.smithi003.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-05-08T21:18:30.815 INFO:teuthology.orchestra.run.smithi003.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-05-08T21:18:30.819 INFO:teuthology.orchestra.run.smithi003.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-05-08T21:18:30.821 INFO:teuthology.orchestra.run.smithi003.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-05-08T21:18:31.066 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-05-08T21:18:31.082 INFO:teuthology.orchestra.run.smithi003.stdout:=================================================================================================================================== 2024-05-08T21:18:31.082 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-05-08T21:18:31.082 INFO:teuthology.orchestra.run.smithi003.stdout:=================================================================================================================================== 2024-05-08T21:18:31.083 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-05-08T21:18:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-15.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-05-08T21:18:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-05-08T21:18:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: ceph x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 6.4 k 2024-05-08T21:18:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 5.5 M 2024-05-08T21:18:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 860 k 2024-05-08T21:18:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 166 k 2024-05-08T21:18:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 1.6 M 2024-05-08T21:18:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 138 k 2024-05-08T21:18:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 3.5 M 2024-05-08T21:18:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 7.4 M 2024-05-08T21:18:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 49 k 2024-05-08T21:18:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 13 M 2024-05-08T21:18:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 47 M 2024-05-08T21:18:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 263 k 2024-05-08T21:18:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 170 k 2024-05-08T21:18:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-mirror x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 261 k 2024-05-08T21:18:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-top noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 18 k 2024-05-08T21:18:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-05-08T21:18:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-05-08T21:18:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-05-08T21:18:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 31 k 2024-05-08T21:18:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 782 k 2024-05-08T21:18:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 139 k 2024-05-08T21:18:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-05-08T21:18:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: libuuid-devel x86_64 2.32.1-44.el8_9.1 rhel-8-for-x86_64-baseos-rpms 100 k 2024-05-08T21:18:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-05-08T21:18:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-05-08T21:18:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-05-08T21:18:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-05-08T21:18:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 197 k 2024-05-08T21:18:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 370 k 2024-05-08T21:18:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 365 k 2024-05-08T21:18:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 107 k 2024-05-08T21:18:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 95 k 2024-05-08T21:18:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 3.5 M 2024-05-08T21:18:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 181 k 2024-05-08T21:18:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-05-08T21:18:31.086 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2024-05-08T21:18:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-05-08T21:18:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-05-08T21:18:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-05-08T21:18:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: libblkid x86_64 2.32.1-44.el8_9.1 rhel-8-for-x86_64-baseos-rpms 221 k 2024-05-08T21:18:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: libfdisk x86_64 2.32.1-44.el8_9.1 rhel-8-for-x86_64-baseos-rpms 254 k 2024-05-08T21:18:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: libmount x86_64 2.32.1-44.el8_9.1 rhel-8-for-x86_64-baseos-rpms 237 k 2024-05-08T21:18:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 3.7 M 2024-05-08T21:18:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 3.6 M 2024-05-08T21:18:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: libsmartcols x86_64 2.32.1-44.el8_9.1 rhel-8-for-x86_64-baseos-rpms 180 k 2024-05-08T21:18:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: libuuid x86_64 2.32.1-44.el8_9.1 rhel-8-for-x86_64-baseos-rpms 99 k 2024-05-08T21:18:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-05-08T21:18:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-05-08T21:18:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: util-linux x86_64 2.32.1-44.el8_9.1 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-05-08T21:18:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: util-linux-user x86_64 2.32.1-44.el8_9.1 rhel-8-for-x86_64-baseos-rpms 103 k 2024-05-08T21:18:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-05-08T21:18:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-05-08T21:18:31.087 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-05-08T21:18:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-05-08T21:18:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 22 M 2024-05-08T21:18:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 24 k 2024-05-08T21:18:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 2.3 M 2024-05-08T21:18:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 245 k 2024-05-08T21:18:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 4.8 M 2024-05-08T21:18:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 18 M 2024-05-08T21:18:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 16 k 2024-05-08T21:18:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 24 k 2024-05-08T21:18:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-05-08T21:18:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-05-08T21:18:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-05-08T21:18:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-05-08T21:18:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-05-08T21:18:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 142 k 2024-05-08T21:18:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-05-08T21:18:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-05-08T21:18:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-05-08T21:18:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-05-08T21:18:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-05-08T21:18:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-05-08T21:18:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 474 k 2024-05-08T21:18:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-05-08T21:18:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 6.5 M 2024-05-08T21:18:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-05-08T21:18:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-05-08T21:18:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-05-08T21:18:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-05-08T21:18:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-05-08T21:18:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-05-08T21:18:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-05-08T21:18:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-05-08T21:18:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-05-08T21:18:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-05-08T21:18:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-05-08T21:18:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-05-08T21:18:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-05-08T21:18:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-05-08T21:18:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-05-08T21:18:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-05-08T21:18:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-05-08T21:18:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 45 k 2024-05-08T21:18:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 129 k 2024-05-08T21:18:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-05-08T21:18:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-05-08T21:18:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-05-08T21:18:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-05-08T21:18:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-05-08T21:18:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-05-08T21:18:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-05-08T21:18:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-05-08T21:18:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-05-08T21:18:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-05-08T21:18:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-05-08T21:18:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-05-08T21:18:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-05-08T21:18:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-05-08T21:18:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-05-08T21:18:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-05-08T21:18:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-05-08T21:18:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-05-08T21:18:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-05-08T21:18:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-05-08T21:18:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-05-08T21:18:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-05-08T21:18:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-05-08T21:18:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-05-08T21:18:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-05-08T21:18:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-05-08T21:18:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-05-08T21:18:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-05-08T21:18:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-05-08T21:18:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-05-08T21:18:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-05-08T21:18:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-05-08T21:18:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-05-08T21:18:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-05-08T21:18:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-05-08T21:18:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-05-08T21:18:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-05-08T21:18:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-05-08T21:18:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-05-08T21:18:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-05-08T21:18:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-05-08T21:18:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-05-08T21:18:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-05-08T21:18:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-05-08T21:18:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-05-08T21:18:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-05-08T21:18:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-05-08T21:18:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-05-08T21:18:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-05-08T21:18:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-05-08T21:18:31.095 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling module streams: 2024-05-08T21:18:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql 10 2024-05-08T21:18:31.096 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:18:31.096 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-05-08T21:18:31.096 INFO:teuthology.orchestra.run.smithi003.stdout:=================================================================================================================================== 2024-05-08T21:18:31.096 INFO:teuthology.orchestra.run.smithi003.stdout:Install 130 Packages 2024-05-08T21:18:31.096 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 16 Packages 2024-05-08T21:18:31.096 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:18:31.097 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 215 M 2024-05-08T21:18:31.097 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-05-08T21:18:31.098 INFO:teuthology.orchestra.run.smithi186.stdout:(17/146): ceph-osd-18.2.2-1322.gd21e4f96.el8.x8 5.7 MB/s | 18 MB 00:03 2024-05-08T21:18:31.139 INFO:teuthology.orchestra.run.smithi186.stdout:(18/146): python3-ceph-argparse-18.2.2-1322.gd2 1.1 MB/s | 45 kB 00:00 2024-05-08T21:18:31.232 INFO:teuthology.orchestra.run.smithi186.stdout:(19/146): python3-ceph-common-18.2.2-1322.gd21e 1.4 MB/s | 129 kB 00:00 2024-05-08T21:18:31.315 INFO:teuthology.orchestra.run.smithi186.stdout:(20/146): python3-cephfs-18.2.2-1322.gd21e4f96. 2.3 MB/s | 197 kB 00:00 2024-05-08T21:18:31.441 INFO:teuthology.orchestra.run.smithi003.stdout:(1/146): ceph-18.2.2-1322.gd21e4f96.el8.x86_64. 31 kB/s | 6.4 kB 00:00 2024-05-08T21:18:31.449 INFO:teuthology.orchestra.run.smithi186.stdout:(21/146): librgw2-18.2.2-1322.gd21e4f96.el8.x86 8.0 MB/s | 6.5 MB 00:00 2024-05-08T21:18:31.474 INFO:teuthology.orchestra.run.smithi186.stdout:(22/146): python3-rados-18.2.2-1322.gd21e4f96.e 2.3 MB/s | 370 kB 00:00 2024-05-08T21:18:31.524 INFO:teuthology.orchestra.run.smithi186.stdout:(23/146): python3-rbd-18.2.2-1322.gd21e4f96.el8 4.7 MB/s | 365 kB 00:00 2024-05-08T21:18:31.558 INFO:teuthology.orchestra.run.smithi186.stdout:(24/146): python3-rgw-18.2.2-1322.gd21e4f96.el8 1.2 MB/s | 107 kB 00:00 2024-05-08T21:18:31.583 INFO:teuthology.orchestra.run.smithi186.stdout:(25/146): rbd-fuse-18.2.2-1322.gd21e4f96.el8.x8 1.6 MB/s | 95 kB 00:00 2024-05-08T21:18:31.675 INFO:teuthology.orchestra.run.smithi186.stdout:(26/146): rbd-nbd-18.2.2-1322.gd21e4f96.el8.x86 1.9 MB/s | 181 kB 00:00 2024-05-08T21:18:31.759 INFO:teuthology.orchestra.run.smithi186.stdout:(27/146): ceph-grafana-dashboards-18.2.2-1322.g 291 kB/s | 24 kB 00:00 2024-05-08T21:18:31.916 INFO:teuthology.orchestra.run.smithi003.stdout:(2/146): ceph-fuse-18.2.2-1322.gd21e4f96.el8.x8 1.8 MB/s | 860 kB 00:00 2024-05-08T21:18:31.983 INFO:teuthology.orchestra.run.smithi003.stdout:(3/146): ceph-immutable-object-cache-18.2.2-132 2.4 MB/s | 166 kB 00:00 2024-05-08T21:18:32.184 INFO:teuthology.orchestra.run.smithi186.stdout:(28/146): ceph-test-18.2.2-1322.gd21e4f96.el8.x 14 MB/s | 47 MB 00:03 2024-05-08T21:18:32.201 INFO:teuthology.orchestra.run.smithi186.stdout:(29/146): ceph-mgr-cephadm-18.2.2-1322.gd21e4f9 313 kB/s | 138 kB 00:00 2024-05-08T21:18:32.268 INFO:teuthology.orchestra.run.smithi186.stdout:(30/146): rbd-mirror-18.2.2-1322.gd21e4f96.el8. 4.9 MB/s | 3.5 MB 00:00 2024-05-08T21:18:32.535 INFO:teuthology.orchestra.run.smithi186.stdout:(31/146): ceph-mgr-dashboard-18.2.2-1322.gd21e4 10 MB/s | 3.5 MB 00:00 2024-05-08T21:18:32.583 INFO:teuthology.orchestra.run.smithi003.stdout:(4/146): ceph-mds-18.2.2-1322.gd21e4f96.el8.x86 3.8 MB/s | 2.3 MB 00:00 2024-05-08T21:18:32.614 INFO:teuthology.orchestra.run.smithi186.stdout:(32/146): ceph-mgr-rook-18.2.2-1322.gd21e4f96.e 625 kB/s | 49 kB 00:00 2024-05-08T21:18:32.667 INFO:teuthology.orchestra.run.smithi003.stdout:(5/146): ceph-base-18.2.2-1322.gd21e4f96.el8.x8 3.9 MB/s | 5.5 MB 00:01 2024-05-08T21:18:32.669 INFO:teuthology.orchestra.run.smithi186.stdout:(33/146): ceph-prometheus-alerts-18.2.2-1322.gd 300 kB/s | 16 kB 00:00 2024-05-08T21:18:32.736 INFO:teuthology.orchestra.run.smithi186.stdout:(34/146): ceph-volume-18.2.2-1322.gd21e4f96.el8 3.9 MB/s | 263 kB 00:00 2024-05-08T21:18:32.843 INFO:teuthology.orchestra.run.smithi186.stdout:(35/146): cephadm-18.2.2-1322.gd21e4f96.el8.noa 1.6 MB/s | 170 kB 00:00 2024-05-08T21:18:32.878 INFO:teuthology.orchestra.run.smithi186.stdout:(36/146): ceph-mgr-modules-core-18.2.2-1322.gd2 403 kB/s | 245 kB 00:00 2024-05-08T21:18:32.912 INFO:teuthology.orchestra.run.smithi186.stdout:(37/146): cephfs-top-18.2.2-1322.gd21e4f96.el8. 265 kB/s | 18 kB 00:00 2024-05-08T21:18:33.017 INFO:teuthology.orchestra.run.smithi003.stdout:(6/146): ceph-mgr-18.2.2-1322.gd21e4f96.el8.x86 3.6 MB/s | 1.6 MB 00:00 2024-05-08T21:18:33.162 INFO:teuthology.orchestra.run.smithi186.stdout:(38/146): ceph-mgr-diskprediction-local-18.2.2- 7.7 MB/s | 7.4 MB 00:00 2024-05-08T21:18:33.192 INFO:teuthology.orchestra.run.smithi003.stdout:(7/146): ceph-mon-18.2.2-1322.gd21e4f96.el8.x86 9.2 MB/s | 4.8 MB 00:00 2024-05-08T21:18:33.528 INFO:teuthology.orchestra.run.smithi186.stdout:(39/146): liboath-2.6.2-3.el8.x86_64.rpm 95 kB/s | 59 kB 00:00 2024-05-08T21:18:33.771 INFO:teuthology.orchestra.run.smithi186.stdout:(40/146): libunwind-1.3.1-3.el8.x86_64.rpm 123 kB/s | 75 kB 00:00 2024-05-08T21:18:33.796 INFO:teuthology.orchestra.run.smithi186.stdout:(41/146): gperftools-libs-2.7-9.el8.x86_64.rpm 334 kB/s | 306 kB 00:00 2024-05-08T21:18:33.871 INFO:teuthology.orchestra.run.smithi186.stdout:(42/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 436 kB/s | 44 kB 00:00 2024-05-08T21:18:33.988 INFO:teuthology.orchestra.run.smithi186.stdout:(43/146): python3-cachetools-3.1.1-4.el8.noarch 283 kB/s | 33 kB 00:00 2024-05-08T21:18:34.042 INFO:teuthology.orchestra.run.smithi003.stdout:(8/146): ceph-radosgw-18.2.2-1322.gd21e4f96.el8 15 MB/s | 13 MB 00:00 2024-05-08T21:18:34.064 INFO:teuthology.orchestra.run.smithi186.stdout:(44/146): python3-beautifulsoup4-4.6.3-2.el8.1. 693 kB/s | 185 kB 00:00 2024-05-08T21:18:34.080 INFO:teuthology.orchestra.run.smithi186.stdout:(45/146): python3-certifi-2018.10.15-7.el8.noar 177 kB/s | 16 kB 00:00 2024-05-08T21:18:34.092 INFO:teuthology.orchestra.run.smithi003.stdout:(9/146): ceph-selinux-18.2.2-1322.gd21e4f96.el8 491 kB/s | 24 kB 00:00 2024-05-08T21:18:34.239 INFO:teuthology.orchestra.run.smithi186.stdout:(46/146): python3-asyncssh-2.7.0-2.el8.noarch.r 670 kB/s | 476 kB 00:00 2024-05-08T21:18:34.331 INFO:teuthology.orchestra.run.smithi186.stdout:(47/146): python3-cheroot-8.5.2-1.el8.noarch.rp 646 kB/s | 173 kB 00:00 2024-05-08T21:18:34.459 INFO:teuthology.orchestra.run.smithi186.stdout:(48/146): python3-google-auth-1.1.1-10.el8.noar 374 kB/s | 81 kB 00:00 2024-05-08T21:18:34.484 INFO:teuthology.orchestra.run.smithi186.stdout:(49/146): python3-jaraco-6.2-6.el8.noarch.rpm 74 kB/s | 11 kB 00:00 2024-05-08T21:18:34.558 INFO:teuthology.orchestra.run.smithi186.stdout:(50/146): python3-jaraco-functools-2.0-4.el8.no 176 kB/s | 18 kB 00:00 2024-05-08T21:18:34.655 INFO:teuthology.orchestra.run.smithi186.stdout:(51/146): python3-cherrypy-18.4.0-1.el8.noarch. 670 kB/s | 384 kB 00:00 2024-05-08T21:18:34.683 INFO:teuthology.orchestra.run.smithi186.stdout:(52/146): python3-logutils-0.3.5-11.el8.noarch. 389 kB/s | 49 kB 00:00 2024-05-08T21:18:34.843 INFO:teuthology.orchestra.run.smithi186.stdout:(53/146): python3-more-itertools-7.2.0-3.el8.no 312 kB/s | 59 kB 00:00 2024-05-08T21:18:34.893 INFO:teuthology.orchestra.run.smithi186.stdout:(54/146): python3-natsort-7.1.1-2.el8.noarch.rp 287 kB/s | 60 kB 00:00 2024-05-08T21:18:35.002 INFO:teuthology.orchestra.run.smithi186.stdout:(55/146): python3-portend-2.6-1.el8.noarch.rpm 143 kB/s | 16 kB 00:00 2024-05-08T21:18:35.144 INFO:teuthology.orchestra.run.smithi186.stdout:(56/146): python3-repoze-lru-0.7-6.el8.noarch.r 238 kB/s | 34 kB 00:00 2024-05-08T21:18:35.303 INFO:teuthology.orchestra.run.smithi186.stdout:(57/146): python3-pecan-1.3.2-9.el8.noarch.rpm 616 kB/s | 283 kB 00:00 2024-05-08T21:18:35.563 INFO:teuthology.orchestra.run.smithi186.stdout:(58/146): python3-rsa-4.9-2.el8.noarch.rpm 247 kB/s | 64 kB 00:00 2024-05-08T21:18:35.587 INFO:teuthology.orchestra.run.smithi186.stdout:(59/146): python3-routes-2.4.1-12.el8.noarch.rp 444 kB/s | 196 kB 00:00 2024-05-08T21:18:35.678 INFO:teuthology.orchestra.run.smithi186.stdout:(60/146): python3-simplegeneric-0.8.1-17.el8.no 163 kB/s | 19 kB 00:00 2024-05-08T21:18:35.704 INFO:teuthology.orchestra.run.smithi186.stdout:(61/146): python3-singledispatch-3.4.0.3-18.el8 205 kB/s | 24 kB 00:00 2024-05-08T21:18:35.812 INFO:teuthology.orchestra.run.smithi186.stdout:(62/146): python3-tempora-1.14.1-5.el8.noarch.r 219 kB/s | 29 kB 00:00 2024-05-08T21:18:35.838 INFO:teuthology.orchestra.run.smithi186.stdout:(63/146): python3-trustme-0.6.0-4.el8.noarch.rp 206 kB/s | 27 kB 00:00 2024-05-08T21:18:36.084 INFO:teuthology.orchestra.run.smithi003.stdout:(10/146): ceph-common-18.2.2-1322.gd21e4f96.el8 4.5 MB/s | 22 MB 00:04 2024-05-08T21:18:36.188 INFO:teuthology.orchestra.run.smithi186.stdout:(64/146): python3-kubernetes-11.0.0-6.el8.noarc 700 kB/s | 1.2 MB 00:01 2024-05-08T21:18:36.280 INFO:teuthology.orchestra.run.smithi186.stdout:(65/146): python3-waitress-1.4.3-1.el8.noarch.r 546 kB/s | 255 kB 00:00 2024-05-08T21:18:36.322 INFO:teuthology.orchestra.run.smithi186.stdout:(66/146): python3-webob-1.8.5-1.el8.1.noarch.rp 519 kB/s | 251 kB 00:00 2024-05-08T21:18:36.422 INFO:teuthology.orchestra.run.smithi186.stdout:(67/146): python3-websocket-client-0.56.0-5.el8 262 kB/s | 61 kB 00:00 2024-05-08T21:18:36.472 INFO:teuthology.orchestra.run.smithi186.stdout:(68/146): python3-zc-lockfile-2.0-2.el8.noarch. 153 kB/s | 23 kB 00:00 2024-05-08T21:18:36.498 INFO:teuthology.orchestra.run.smithi186.stdout:(69/146): python3-webtest-2.0.33-1.el8.noarch.r 393 kB/s | 85 kB 00:00 2024-05-08T21:18:36.523 INFO:teuthology.orchestra.run.smithi186.stdout:(70/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2024-05-08T21:18:36.676 INFO:teuthology.orchestra.run.smithi003.stdout:(11/146): ceph-test-18.2.2-1322.gd21e4f96.el8.x 18 MB/s | 47 MB 00:02 2024-05-08T21:18:36.701 INFO:teuthology.orchestra.run.smithi003.stdout:(12/146): cephfs-mirror-18.2.2-1322.gd21e4f96.e 423 kB/s | 261 kB 00:00 2024-05-08T21:18:36.706 INFO:teuthology.orchestra.run.smithi186.stdout:(71/146): python3-jsonpointer-1.10-11.el8.noarc 96 kB/s | 20 kB 00:00 2024-05-08T21:18:36.726 INFO:teuthology.orchestra.run.smithi003.stdout:(13/146): libcephfs-devel-18.2.2-1322.gd21e4f96 620 kB/s | 31 kB 00:00 2024-05-08T21:18:36.732 INFO:teuthology.orchestra.run.smithi186.stdout:(72/146): python3-prettytable-0.7.2-14.el8.noar 216 kB/s | 44 kB 00:00 2024-05-08T21:18:36.790 INFO:teuthology.orchestra.run.smithi186.stdout:(73/146): python3-pyasn1-modules-0.3.7-6.el8.no 1.3 MB/s | 110 kB 00:00 2024-05-08T21:18:36.802 INFO:teuthology.orchestra.run.smithi003.stdout:(14/146): libcephsqlite-18.2.2-1322.gd21e4f96.e 1.8 MB/s | 142 kB 00:00 2024-05-08T21:18:36.824 INFO:teuthology.orchestra.run.smithi186.stdout:(74/146): python3-jsonpatch-1.21-2.el8.noarch.r 289 kB/s | 27 kB 00:00 2024-05-08T21:18:36.860 INFO:teuthology.orchestra.run.smithi003.stdout:(15/146): librados-devel-18.2.2-1322.gd21e4f96. 2.3 MB/s | 139 kB 00:00 2024-05-08T21:18:36.919 INFO:teuthology.orchestra.run.smithi003.stdout:(16/146): libradosstriper1-18.2.2-1322.gd21e4f9 8.0 MB/s | 474 kB 00:00 2024-05-08T21:18:36.932 INFO:teuthology.orchestra.run.smithi186.stdout:(75/146): python3-pyasn1-0.3.7-6.el8.noarch.rpm 891 kB/s | 126 kB 00:00 2024-05-08T21:18:36.944 INFO:teuthology.orchestra.run.smithi003.stdout:(17/146): libcephfs2-18.2.2-1322.gd21e4f96.el8. 3.1 MB/s | 782 kB 00:00 2024-05-08T21:18:36.974 INFO:teuthology.orchestra.run.smithi186.stdout:(76/146): python3-werkzeug-0.12.2-4.el8.noarch. 3.0 MB/s | 457 kB 00:00 2024-05-08T21:18:37.041 INFO:teuthology.orchestra.run.smithi186.stdout:(77/146): bison-3.0.4-10.el8.x86_64.rpm 6.2 MB/s | 688 kB 00:00 2024-05-08T21:18:37.044 INFO:teuthology.orchestra.run.smithi003.stdout:(18/146): python3-ceph-argparse-18.2.2-1322.gd2 448 kB/s | 45 kB 00:00 2024-05-08T21:18:37.075 INFO:teuthology.orchestra.run.smithi186.stdout:(78/146): flex-2.6.1-9.el8.x86_64.rpm 3.1 MB/s | 320 kB 00:00 2024-05-08T21:18:37.144 INFO:teuthology.orchestra.run.smithi003.stdout:(19/146): python3-ceph-common-18.2.2-1322.gd21e 1.3 MB/s | 129 kB 00:00 2024-05-08T21:18:37.217 INFO:teuthology.orchestra.run.smithi186.stdout:(79/146): python3-markupsafe-0.23-19.el8.x86_64 274 kB/s | 39 kB 00:00 2024-05-08T21:18:37.251 INFO:teuthology.orchestra.run.smithi186.stdout:(80/146): libtool-2.4.6-25.el8.x86_64.rpm 3.3 MB/s | 709 kB 00:00 2024-05-08T21:18:37.270 INFO:teuthology.orchestra.run.smithi003.stdout:(20/146): librgw2-18.2.2-1322.gd21e4f96.el8.x86 19 MB/s | 6.5 MB 00:00 2024-05-08T21:18:37.287 INFO:teuthology.orchestra.run.smithi003.stdout:(21/146): python3-cephfs-18.2.2-1322.gd21e4f96. 1.4 MB/s | 197 kB 00:00 2024-05-08T21:18:37.301 INFO:teuthology.orchestra.run.smithi186.stdout:(81/146): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-05-08T21:18:37.329 INFO:teuthology.orchestra.run.smithi003.stdout:(22/146): python3-rados-18.2.2-1322.gd21e4f96.e 6.2 MB/s | 370 kB 00:00 2024-05-08T21:18:37.369 INFO:teuthology.orchestra.run.smithi186.stdout:(82/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.7 MB/s | 259 kB 00:00 2024-05-08T21:18:37.396 INFO:teuthology.orchestra.run.smithi003.stdout:(23/146): python3-rgw-18.2.2-1322.gd21e4f96.el8 1.6 MB/s | 107 kB 00:00 2024-05-08T21:18:37.421 INFO:teuthology.orchestra.run.smithi003.stdout:(24/146): python3-rbd-18.2.2-1322.gd21e4f96.el8 2.7 MB/s | 365 kB 00:00 2024-05-08T21:18:37.435 INFO:teuthology.orchestra.run.smithi186.stdout:(83/146): python3-pyOpenSSL-19.0.0-1.el8.noarch 776 kB/s | 103 kB 00:00 2024-05-08T21:18:37.446 INFO:teuthology.orchestra.run.smithi003.stdout:(25/146): rbd-fuse-18.2.2-1322.gd21e4f96.el8.x8 1.9 MB/s | 95 kB 00:00 2024-05-08T21:18:37.494 INFO:teuthology.orchestra.run.smithi186.stdout:(84/146): python3-jinja2-2.10.1-3.el8.noarch.rp 9.0 MB/s | 538 kB 00:00 2024-05-08T21:18:37.496 INFO:teuthology.orchestra.run.smithi003.stdout:(26/146): rbd-nbd-18.2.2-1322.gd21e4f96.el8.x86 3.6 MB/s | 181 kB 00:00 2024-05-08T21:18:37.539 INFO:teuthology.orchestra.run.smithi003.stdout:(27/146): ceph-grafana-dashboards-18.2.2-1322.g 578 kB/s | 24 kB 00:00 2024-05-08T21:18:37.597 INFO:teuthology.orchestra.run.smithi003.stdout:(28/146): ceph-mgr-cephadm-18.2.2-1322.gd21e4f9 2.3 MB/s | 138 kB 00:00 2024-05-08T21:18:37.669 INFO:teuthology.orchestra.run.smithi186.stdout:(85/146): python3-babel-2.5.1-7.el8.noarch.rpm 16 MB/s | 4.8 MB 00:00 2024-05-08T21:18:37.761 INFO:teuthology.orchestra.run.smithi186.stdout:(86/146): python3-numpy-1.14.3-10.el8.x86_64.rp 14 MB/s | 3.7 MB 00:00 2024-05-08T21:18:37.786 INFO:teuthology.orchestra.run.smithi186.stdout:(87/146): socat-1.7.4.1-1.el8.x86_64.rpm 2.7 MB/s | 323 kB 00:00 2024-05-08T21:18:37.822 INFO:teuthology.orchestra.run.smithi003.stdout:(29/146): ceph-mgr-dashboard-18.2.2-1322.gd21e4 16 MB/s | 3.5 MB 00:00 2024-05-08T21:18:37.878 INFO:teuthology.orchestra.run.smithi186.stdout:(88/146): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.9 MB/s | 225 kB 00:00 2024-05-08T21:18:37.937 INFO:teuthology.orchestra.run.smithi186.stdout:(89/146): librdkafka-0.11.4-3.el8.x86_64.rpm 2.3 MB/s | 354 kB 00:00 2024-05-08T21:18:38.072 INFO:teuthology.orchestra.run.smithi186.stdout:(90/146): python3-mako-1.0.6-14.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2024-05-08T21:18:38.146 INFO:teuthology.orchestra.run.smithi186.stdout:(91/146): openblas-threads-0.3.15-6.el8.x86_64. 19 MB/s | 5.2 MB 00:00 2024-05-08T21:18:38.248 INFO:teuthology.orchestra.run.smithi003.stdout:(30/146): ceph-mgr-diskprediction-local-18.2.2- 17 MB/s | 7.4 MB 00:00 2024-05-08T21:18:38.263 INFO:teuthology.orchestra.run.smithi186.stdout:(92/146): openblas-0.3.15-6.el8.x86_64.rpm 27 MB/s | 5.0 MB 00:00 2024-05-08T21:18:38.298 INFO:teuthology.orchestra.run.smithi003.stdout:(31/146): rbd-mirror-18.2.2-1322.gd21e4f96.el8. 3.9 MB/s | 3.5 MB 00:00 2024-05-08T21:18:38.315 INFO:teuthology.orchestra.run.smithi003.stdout:(32/146): ceph-mgr-modules-core-18.2.2-1322.gd2 3.6 MB/s | 245 kB 00:00 2024-05-08T21:18:38.321 INFO:teuthology.orchestra.run.smithi186.stdout:(93/146): automake-1.16.1-8.el8.noarch.rpm 4.0 MB/s | 714 kB 00:00 2024-05-08T21:18:38.357 INFO:teuthology.orchestra.run.smithi003.stdout:(33/146): ceph-mgr-rook-18.2.2-1322.gd21e4f96.e 843 kB/s | 49 kB 00:00 2024-05-08T21:18:38.382 INFO:teuthology.orchestra.run.smithi003.stdout:(34/146): ceph-prometheus-alerts-18.2.2-1322.gd 244 kB/s | 16 kB 00:00 2024-05-08T21:18:38.416 INFO:teuthology.orchestra.run.smithi003.stdout:(35/146): ceph-volume-18.2.2-1322.gd21e4f96.el8 4.4 MB/s | 263 kB 00:00 2024-05-08T21:18:38.421 INFO:teuthology.orchestra.run.smithi186.stdout:(94/146): libpq-13.11-1.el8.x86_64.rpm 1.2 MB/s | 198 kB 00:00 2024-05-08T21:18:38.441 INFO:teuthology.orchestra.run.smithi003.stdout:(36/146): cephadm-18.2.2-1322.gd21e4f96.el8.noa 2.8 MB/s | 170 kB 00:00 2024-05-08T21:18:38.455 INFO:teuthology.orchestra.run.smithi186.stdout:(95/146): jq-1.6-7.el8.x86_64.rpm 1.5 MB/s | 202 kB 00:00 2024-05-08T21:18:38.564 INFO:teuthology.orchestra.run.smithi186.stdout:(96/146): python3-pytz-2017.2-11.el8.noarch.rpm 382 kB/s | 54 kB 00:00 2024-05-08T21:18:38.650 INFO:teuthology.orchestra.run.smithi003.stdout:(37/146): ceph-osd-18.2.2-1322.gd21e4f96.el8.x8 3.3 MB/s | 18 MB 00:05 2024-05-08T21:18:38.672 INFO:teuthology.orchestra.run.smithi186.stdout:(97/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-05-08T21:18:38.683 INFO:teuthology.orchestra.run.smithi003.stdout:(38/146): cephfs-top-18.2.2-1322.gd21e4f96.el8. 68 kB/s | 18 kB 00:00 2024-05-08T21:18:38.881 INFO:teuthology.orchestra.run.smithi186.stdout:(98/146): python3-scipy-1.0.0-21.module+el8.9.0 32 MB/s | 14 MB 00:00 2024-05-08T21:18:38.923 INFO:teuthology.orchestra.run.smithi186.stdout:(99/146): thrift-0.13.0-2.el8.x86_64.rpm 725 kB/s | 1.7 MB 00:02 2024-05-08T21:18:38.998 INFO:teuthology.orchestra.run.smithi186.stdout:(100/146): postgresql-contrib-10.23-4.module+el 6.8 MB/s | 812 kB 00:00 2024-05-08T21:18:39.057 INFO:teuthology.orchestra.run.smithi186.stdout:(101/146): postgresql-10.23-4.module+el8.9.0+21 3.9 MB/s | 1.5 MB 00:00 2024-05-08T21:18:39.067 INFO:teuthology.orchestra.run.smithi003.stdout:(39/146): liboath-2.6.2-3.el8.x86_64.rpm 143 kB/s | 59 kB 00:00 2024-05-08T21:18:39.116 INFO:teuthology.orchestra.run.smithi186.stdout:(102/146): libverto-devel-0.3.0-5.el8.x86_64.rp 160 kB/s | 18 kB 00:00 2024-05-08T21:18:39.125 INFO:teuthology.orchestra.run.smithi003.stdout:(40/146): libunwind-1.3.1-3.el8.x86_64.rpm 170 kB/s | 75 kB 00:00 2024-05-08T21:18:39.176 INFO:teuthology.orchestra.run.smithi003.stdout:(41/146): gperftools-libs-2.7-9.el8.x86_64.rpm 417 kB/s | 306 kB 00:00 2024-05-08T21:18:39.251 INFO:teuthology.orchestra.run.smithi003.stdout:(42/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 348 kB/s | 44 kB 00:00 2024-05-08T21:18:39.310 INFO:teuthology.orchestra.run.smithi003.stdout:(43/146): python3-beautifulsoup4-4.6.3-2.el8.1. 1.4 MB/s | 185 kB 00:00 2024-05-08T21:18:39.324 INFO:teuthology.orchestra.run.smithi186.stdout:(103/146): postgresql-server-10.23-4.module+el8 13 MB/s | 5.1 MB 00:00 2024-05-08T21:18:39.368 INFO:teuthology.orchestra.run.smithi003.stdout:(44/146): python3-cachetools-3.1.1-4.el8.noarch 282 kB/s | 33 kB 00:00 2024-05-08T21:18:39.410 INFO:teuthology.orchestra.run.smithi003.stdout:(45/146): python3-asyncssh-2.7.0-2.el8.noarch.r 1.4 MB/s | 476 kB 00:00 2024-05-08T21:18:39.435 INFO:teuthology.orchestra.run.smithi003.stdout:(46/146): python3-certifi-2018.10.15-7.el8.noar 130 kB/s | 16 kB 00:00 2024-05-08T21:18:39.544 INFO:teuthology.orchestra.run.smithi003.stdout:(47/146): python3-google-auth-1.1.1-10.el8.noar 750 kB/s | 81 kB 00:00 2024-05-08T21:18:39.577 INFO:teuthology.orchestra.run.smithi003.stdout:(48/146): python3-cherrypy-18.4.0-1.el8.noarch. 2.2 MB/s | 384 kB 00:00 2024-05-08T21:18:39.583 INFO:teuthology.orchestra.run.smithi186.stdout:(104/146): python3-jwt-1.6.1-2.el8.noarch.rpm 81 kB/s | 43 kB 00:00 2024-05-08T21:18:39.603 INFO:teuthology.orchestra.run.smithi003.stdout:(49/146): python3-cheroot-8.5.2-1.el8.noarch.rp 736 kB/s | 173 kB 00:00 2024-05-08T21:18:39.644 INFO:teuthology.orchestra.run.smithi003.stdout:(50/146): python3-jaraco-6.2-6.el8.noarch.rpm 110 kB/s | 11 kB 00:00 2024-05-08T21:18:39.703 INFO:teuthology.orchestra.run.smithi003.stdout:(51/146): python3-jaraco-functools-2.0-4.el8.no 140 kB/s | 18 kB 00:00 2024-05-08T21:18:39.716 INFO:teuthology.orchestra.run.smithi186.stdout:(105/146): python3-oauthlib-2.1.0-1.el8.noarch. 258 kB/s | 155 kB 00:00 2024-05-08T21:18:39.733 INFO:teuthology.orchestra.run.smithi186.stdout:(106/146): python3-pycparser-2.14-14.el8.noarch 267 kB/s | 109 kB 00:00 2024-05-08T21:18:39.754 INFO:teuthology.orchestra.run.smithi003.stdout:(52/146): python3-logutils-0.3.5-11.el8.noarch. 449 kB/s | 49 kB 00:00 2024-05-08T21:18:39.758 INFO:teuthology.orchestra.run.smithi186.stdout:(107/146): python3-requests-oauthlib-1.0.0-1.el 243 kB/s | 43 kB 00:00 2024-05-08T21:18:39.804 INFO:teuthology.orchestra.run.smithi003.stdout:(53/146): python3-more-itertools-7.2.0-3.el8.no 587 kB/s | 59 kB 00:00 2024-05-08T21:18:39.850 INFO:teuthology.orchestra.run.smithi186.stdout:(108/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 376 kB/s | 44 kB 00:00 2024-05-08T21:18:39.862 INFO:teuthology.orchestra.run.smithi003.stdout:(54/146): python3-natsort-7.1.1-2.el8.noarch.rp 555 kB/s | 60 kB 00:00 2024-05-08T21:18:39.867 INFO:teuthology.orchestra.run.smithi186.stdout:(109/146): libaio-devel-0.3.112-1.el8.x86_64.rp 140 kB/s | 19 kB 00:00 2024-05-08T21:18:39.929 INFO:teuthology.orchestra.run.smithi003.stdout:(55/146): python3-kubernetes-11.0.0-6.el8.noarc 3.6 MB/s | 1.2 MB 00:00 2024-05-08T21:18:39.955 INFO:teuthology.orchestra.run.smithi003.stdout:(56/146): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2024-05-08T21:18:39.976 INFO:teuthology.orchestra.run.smithi186.stdout:(110/146): pcre2-devel-10.32-2.el8.x86_64.rpm 2.7 MB/s | 605 kB 00:00 2024-05-08T21:18:39.980 INFO:teuthology.orchestra.run.smithi003.stdout:(57/146): python3-portend-2.6-1.el8.noarch.rpm 133 kB/s | 16 kB 00:00 2024-05-08T21:18:39.993 INFO:teuthology.orchestra.run.smithi186.stdout:(111/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.7 MB/s | 220 kB 00:00 2024-05-08T21:18:40.018 INFO:teuthology.orchestra.run.smithi186.stdout:(112/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.3 MB/s | 229 kB 00:00 2024-05-08T21:18:40.030 INFO:teuthology.orchestra.run.smithi003.stdout:(58/146): python3-repoze-lru-0.7-6.el8.noarch.r 333 kB/s | 34 kB 00:00 2024-05-08T21:18:40.081 INFO:teuthology.orchestra.run.smithi003.stdout:(59/146): python3-rsa-4.9-2.el8.noarch.rpm 640 kB/s | 64 kB 00:00 2024-05-08T21:18:40.106 INFO:teuthology.orchestra.run.smithi003.stdout:(60/146): python3-routes-2.4.1-12.el8.noarch.rp 1.3 MB/s | 196 kB 00:00 2024-05-08T21:18:40.139 INFO:teuthology.orchestra.run.smithi003.stdout:(61/146): python3-simplegeneric-0.8.1-17.el8.no 172 kB/s | 19 kB 00:00 2024-05-08T21:18:40.140 INFO:teuthology.orchestra.run.smithi186.stdout:(113/146): keyutils-libs-devel-1.5.10-9.el8.x86 328 kB/s | 48 kB 00:00 2024-05-08T21:18:40.169 INFO:teuthology.orchestra.run.smithi186.stdout:(114/146): libselinux-devel-2.9-5.el8.x86_64.rp 1.0 MB/s | 200 kB 00:00 2024-05-08T21:18:40.181 INFO:teuthology.orchestra.run.smithi003.stdout:(62/146): python3-singledispatch-3.4.0.3-18.el8 238 kB/s | 24 kB 00:00 2024-05-08T21:18:40.186 INFO:teuthology.orchestra.run.smithi186.stdout:(115/146): libsepol-devel-2.9-3.el8.x86_64.rpm 520 kB/s | 87 kB 00:00 2024-05-08T21:18:40.206 INFO:teuthology.orchestra.run.smithi003.stdout:(63/146): python3-tempora-1.14.1-5.el8.noarch.r 291 kB/s | 29 kB 00:00 2024-05-08T21:18:40.228 INFO:teuthology.orchestra.run.smithi186.stdout:(116/146): zlib-devel-1.2.11-18.el8_5.x86_64.rp 660 kB/s | 58 kB 00:00 2024-05-08T21:18:40.261 INFO:teuthology.orchestra.run.smithi003.stdout:(64/146): python3-trustme-0.6.0-4.el8.noarch.rp 226 kB/s | 27 kB 00:00 2024-05-08T21:18:40.303 INFO:teuthology.orchestra.run.smithi186.stdout:(117/146): fuse-2.9.7-17.el8.x86_64.rpm 710 kB/s | 83 kB 00:00 2024-05-08T21:18:40.307 INFO:teuthology.orchestra.run.smithi003.stdout:(65/146): python3-waitress-1.4.3-1.el8.noarch.r 2.0 MB/s | 255 kB 00:00 2024-05-08T21:18:40.328 INFO:teuthology.orchestra.run.smithi186.stdout:(118/146): libcom_err-devel-1.45.6-4.el8.x86_64 242 kB/s | 39 kB 00:00 2024-05-08T21:18:40.357 INFO:teuthology.orchestra.run.smithi003.stdout:(66/146): python3-webob-1.8.5-1.el8.1.noarch.rp 1.6 MB/s | 251 kB 00:00 2024-05-08T21:18:40.386 INFO:teuthology.orchestra.run.smithi003.stdout:(67/146): python3-websocket-client-0.56.0-5.el8 505 kB/s | 61 kB 00:00 2024-05-08T21:18:40.387 INFO:teuthology.orchestra.run.smithi186.stdout:(119/146): libgfortran-8.5.0-20.el8.x86_64.rpm 4.0 MB/s | 645 kB 00:00 2024-05-08T21:18:40.404 INFO:teuthology.orchestra.run.smithi186.stdout:(120/146): libquadmath-8.5.0-20.el8.x86_64.rpm 1.7 MB/s | 172 kB 00:00 2024-05-08T21:18:40.416 INFO:teuthology.orchestra.run.smithi003.stdout:(68/146): python3-webtest-2.0.33-1.el8.noarch.r 786 kB/s | 85 kB 00:00 2024-05-08T21:18:40.433 INFO:teuthology.orchestra.run.smithi003.stdout:(69/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 4.1 MB/s | 70 kB 00:00 2024-05-08T21:18:40.450 INFO:teuthology.orchestra.run.smithi003.stdout:(70/146): python3-zc-lockfile-2.0-2.el8.noarch. 249 kB/s | 23 kB 00:00 2024-05-08T21:18:40.462 INFO:teuthology.orchestra.run.smithi186.stdout:(121/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.7 MB/s | 238 kB 00:00 2024-05-08T21:18:40.488 INFO:teuthology.orchestra.run.smithi186.stdout:(122/146): xfsprogs-devel-5.0.0-12.el8.x86_64.r 723 kB/s | 73 kB 00:00 2024-05-08T21:18:40.513 INFO:teuthology.orchestra.run.smithi186.stdout:(123/146): elfutils-libelf-devel-0.189-3.el8.x8 567 kB/s | 62 kB 00:00 2024-05-08T21:18:40.550 INFO:teuthology.orchestra.run.smithi003.stdout:(71/146): python3-jsonpointer-1.10-11.el8.noarc 171 kB/s | 20 kB 00:00 2024-05-08T21:18:40.555 INFO:teuthology.orchestra.run.smithi186.stdout:(124/146): krb5-devel-1.18.2-26.el8_9.x86_64.rp 6.0 MB/s | 562 kB 00:00 2024-05-08T21:18:40.592 INFO:teuthology.orchestra.run.smithi003.stdout:(72/146): thrift-0.13.0-2.el8.x86_64.rpm 8.3 MB/s | 1.7 MB 00:00 2024-05-08T21:18:40.609 INFO:teuthology.orchestra.run.smithi003.stdout:(73/146): python3-prettytable-0.7.2-14.el8.noar 277 kB/s | 44 kB 00:00 2024-05-08T21:18:40.614 INFO:teuthology.orchestra.run.smithi186.stdout:(125/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 470 kB/s | 47 kB 00:00 2024-05-08T21:18:40.681 INFO:teuthology.orchestra.run.smithi186.stdout:(126/146): openssl-devel-1.1.1k-12.el8_9.x86_64 12 MB/s | 2.3 MB 00:00 2024-05-08T21:18:40.684 INFO:teuthology.orchestra.run.smithi003.stdout:(74/146): python3-pyasn1-modules-0.3.7-6.el8.no 822 kB/s | 110 kB 00:00 2024-05-08T21:18:40.731 INFO:teuthology.orchestra.run.smithi186.stdout:(127/146): python3-cryptography-3.2.1-7.el8_9.x 3.1 MB/s | 559 kB 00:00 2024-05-08T21:18:40.734 INFO:teuthology.orchestra.run.smithi003.stdout:(75/146): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2024-05-08T21:18:40.756 INFO:teuthology.orchestra.run.smithi186.stdout:(128/146): NetworkManager-initscripts-updown-1. 1.0 MB/s | 144 kB 00:00 2024-05-08T21:18:40.777 INFO:teuthology.orchestra.run.smithi003.stdout:(76/146): python3-jsonpatch-1.21-2.el8.noarch.r 144 kB/s | 27 kB 00:00 2024-05-08T21:18:40.839 INFO:teuthology.orchestra.run.smithi003.stdout:(77/146): python3-werkzeug-0.12.2-4.el8.noarch. 2.9 MB/s | 457 kB 00:00 2024-05-08T21:18:40.865 INFO:teuthology.orchestra.run.smithi186.stdout:(129/146): libuuid-devel-2.32.1-44.el8_9.1.x86_ 543 kB/s | 100 kB 00:00 2024-05-08T21:18:40.936 INFO:teuthology.orchestra.run.smithi003.stdout:(78/146): flex-2.6.1-9.el8.x86_64.rpm 2.0 MB/s | 320 kB 00:00 2024-05-08T21:18:40.948 INFO:teuthology.orchestra.run.smithi186.stdout:(130/146): python3-packaging-16.8-9.el8.noarch. 222 kB/s | 48 kB 00:00 2024-05-08T21:18:41.002 INFO:teuthology.orchestra.run.smithi003.stdout:(79/146): bison-3.0.4-10.el8.x86_64.rpm 2.5 MB/s | 688 kB 00:00 2024-05-08T21:18:41.061 INFO:teuthology.orchestra.run.smithi003.stdout:(80/146): libtool-2.4.6-25.el8.x86_64.rpm 3.1 MB/s | 709 kB 00:00 2024-05-08T21:18:41.086 INFO:teuthology.orchestra.run.smithi003.stdout:(81/146): python3-markupsafe-0.23-19.el8.x86_64 258 kB/s | 39 kB 00:00 2024-05-08T21:18:41.115 INFO:teuthology.orchestra.run.smithi186.stdout:(131/146): elfutils-libelf-0.189-3.el8.x86_64.r 1.4 MB/s | 232 kB 00:00 2024-05-08T21:18:41.365 INFO:teuthology.orchestra.run.smithi186.stdout:(132/146): elfutils-libs-0.189-3.el8.x86_64.rpm 1.2 MB/s | 303 kB 00:00 2024-05-08T21:18:41.445 INFO:teuthology.orchestra.run.smithi003.stdout:(82/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 586 kB/s | 259 kB 00:00 2024-05-08T21:18:41.516 INFO:teuthology.orchestra.run.smithi186.stdout:(133/146): iproute-6.2.0-5.el8_9.x86_64.rpm 5.6 MB/s | 854 kB 00:00 2024-05-08T21:18:41.570 INFO:teuthology.orchestra.run.smithi003.stdout:(83/146): uuid-1.6.2-43.el8.x86_64.rpm 126 kB/s | 64 kB 00:00 2024-05-08T21:18:41.587 INFO:teuthology.orchestra.run.smithi003.stdout:(84/146): python3-pyOpenSSL-19.0.0-1.el8.noarch 207 kB/s | 103 kB 00:00 2024-05-08T21:18:41.633 INFO:teuthology.orchestra.run.smithi186.stdout:(134/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 6.0 MB/s | 711 kB 00:00 2024-05-08T21:18:41.721 INFO:teuthology.orchestra.run.smithi003.stdout:(85/146): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2024-05-08T21:18:41.733 INFO:teuthology.orchestra.run.smithi186.stdout:(135/146): librbd1-18.2.2-1322.gd21e4f96.el8.x8 4.1 MB/s | 3.6 MB 00:00 2024-05-08T21:18:41.754 INFO:teuthology.orchestra.run.smithi003.stdout:(86/146): python3-jinja2-2.10.1-3.el8.noarch.rp 2.9 MB/s | 538 kB 00:00 2024-05-08T21:18:41.800 INFO:teuthology.orchestra.run.smithi186.stdout:(136/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 8.8 MB/s | 1.5 MB 00:00 2024-05-08T21:18:41.834 INFO:teuthology.orchestra.run.smithi186.stdout:(137/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 3.2 MB/s | 333 kB 00:00 2024-05-08T21:18:41.846 INFO:teuthology.orchestra.run.smithi003.stdout:(87/146): python3-numpy-1.14.3-10.el8.x86_64.rp 14 MB/s | 3.7 MB 00:00 2024-05-08T21:18:41.880 INFO:teuthology.orchestra.run.smithi003.stdout:(88/146): socat-1.7.4.1-1.el8.x86_64.rpm 2.0 MB/s | 323 kB 00:00 2024-05-08T21:18:41.905 INFO:teuthology.orchestra.run.smithi003.stdout:(89/146): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.5 MB/s | 225 kB 00:00 2024-05-08T21:18:41.939 INFO:teuthology.orchestra.run.smithi003.stdout:(90/146): librdkafka-0.11.4-3.el8.x86_64.rpm 3.8 MB/s | 354 kB 00:00 2024-05-08T21:18:41.941 INFO:teuthology.orchestra.run.smithi186.stdout:(138/146): librados2-18.2.2-1322.gd21e4f96.el8. 3.1 MB/s | 3.7 MB 00:01 2024-05-08T21:18:41.993 INFO:teuthology.orchestra.run.smithi186.stdout:(139/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 5.6 MB/s | 1.1 MB 00:00 2024-05-08T21:18:42.014 INFO:teuthology.orchestra.run.smithi003.stdout:(91/146): python3-mako-1.0.6-14.el8.noarch.rpm 1.4 MB/s | 157 kB 00:00 2024-05-08T21:18:42.018 INFO:teuthology.orchestra.run.smithi186.stdout:(140/146): libblkid-2.32.1-44.el8_9.1.x86_64.rp 1.2 MB/s | 221 kB 00:00 2024-05-08T21:18:42.068 INFO:teuthology.orchestra.run.smithi186.stdout:(141/146): libfdisk-2.32.1-44.el8_9.1.x86_64.rp 2.0 MB/s | 254 kB 00:00 2024-05-08T21:18:42.110 INFO:teuthology.orchestra.run.smithi186.stdout:(142/146): libmount-2.32.1-44.el8_9.1.x86_64.rp 2.0 MB/s | 237 kB 00:00 2024-05-08T21:18:42.164 INFO:teuthology.orchestra.run.smithi003.stdout:(92/146): openblas-threads-0.3.15-6.el8.x86_64. 18 MB/s | 5.2 MB 00:00 2024-05-08T21:18:42.202 INFO:teuthology.orchestra.run.smithi186.stdout:(143/146): libuuid-2.32.1-44.el8_9.1.x86_64.rpm 742 kB/s | 99 kB 00:00 2024-05-08T21:18:42.215 INFO:teuthology.orchestra.run.smithi003.stdout:(93/146): automake-1.16.1-8.el8.noarch.rpm 3.5 MB/s | 714 kB 00:00 2024-05-08T21:18:42.236 INFO:teuthology.orchestra.run.smithi186.stdout:(144/146): libsmartcols-2.32.1-44.el8_9.1.x86_6 827 kB/s | 180 kB 00:00 2024-05-08T21:18:42.290 INFO:teuthology.orchestra.run.smithi003.stdout:(94/146): openblas-0.3.15-6.el8.x86_64.rpm 14 MB/s | 5.0 MB 00:00 2024-05-08T21:18:42.294 INFO:teuthology.orchestra.run.smithi186.stdout:(145/146): util-linux-2.32.1-44.el8_9.1.x86_64. 14 MB/s | 2.5 MB 00:00 2024-05-08T21:18:42.311 INFO:teuthology.orchestra.run.smithi186.stdout:(146/146): util-linux-user-2.32.1-44.el8_9.1.x8 963 kB/s | 103 kB 00:00 2024-05-08T21:18:42.315 INFO:teuthology.orchestra.run.smithi003.stdout:(95/146): libpq-13.11-1.el8.x86_64.rpm 1.3 MB/s | 198 kB 00:00 2024-05-08T21:18:42.315 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-08T21:18:42.315 INFO:teuthology.orchestra.run.smithi186.stdout:Total 14 MB/s | 215 MB 00:15 2024-05-08T21:18:42.349 INFO:teuthology.orchestra.run.smithi003.stdout:(96/146): python3-pytz-2017.2-11.el8.noarch.rpm 927 kB/s | 54 kB 00:00 2024-05-08T21:18:42.374 INFO:teuthology.orchestra.run.smithi003.stdout:(97/146): jq-1.6-7.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2024-05-08T21:18:42.475 INFO:teuthology.orchestra.run.smithi003.stdout:(98/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-05-08T21:18:42.542 INFO:teuthology.orchestra.run.smithi003.stdout:(99/146): postgresql-10.23-4.module+el8.9.0+213 9.0 MB/s | 1.5 MB 00:00 2024-05-08T21:18:42.650 INFO:teuthology.orchestra.run.smithi003.stdout:(100/146): postgresql-contrib-10.23-4.module+el 4.6 MB/s | 812 kB 00:00 2024-05-08T21:18:42.750 INFO:teuthology.orchestra.run.smithi003.stdout:(101/146): python3-packaging-16.8-9.el8.noarch. 483 kB/s | 48 kB 00:00 2024-05-08T21:18:42.909 INFO:teuthology.orchestra.run.smithi003.stdout:(102/146): python3-scipy-1.0.0-21.module+el8.9. 23 MB/s | 14 MB 00:00 2024-05-08T21:18:42.934 INFO:teuthology.orchestra.run.smithi003.stdout:(103/146): libverto-devel-0.3.0-5.el8.x86_64.rp 98 kB/s | 18 kB 00:00 2024-05-08T21:18:43.009 INFO:teuthology.orchestra.run.smithi003.stdout:(104/146): postgresql-server-10.23-4.module+el8 11 MB/s | 5.1 MB 00:00 2024-05-08T21:18:43.051 INFO:teuthology.orchestra.run.smithi003.stdout:(105/146): python3-oauthlib-2.1.0-1.el8.noarch. 1.3 MB/s | 155 kB 00:00 2024-05-08T21:18:43.068 INFO:teuthology.orchestra.run.smithi003.stdout:(106/146): python3-jwt-1.6.1-2.el8.noarch.rpm 268 kB/s | 43 kB 00:00 2024-05-08T21:18:43.143 INFO:teuthology.orchestra.run.smithi003.stdout:(107/146): python3-pycparser-2.14-14.el8.noarch 816 kB/s | 109 kB 00:00 2024-05-08T21:18:43.227 INFO:teuthology.orchestra.run.smithi003.stdout:(108/146): libaio-devel-0.3.112-1.el8.x86_64.rp 118 kB/s | 19 kB 00:00 2024-05-08T21:18:43.244 INFO:teuthology.orchestra.run.smithi003.stdout:(109/146): python3-requests-oauthlib-1.0.0-1.el 222 kB/s | 43 kB 00:00 2024-05-08T21:18:43.278 INFO:teuthology.orchestra.run.smithi003.stdout:(110/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 327 kB/s | 44 kB 00:00 2024-05-08T21:18:43.378 INFO:teuthology.orchestra.run.smithi003.stdout:(111/146): pcre2-devel-10.32-2.el8.x86_64.rpm 3.9 MB/s | 605 kB 00:00 2024-05-08T21:18:43.403 INFO:teuthology.orchestra.run.smithi003.stdout:(112/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.4 MB/s | 229 kB 00:00 2024-05-08T21:18:43.412 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-08T21:18:43.428 INFO:teuthology.orchestra.run.smithi003.stdout:(113/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.4 MB/s | 220 kB 00:00 2024-05-08T21:18:43.479 INFO:teuthology.orchestra.run.smithi003.stdout:(114/146): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2024-05-08T21:18:43.504 INFO:teuthology.orchestra.run.smithi003.stdout:(115/146): libselinux-devel-2.9-5.el8.x86_64.rp 1.6 MB/s | 200 kB 00:00 2024-05-08T21:18:43.529 INFO:teuthology.orchestra.run.smithi003.stdout:(116/146): keyutils-libs-devel-1.5.10-9.el8.x86 383 kB/s | 48 kB 00:00 2024-05-08T21:18:43.588 INFO:teuthology.orchestra.run.smithi003.stdout:(117/146): libcom_err-devel-1.45.6-4.el8.x86_64 462 kB/s | 39 kB 00:00 2024-05-08T21:18:43.605 INFO:teuthology.orchestra.run.smithi003.stdout:(118/146): zlib-devel-1.2.11-18.el8_5.x86_64.rp 459 kB/s | 58 kB 00:00 2024-05-08T21:18:43.672 INFO:teuthology.orchestra.run.smithi003.stdout:(119/146): fuse-2.9.7-17.el8.x86_64.rpm 583 kB/s | 83 kB 00:00 2024-05-08T21:18:43.722 INFO:teuthology.orchestra.run.smithi003.stdout:(120/146): libgfortran-8.5.0-20.el8.x86_64.rpm 4.7 MB/s | 645 kB 00:00 2024-05-08T21:18:43.739 INFO:teuthology.orchestra.run.smithi003.stdout:(121/146): libquadmath-8.5.0-20.el8.x86_64.rpm 1.3 MB/s | 172 kB 00:00 2024-05-08T21:18:43.797 INFO:teuthology.orchestra.run.smithi003.stdout:(122/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.9 MB/s | 238 kB 00:00 2024-05-08T21:18:43.809 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-08T21:18:43.810 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-08T21:18:43.823 INFO:teuthology.orchestra.run.smithi003.stdout:(123/146): xfsprogs-devel-5.0.0-12.el8.x86_64.r 722 kB/s | 73 kB 00:00 2024-05-08T21:18:43.848 INFO:teuthology.orchestra.run.smithi003.stdout:(124/146): elfutils-libelf-devel-0.189-3.el8.x8 568 kB/s | 62 kB 00:00 2024-05-08T21:18:43.949 INFO:teuthology.orchestra.run.smithi003.stdout:(125/146): openssl-devel-1.1.1k-12.el8_9.x86_64 19 MB/s | 2.3 MB 00:00 2024-05-08T21:18:43.982 INFO:teuthology.orchestra.run.smithi003.stdout:(126/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 352 kB/s | 47 kB 00:00 2024-05-08T21:18:44.007 INFO:teuthology.orchestra.run.smithi003.stdout:(127/146): krb5-devel-1.18.2-26.el8_9.x86_64.rp 2.6 MB/s | 562 kB 00:00 2024-05-08T21:18:44.099 INFO:teuthology.orchestra.run.smithi003.stdout:(128/146): NetworkManager-initscripts-updown-1. 1.2 MB/s | 144 kB 00:00 2024-05-08T21:18:44.149 INFO:teuthology.orchestra.run.smithi003.stdout:(129/146): python3-cryptography-3.2.1-7.el8_9.x 2.7 MB/s | 559 kB 00:00 2024-05-08T21:18:44.191 INFO:teuthology.orchestra.run.smithi003.stdout:(130/146): libuuid-devel-2.32.1-44.el8_9.1.x86_ 543 kB/s | 100 kB 00:00 2024-05-08T21:18:44.308 INFO:teuthology.orchestra.run.smithi003.stdout:(131/146): elfutils-libelf-0.189-3.el8.x86_64.r 1.9 MB/s | 232 kB 00:00 2024-05-08T21:18:44.450 INFO:teuthology.orchestra.run.smithi003.stdout:(132/146): elfutils-libs-0.189-3.el8.x86_64.rpm 2.1 MB/s | 303 kB 00:00 2024-05-08T21:18:44.509 INFO:teuthology.orchestra.run.smithi003.stdout:(133/146): librbd1-18.2.2-1322.gd21e4f96.el8.x8 9.9 MB/s | 3.6 MB 00:00 2024-05-08T21:18:44.559 INFO:teuthology.orchestra.run.smithi003.stdout:(134/146): iproute-6.2.0-5.el8_9.x86_64.rpm 7.7 MB/s | 854 kB 00:00 2024-05-08T21:18:44.744 INFO:teuthology.orchestra.run.smithi003.stdout:(135/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 3.0 MB/s | 711 kB 00:00 2024-05-08T21:18:44.801 INFO:teuthology.orchestra.run.smithi003.stdout:(136/146): librados2-18.2.2-1322.gd21e4f96.el8. 5.3 MB/s | 3.7 MB 00:00 2024-05-08T21:18:44.852 INFO:teuthology.orchestra.run.smithi003.stdout:(137/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 5.0 MB/s | 1.5 MB 00:00 2024-05-08T21:18:44.885 INFO:teuthology.orchestra.run.smithi003.stdout:(138/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 2.3 MB/s | 333 kB 00:00 2024-05-08T21:18:44.978 INFO:teuthology.orchestra.run.smithi003.stdout:(139/146): libblkid-2.32.1-44.el8_9.1.x86_64.rp 1.7 MB/s | 221 kB 00:00 2024-05-08T21:18:45.019 INFO:teuthology.orchestra.run.smithi003.stdout:(140/146): libfdisk-2.32.1-44.el8_9.1.x86_64.rp 1.9 MB/s | 254 kB 00:00 2024-05-08T21:18:45.069 INFO:teuthology.orchestra.run.smithi003.stdout:(141/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 4.0 MB/s | 1.1 MB 00:00 2024-05-08T21:18:45.095 INFO:teuthology.orchestra.run.smithi003.stdout:(142/146): libmount-2.32.1-44.el8_9.1.x86_64.rp 2.0 MB/s | 237 kB 00:00 2024-05-08T21:18:45.110 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-08T21:18:45.120 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-08T21:18:45.136 INFO:teuthology.orchestra.run.smithi003.stdout:(143/146): libsmartcols-2.32.1-44.el8_9.1.x86_6 1.5 MB/s | 180 kB 00:00 2024-05-08T21:18:45.170 INFO:teuthology.orchestra.run.smithi003.stdout:(144/146): libuuid-2.32.1-44.el8_9.1.x86_64.rpm 985 kB/s | 99 kB 00:00 2024-05-08T21:18:45.254 INFO:teuthology.orchestra.run.smithi003.stdout:(145/146): util-linux-user-2.32.1-44.el8_9.1.x8 879 kB/s | 103 kB 00:00 2024-05-08T21:18:45.312 INFO:teuthology.orchestra.run.smithi003.stdout:(146/146): util-linux-2.32.1-44.el8_9.1.x86_64. 11 MB/s | 2.5 MB 00:00 2024-05-08T21:18:45.316 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-05-08T21:18:45.316 INFO:teuthology.orchestra.run.smithi003.stdout:Total 15 MB/s | 215 MB 00:14 2024-05-08T21:18:46.445 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-05-08T21:18:46.897 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-05-08T21:18:46.897 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-05-08T21:18:48.231 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-05-08T21:18:48.237 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-05-08T21:18:51.100 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-08T21:18:51.236 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-05-08T21:18:51.491 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-05-08T21:18:51.709 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-05-08T21:18:51.927 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-05-08T21:18:52.073 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libuuid-2.32.1-44.el8_9.1.x86_64 3/162 2024-05-08T21:18:52.178 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libuuid-2.32.1-44.el8_9.1.x86_64 3/162 2024-05-08T21:18:52.332 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libblkid-2.32.1-44.el8_9.1.x86_64 4/162 2024-05-08T21:18:52.406 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libblkid-2.32.1-44.el8_9.1.x86_64 4/162 2024-05-08T21:18:52.662 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-05-08T21:18:52.865 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/162 2024-05-08T21:18:52.932 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/162 2024-05-08T21:18:53.157 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 7/162 2024-05-08T21:18:53.373 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-05-08T21:18:53.781 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-05-08T21:18:53.953 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-05-08T21:18:53.989 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 9/162 2024-05-08T21:18:54.102 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-05-08T21:18:54.362 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-05-08T21:18:54.388 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 9/162 2024-05-08T21:18:54.589 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 10/162 2024-05-08T21:18:54.595 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-05-08T21:18:54.740 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 10/162 2024-05-08T21:18:54.836 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-05-08T21:18:54.932 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 11/162 2024-05-08T21:18:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libuuid-2.32.1-44.el8_9.1.x86_64 3/162 2024-05-08T21:18:55.019 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 11/162 2024-05-08T21:18:55.082 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libuuid-2.32.1-44.el8_9.1.x86_64 3/162 2024-05-08T21:18:55.248 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libblkid-2.32.1-44.el8_9.1.x86_64 4/162 2024-05-08T21:18:55.308 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:18.2.2-1322.gd21e4f96.el8.x86_64 12/162 2024-05-08T21:18:55.343 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libblkid-2.32.1-44.el8_9.1.x86_64 4/162 2024-05-08T21:18:55.541 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-05-08T21:18:55.597 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-05-08T21:18:55.689 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/162 2024-05-08T21:18:55.759 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/162 2024-05-08T21:18:55.796 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-05-08T21:18:55.971 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-05-08T21:18:55.988 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 7/162 2024-05-08T21:18:56.180 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-05-08T21:18:56.219 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-05-08T21:18:56.392 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-05-08T21:18:56.628 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-05-08T21:18:56.786 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-05-08T21:18:56.814 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 9/162 2024-05-08T21:18:57.026 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-05-08T21:18:57.181 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:18.2.2-1322.gd21e4f96.el8.x86_64 19/162 2024-05-08T21:18:57.235 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 9/162 2024-05-08T21:18:57.276 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1322.gd21e4f96.el8.x86_64 19/162 2024-05-08T21:18:57.439 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:18.2.2-1322.gd21e4f96.el8.x86 20/162 2024-05-08T21:18:57.498 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1322.gd21e4f96.el8.x86 20/162 2024-05-08T21:18:57.614 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 10/162 2024-05-08T21:18:57.714 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 21/162 2024-05-08T21:18:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 10/162 2024-05-08T21:18:57.895 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 21/162 2024-05-08T21:18:57.932 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 11/162 2024-05-08T21:18:58.032 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 11/162 2024-05-08T21:18:58.081 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 22/162 2024-05-08T21:18:58.366 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-2:18.2.2-1322.gd21e4f96.el8.x86_64 12/162 2024-05-08T21:18:58.580 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-05-08T21:18:58.671 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-05-08T21:18:58.708 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 22/162 2024-05-08T21:18:58.865 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-05-08T21:18:58.872 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-05-08T21:18:59.096 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-05-08T21:18:59.270 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-05-08T21:18:59.506 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-05-08T21:18:59.707 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-05-08T21:18:59.714 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.15-6.el8.x86_64 24/162 2024-05-08T21:18:59.952 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-05-08T21:19:00.088 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephsqlite-2:18.2.2-1322.gd21e4f96.el8.x86_64 19/162 2024-05-08T21:19:00.199 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1322.gd21e4f96.el8.x86_64 19/162 2024-05-08T21:19:00.355 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libradosstriper1-2:18.2.2-1322.gd21e4f96.el8.x86 20/162 2024-05-08T21:19:00.383 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 24/162 2024-05-08T21:19:00.426 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1322.gd21e4f96.el8.x86 20/162 2024-05-08T21:19:00.621 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 21/162 2024-05-08T21:19:00.662 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/162 2024-05-08T21:19:00.770 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 21/162 2024-05-08T21:19:00.852 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 26/162 2024-05-08T21:19:00.921 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 22/162 2024-05-08T21:19:01.059 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 27/162 2024-05-08T21:19:01.382 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/162 2024-05-08T21:19:01.612 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 22/162 2024-05-08T21:19:01.661 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 29/162 2024-05-08T21:19:01.830 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-05-08T21:19:01.948 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-05-08T21:19:02.161 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:18.2.2-1322.gd21e4f96.el8. 31/162 2024-05-08T21:19:02.307 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:18.2.2-1322.gd21e4f96.el 32/162 2024-05-08T21:19:02.501 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-05-08T21:19:02.567 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:18.2.2-1322.gd21e4f96.el8.x86_6 33/162 2024-05-08T21:19:02.738 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-0.3.15-6.el8.x86_64 24/162 2024-05-08T21:19:03.519 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 24/162 2024-05-08T21:19:03.561 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/162 2024-05-08T21:19:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/162 2024-05-08T21:19:03.913 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 26/162 2024-05-08T21:19:03.919 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/162 2024-05-08T21:19:04.103 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 27/162 2024-05-08T21:19:04.198 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/162 2024-05-08T21:19:04.371 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-05-08T21:19:04.403 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/162 2024-05-08T21:19:04.713 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 29/162 2024-05-08T21:19:04.975 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-05-08T21:19:05.120 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-common-2:18.2.2-1322.gd21e4f96.el8. 31/162 2024-05-08T21:19:05.277 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-argparse-2:18.2.2-1322.gd21e4f96.el 32/162 2024-05-08T21:19:05.553 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-2:18.2.2-1322.gd21e4f96.el8.x86_6 33/162 2024-05-08T21:19:06.148 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-05-08T21:19:06.539 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 38/162 2024-05-08T21:19:06.627 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/162 2024-05-08T21:19:06.774 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-05-08T21:19:06.962 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/162 2024-05-08T21:19:07.015 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-05-08T21:19:07.246 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 41/162 2024-05-08T21:19:07.261 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/162 2024-05-08T21:19:07.453 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-05-08T21:19:07.461 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:18.2.2-1322.gd21e4f96.el8.x86_6 42/162 2024-05-08T21:19:07.679 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libfdisk-2.32.1-44.el8_9.1.x86_64 43/162 2024-05-08T21:19:07.759 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libfdisk-2.32.1-44.el8_9.1.x86_64 43/162 2024-05-08T21:19:07.988 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libmount-2.32.1-44.el8_9.1.x86_64 44/162 2024-05-08T21:19:08.094 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libmount-2.32.1-44.el8_9.1.x86_64 44/162 2024-05-08T21:19:08.342 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libuuid-devel-2.32.1-44.el8_9.1.x86_64 45/162 2024-05-08T21:19:08.614 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-05-08T21:19:08.862 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libsmartcols-2.32.1-44.el8_9.1.x86_64 47/162 2024-05-08T21:19:09.335 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libsmartcols-2.32.1-44.el8_9.1.x86_64 47/162 2024-05-08T21:19:09.445 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-05-08T21:19:09.713 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : util-linux-2.32.1-44.el8_9.1.x86_64 48/162 2024-05-08T21:19:09.740 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: util-linux-2.32.1-44.el8_9.1.x86_64 48/162 2024-05-08T21:19:09.754 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 38/162 2024-05-08T21:19:09.938 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-packaging-16.8-9.el8.noarch 49/162 2024-05-08T21:19:09.991 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-05-08T21:19:10.166 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.7-17.el8.x86_64 50/162 2024-05-08T21:19:10.194 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-05-08T21:19:10.417 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 51/162 2024-05-08T21:19:10.433 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 41/162 2024-05-08T21:19:10.624 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 52/162 2024-05-08T21:19:10.646 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-2:18.2.2-1322.gd21e4f96.el8.x86_6 42/162 2024-05-08T21:19:10.848 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 53/162 2024-05-08T21:19:10.853 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libfdisk-2.32.1-44.el8_9.1.x86_64 43/162 2024-05-08T21:19:10.921 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libfdisk-2.32.1-44.el8_9.1.x86_64 43/162 2024-05-08T21:19:11.082 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 54/162 2024-05-08T21:19:11.187 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libmount-2.32.1-44.el8_9.1.x86_64 44/162 2024-05-08T21:19:11.248 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libmount-2.32.1-44.el8_9.1.x86_64 44/162 2024-05-08T21:19:11.407 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 55/162 2024-05-08T21:19:11.480 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libuuid-devel-2.32.1-44.el8_9.1.x86_64 45/162 2024-05-08T21:19:11.700 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/162 2024-05-08T21:19:11.724 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-05-08T21:19:11.911 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libsmartcols-2.32.1-44.el8_9.1.x86_64 47/162 2024-05-08T21:19:11.975 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/162 2024-05-08T21:19:12.124 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 58/162 2024-05-08T21:19:12.364 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 59/162 2024-05-08T21:19:12.396 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libsmartcols-2.32.1-44.el8_9.1.x86_64 47/162 2024-05-08T21:19:12.670 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : util-linux-2.32.1-44.el8_9.1.x86_64 48/162 2024-05-08T21:19:12.691 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: util-linux-2.32.1-44.el8_9.1.x86_64 48/162 2024-05-08T21:19:12.740 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 60/162 2024-05-08T21:19:13.025 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fuse-2.9.7-17.el8.x86_64 49/162 2024-05-08T21:19:13.098 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 61/162 2024-05-08T21:19:13.230 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 50/162 2024-05-08T21:19:13.435 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 51/162 2024-05-08T21:19:13.439 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 62/162 2024-05-08T21:19:13.608 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 52/162 2024-05-08T21:19:13.679 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 63/162 2024-05-08T21:19:13.826 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 53/162 2024-05-08T21:19:13.930 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-05-08T21:19:14.179 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 54/162 2024-05-08T21:19:14.191 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 65/162 2024-05-08T21:19:14.448 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 55/162 2024-05-08T21:19:14.508 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 66/162 2024-05-08T21:19:14.667 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 56/162 2024-05-08T21:19:14.754 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 67/162 2024-05-08T21:19:14.822 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 57/162 2024-05-08T21:19:14.981 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 68/162 2024-05-08T21:19:15.064 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 58/162 2024-05-08T21:19:15.150 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 69/162 2024-05-08T21:19:15.423 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 70/162 2024-05-08T21:19:15.589 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 59/162 2024-05-08T21:19:15.645 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 71/162 2024-05-08T21:19:15.730 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 71/162 2024-05-08T21:19:15.893 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 60/162 2024-05-08T21:19:16.063 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : jq-1.6-7.el8.x86_64 72/162 2024-05-08T21:19:16.213 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 61/162 2024-05-08T21:19:16.538 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 62/162 2024-05-08T21:19:16.706 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : automake-1.16.1-8.el8.noarch 73/162 2024-05-08T21:19:16.727 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/162 2024-05-08T21:19:16.961 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 64/162 2024-05-08T21:19:16.969 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : uuid-1.6.2-43.el8.x86_64 74/162 2024-05-08T21:19:17.034 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 74/162 2024-05-08T21:19:17.179 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 65/162 2024-05-08T21:19:17.227 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 75/162 2024-05-08T21:19:17.378 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 76/162 2024-05-08T21:19:17.431 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 66/162 2024-05-08T21:19:17.555 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 77/162 2024-05-08T21:19:17.638 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 67/162 2024-05-08T21:19:17.728 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-05-08T21:19:17.859 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 68/162 2024-05-08T21:19:17.931 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-05-08T21:19:18.183 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 69/162 2024-05-08T21:19:18.228 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-05-08T21:19:18.375 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-packaging-16.8-9.el8.noarch 70/162 2024-05-08T21:19:18.443 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-05-08T21:19:18.594 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 71/162 2024-05-08T21:19:18.666 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-05-08T21:19:18.667 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 71/162 2024-05-08T21:19:18.881 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-05-08T21:19:18.963 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : jq-1.6-7.el8.x86_64 72/162 2024-05-08T21:19:19.130 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-05-08T21:19:19.197 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : automake-1.16.1-8.el8.noarch 73/162 2024-05-08T21:19:19.374 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-05-08T21:19:19.385 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : uuid-1.6.2-43.el8.x86_64 74/162 2024-05-08T21:19:19.462 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 74/162 2024-05-08T21:19:19.634 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-05-08T21:19:19.643 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 75/162 2024-05-08T21:19:19.811 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 76/162 2024-05-08T21:19:19.831 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-05-08T21:19:19.980 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 77/162 2024-05-08T21:19:20.033 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-05-08T21:19:20.145 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-05-08T21:19:20.316 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-05-08T21:19:20.367 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-05-08T21:19:20.531 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-05-08T21:19:20.632 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-05-08T21:19:20.739 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-05-08T21:19:20.868 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-05-08T21:19:21.033 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-05-08T21:19:21.065 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-05-08T21:19:21.231 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-05-08T21:19:21.293 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-05-08T21:19:21.465 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-05-08T21:19:21.502 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-05-08T21:19:21.667 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-05-08T21:19:21.736 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-05-08T21:19:21.879 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-05-08T21:19:22.072 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-05-08T21:19:22.242 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-05-08T21:19:22.478 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-05-08T21:19:22.690 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-05-08T21:19:22.881 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-05-08T21:19:22.912 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-05-08T21:19:23.189 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-05-08T21:19:23.273 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-05-08T21:19:23.402 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-05-08T21:19:23.519 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-05-08T21:19:23.586 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-05-08T21:19:23.822 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-05-08T21:19:23.882 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-05-08T21:19:24.186 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-05-08T21:19:24.402 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-05-08T21:19:24.994 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-05-08T21:19:25.193 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-05-08T21:19:25.296 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-05-08T21:19:25.343 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:18.2.2-1322.gd21e4f96.el8.x86_64 103/162 2024-05-08T21:19:25.419 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:18.2.2-1322.gd21e4f96.el8.x86_64 103/162 2024-05-08T21:19:25.531 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-05-08T21:19:25.619 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 104/162 2024-05-08T21:19:25.866 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-05-08T21:19:26.145 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-05-08T21:19:26.347 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-05-08T21:19:27.162 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-05-08T21:19:27.317 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librgw2-2:18.2.2-1322.gd21e4f96.el8.x86_64 103/162 2024-05-08T21:19:27.397 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librgw2-2:18.2.2-1322.gd21e4f96.el8.x86_64 103/162 2024-05-08T21:19:27.710 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 104/162 2024-05-08T21:19:29.960 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 105/162 2024-05-08T21:19:30.209 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 105/162 2024-05-08T21:19:30.803 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 105/162 2024-05-08T21:19:31.128 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 106/162 2024-05-08T21:19:31.247 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 106/162 2024-05-08T21:19:31.443 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:18.2.2-1322.gd21e4f96.el8.x86_64 107/162 2024-05-08T21:19:32.514 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 105/162 2024-05-08T21:19:32.734 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 105/162 2024-05-08T21:19:33.389 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 105/162 2024-05-08T21:19:33.690 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 106/162 2024-05-08T21:19:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 106/162 2024-05-08T21:19:33.933 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-selinux-2:18.2.2-1322.gd21e4f96.el8.x86_64 107/162 2024-05-08T21:19:51.126 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1322.gd21e4f96.el8.x86_64 107/162 2024-05-08T21:19:51.127 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2024-05-08T21:19:51.127 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2024-05-08T21:19:51.127 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /dev 2024-05-08T21:19:51.127 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /run 2024-05-08T21:19:51.127 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2024-05-08T21:19:51.127 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2024-05-08T21:19:51.127 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2024-05-08T21:19:51.127 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2024-05-08T21:19:51.127 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /dev 2024-05-08T21:19:51.127 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:19:51.498 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:18.2.2-1322.gd21e4f96.el8.noarch 108/162 2024-05-08T21:19:53.137 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:18.2.2-1322.gd21e4f96.el8.noarch 108/162 2024-05-08T21:19:53.374 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:18.2.2-1322.gd21e4f96.el8.x86_64 109/162 2024-05-08T21:19:53.594 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:18.2.2-1322.gd21e4f96.el8.x86_64 109/162 2024-05-08T21:19:53.798 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:18.2.2-1322.gd21e4f96.el8.x86_64 110/162 2024-05-08T21:19:54.273 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:18.2.2-1322.gd21e4f96.el8.x86_64 110/162 2024-05-08T21:19:54.472 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:18.2.2-1322.gd21e4f96.el8.x86_64 111/162 2024-05-08T21:19:54.491 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:18.2.2-1322.gd21e4f96.el8.x86_64 111/162 2024-05-08T21:19:56.242 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1322.gd21e4f96.el8.x86_64 107/162 2024-05-08T21:19:56.242 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /sys 2024-05-08T21:19:56.243 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /proc 2024-05-08T21:19:56.243 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /dev 2024-05-08T21:19:56.243 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /run 2024-05-08T21:19:56.243 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /mnt 2024-05-08T21:19:56.243 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /var/tmp 2024-05-08T21:19:56.243 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /home 2024-05-08T21:19:56.243 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /tmp 2024-05-08T21:19:56.243 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /dev 2024-05-08T21:19:56.243 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:19:56.389 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 112/162 2024-05-08T21:19:56.543 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 112/162 2024-05-08T21:19:56.630 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-volume-2:18.2.2-1322.gd21e4f96.el8.noarch 108/162 2024-05-08T21:19:56.695 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1322.gd21e4f96.e 113/162 2024-05-08T21:19:56.924 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1322.gd21e4f96. 114/162 2024-05-08T21:19:57.169 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1322.gd21e4f96.el8.noa 115/162 2024-05-08T21:19:58.203 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1322.gd21e4f96.el8.noa 115/162 2024-05-08T21:19:58.441 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-volume-2:18.2.2-1322.gd21e4f96.el8.noarch 108/162 2024-05-08T21:19:58.481 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1322.gd21e4f96.el8.n 116/162 2024-05-08T21:19:58.686 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-2:18.2.2-1322.gd21e4f96.el8.x86_64 109/162 2024-05-08T21:19:58.928 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-2:18.2.2-1322.gd21e4f96.el8.x86_64 109/162 2024-05-08T21:19:59.087 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-2:18.2.2-1322.gd21e4f96.el8.x86_64 110/162 2024-05-08T21:19:59.449 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1322.gd21e4f96.el8.n 116/162 2024-05-08T21:19:59.614 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-2:18.2.2-1322.gd21e4f96.el8.x86_64 110/162 2024-05-08T21:19:59.655 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1322.gd21 117/162 2024-05-08T21:19:59.749 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1322.gd21 117/162 2024-05-08T21:19:59.787 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-2:18.2.2-1322.gd21e4f96.el8.x86_64 111/162 2024-05-08T21:19:59.809 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-2:18.2.2-1322.gd21e4f96.el8.x86_64 111/162 2024-05-08T21:20:00.161 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1322.gd21e4f96.el 118/162 2024-05-08T21:20:00.373 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 119/162 2024-05-08T21:20:00.416 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 119/162 2024-05-08T21:20:00.605 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:18.2.2-1322.gd21e4f96.el8.noarch 120/162 2024-05-08T21:20:00.618 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1322.gd21e4f96.el8.noarch 120/162 2024-05-08T21:20:01.906 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 112/162 2024-05-08T21:20:01.917 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:18.2.2-1322.gd21e4f96.el8.x86_64 121/162 2024-05-08T21:20:02.131 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 122/162 2024-05-08T21:20:02.166 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 122/162 2024-05-08T21:20:02.205 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 112/162 2024-05-08T21:20:02.320 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1322.gd21e4f96.e 113/162 2024-05-08T21:20:02.388 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1322.gd21e4 123/162 2024-05-08T21:20:02.432 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1322.gd21e4 123/162 2024-05-08T21:20:02.541 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1322.gd21e4f96. 114/162 2024-05-08T21:20:02.613 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 124/162 2024-05-08T21:20:02.802 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1322.gd21e4f96.el8.noa 115/162 2024-05-08T21:20:02.961 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 124/162 2024-05-08T21:20:03.096 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 125/162 2024-05-08T21:20:03.943 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1322.gd21e4f96.el8.noa 115/162 2024-05-08T21:20:04.168 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1322.gd21e4f96.el8.n 116/162 2024-05-08T21:20:05.243 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1322.gd21e4f96.el8.n 116/162 2024-05-08T21:20:05.419 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1322.gd21 117/162 2024-05-08T21:20:05.527 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1322.gd21 117/162 2024-05-08T21:20:05.913 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1322.gd21e4f96.el 118/162 2024-05-08T21:20:06.076 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 119/162 2024-05-08T21:20:06.120 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 119/162 2024-05-08T21:20:06.303 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-rook-2:18.2.2-1322.gd21e4f96.el8.noarch 120/162 2024-05-08T21:20:06.318 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1322.gd21e4f96.el8.noarch 120/162 2024-05-08T21:20:07.562 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 125/162 2024-05-08T21:20:07.713 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-2:18.2.2-1322.gd21e4f96.el8.x86_64 121/162 2024-05-08T21:20:07.901 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 122/162 2024-05-08T21:20:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 122/162 2024-05-08T21:20:08.066 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:18.2.2-1322.gd21e4f96.el8.x86_64 126/162 2024-05-08T21:20:08.170 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1322.gd21e4 123/162 2024-05-08T21:20:08.219 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1322.gd21e4 123/162 2024-05-08T21:20:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephfs-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 124/162 2024-05-08T21:20:08.431 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 127/162 2024-05-08T21:20:08.670 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libtool-2.4.6-25.el8.x86_64 128/162 2024-05-08T21:20:08.775 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 124/162 2024-05-08T21:20:08.919 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 125/162 2024-05-08T21:20:09.381 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 128/162 2024-05-08T21:20:09.675 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 129/162 2024-05-08T21:20:09.961 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 130/162 2024-05-08T21:20:10.193 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 131/162 2024-05-08T21:20:10.448 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : util-linux-user-2.32.1-44.el8_9.1.x86_64 132/162 2024-05-08T21:20:10.672 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-05-08T21:20:10.888 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:18.2.2-1322.gd21e4f96.el8.x86_ 134/162 2024-05-08T21:20:13.322 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-05-08T21:20:13.628 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-05-08T21:20:13.667 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-05-08T21:20:13.853 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 125/162 2024-05-08T21:20:14.101 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-05-08T21:20:14.301 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-2:18.2.2-1322.gd21e4f96.el8.x86_64 126/162 2024-05-08T21:20:14.502 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-05-08T21:20:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 127/162 2024-05-08T21:20:14.790 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-05-08T21:20:14.793 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libtool-2.4.6-25.el8.x86_64 128/162 2024-05-08T21:20:15.028 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-top-2:18.2.2-1322.gd21e4f96.el8.noarch 139/162 2024-05-08T21:20:15.241 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 140/162 2024-05-08T21:20:15.499 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 141/162 2024-05-08T21:20:15.604 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 128/162 2024-05-08T21:20:15.788 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-05-08T21:20:15.874 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 129/162 2024-05-08T21:20:15.986 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-15.e 143/162 2024-05-08T21:20:15.998 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-15.e 143/162 2024-05-08T21:20:16.160 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 130/162 2024-05-08T21:20:16.241 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 144/162 2024-05-08T21:20:16.335 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 131/162 2024-05-08T21:20:16.477 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flex-2.6.1-9.el8.x86_64 145/162 2024-05-08T21:20:16.548 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : util-linux-user-2.32.1-44.el8_9.1.x86_64 132/162 2024-05-08T21:20:16.559 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 145/162 2024-05-08T21:20:16.780 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-05-08T21:20:16.852 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : bison-3.0.4-10.el8.x86_64 146/162 2024-05-08T21:20:16.867 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 146/162 2024-05-08T21:20:17.059 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-05-08T21:20:17.085 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs-devel-2:18.2.2-1322.gd21e4f96.el8.x86_ 134/162 2024-05-08T21:20:17.280 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-05-08T21:20:17.432 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-05-08T21:20:17.432 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-05-08T21:20:17.781 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-05-08T21:20:17.781 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-05-08T21:20:18.006 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-05-08T21:20:18.007 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-05-08T21:20:18.267 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-05-08T21:20:18.521 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-05-08T21:20:18.722 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-05-08T21:20:18.979 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-05-08T21:20:18.980 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-05-08T21:20:19.230 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-05-08T21:20:19.230 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-05-08T21:20:19.463 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-05-08T21:20:19.463 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-05-08T21:20:19.721 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-05-08T21:20:19.721 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-05-08T21:20:19.724 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-05-08T21:20:19.980 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-05-08T21:20:19.980 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-05-08T21:20:19.984 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-05-08T21:20:20.026 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-05-08T21:20:20.263 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-05-08T21:20:20.413 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-05-08T21:20:20.413 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-05-08T21:20:20.461 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-05-08T21:20:20.821 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-05-08T21:20:21.031 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-05-08T21:20:21.210 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephfs-top-2:18.2.2-1322.gd21e4f96.el8.noarch 139/162 2024-05-08T21:20:21.436 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 140/162 2024-05-08T21:20:21.455 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-05-08T21:20:21.455 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/162 2024-05-08T21:20:21.455 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 2/162 2024-05-08T21:20:21.455 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 3/162 2024-05-08T21:20:21.455 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 4/162 2024-05-08T21:20:21.455 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1322.gd21e4 5/162 2024-05-08T21:20:21.455 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:18.2.2-1322.gd21e4f96.el8.x86_64 6/162 2024-05-08T21:20:21.455 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 7/162 2024-05-08T21:20:21.455 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:18.2.2-1322.gd21e4f96.el8.x86_64 8/162 2024-05-08T21:20:21.455 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:18.2.2-1322.gd21e4f96.el8.x86_64 9/162 2024-05-08T21:20:21.455 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 10/162 2024-05-08T21:20:21.456 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:18.2.2-1322.gd21e4f96.el8.x86_64 11/162 2024-05-08T21:20:21.456 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:18.2.2-1322.gd21e4f96.el8.x86_64 12/162 2024-05-08T21:20:21.456 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 13/162 2024-05-08T21:20:21.456 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:18.2.2-1322.gd21e4f96.el8.x86_ 14/162 2024-05-08T21:20:21.456 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 15/162 2024-05-08T21:20:21.456 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:18.2.2-1322.gd21e4f96.el8.x86_64 16/162 2024-05-08T21:20:21.456 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:18.2.2-1322.gd21e4f96.el8.x86_6 17/162 2024-05-08T21:20:21.456 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:18.2.2-1322.gd21e4f96.el8.x86 18/162 2024-05-08T21:20:21.456 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:18.2.2-1322.gd21e4f96.el8.x86_64 19/162 2024-05-08T21:20:21.456 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1322.gd21e4f96.el 20/162 2024-05-08T21:20:21.456 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:18.2.2-1322.gd21e4f96.el8. 21/162 2024-05-08T21:20:21.456 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:18.2.2-1322.gd21e4f96.el8.x86_6 22/162 2024-05-08T21:20:21.456 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:18.2.2-1322.gd21e4f96.el8.x86_64 23/162 2024-05-08T21:20:21.456 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 24/162 2024-05-08T21:20:21.457 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 25/162 2024-05-08T21:20:21.457 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 26/162 2024-05-08T21:20:21.457 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 27/162 2024-05-08T21:20:21.457 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 28/162 2024-05-08T21:20:21.457 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1322.gd21e4f96. 29/162 2024-05-08T21:20:21.457 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1322.gd21e4f96.el8.noa 30/162 2024-05-08T21:20:21.457 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1322.gd21e4f96.el8.n 31/162 2024-05-08T21:20:21.457 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1322.gd21 32/162 2024-05-08T21:20:21.457 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1322.gd21e4f96.el 33/162 2024-05-08T21:20:21.457 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1322.gd21e4f96.el8.noarch 34/162 2024-05-08T21:20:21.457 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1322.gd21e4f96.e 35/162 2024-05-08T21:20:21.457 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:18.2.2-1322.gd21e4f96.el8.noarch 36/162 2024-05-08T21:20:21.457 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 37/162 2024-05-08T21:20:21.457 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-top-2:18.2.2-1322.gd21e4f96.el8.noarch 38/162 2024-05-08T21:20:21.458 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-05-08T21:20:21.458 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-05-08T21:20:21.458 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-05-08T21:20:21.458 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-05-08T21:20:21.458 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-05-08T21:20:21.458 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-05-08T21:20:21.458 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-05-08T21:20:21.458 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-05-08T21:20:21.458 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-05-08T21:20:21.458 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-05-08T21:20:21.458 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-05-08T21:20:21.458 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-05-08T21:20:21.459 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-05-08T21:20:21.459 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-05-08T21:20:21.459 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-05-08T21:20:21.459 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-05-08T21:20:21.459 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-05-08T21:20:21.459 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-05-08T21:20:21.459 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-05-08T21:20:21.459 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-05-08T21:20:21.459 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-05-08T21:20:21.459 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-05-08T21:20:21.459 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-05-08T21:20:21.459 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-05-08T21:20:21.459 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-05-08T21:20:21.459 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-05-08T21:20:21.460 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-05-08T21:20:21.460 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-05-08T21:20:21.460 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-05-08T21:20:21.460 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-05-08T21:20:21.460 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-05-08T21:20:21.460 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-05-08T21:20:21.460 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-05-08T21:20:21.460 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 72/162 2024-05-08T21:20:21.460 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/162 2024-05-08T21:20:21.460 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 74/162 2024-05-08T21:20:21.460 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 75/162 2024-05-08T21:20:21.461 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 76/162 2024-05-08T21:20:21.461 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/162 2024-05-08T21:20:21.461 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/162 2024-05-08T21:20:21.461 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flex-2.6.1-9.el8.x86_64 79/162 2024-05-08T21:20:21.461 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 80/162 2024-05-08T21:20:21.461 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 81/162 2024-05-08T21:20:21.461 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 82/162 2024-05-08T21:20:21.461 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 83/162 2024-05-08T21:20:21.461 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 84/162 2024-05-08T21:20:21.461 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 85/162 2024-05-08T21:20:21.461 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 86/162 2024-05-08T21:20:21.461 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/162 2024-05-08T21:20:21.461 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 88/162 2024-05-08T21:20:21.461 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/162 2024-05-08T21:20:21.462 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 90/162 2024-05-08T21:20:21.462 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 91/162 2024-05-08T21:20:21.462 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 92/162 2024-05-08T21:20:21.462 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 93/162 2024-05-08T21:20:21.462 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : automake-1.16.1-8.el8.noarch 94/162 2024-05-08T21:20:21.462 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpq-13.11-1.el8.x86_64 95/162 2024-05-08T21:20:21.462 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : jq-1.6-7.el8.x86_64 96/162 2024-05-08T21:20:21.462 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 97/162 2024-05-08T21:20:21.462 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 98/162 2024-05-08T21:20:21.462 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 99/162 2024-05-08T21:20:21.462 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 100/162 2024-05-08T21:20:21.463 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 101/162 2024-05-08T21:20:21.463 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 102/162 2024-05-08T21:20:21.463 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 103/162 2024-05-08T21:20:21.463 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 104/162 2024-05-08T21:20:21.463 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 105/162 2024-05-08T21:20:21.463 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 106/162 2024-05-08T21:20:21.463 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 107/162 2024-05-08T21:20:21.463 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 108/162 2024-05-08T21:20:21.463 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 109/162 2024-05-08T21:20:21.463 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 110/162 2024-05-08T21:20:21.463 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/162 2024-05-08T21:20:21.463 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 112/162 2024-05-08T21:20:21.463 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 113/162 2024-05-08T21:20:21.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 114/162 2024-05-08T21:20:21.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 115/162 2024-05-08T21:20:21.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 116/162 2024-05-08T21:20:21.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 117/162 2024-05-08T21:20:21.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 118/162 2024-05-08T21:20:21.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 119/162 2024-05-08T21:20:21.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 120/162 2024-05-08T21:20:21.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 121/162 2024-05-08T21:20:21.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 122/162 2024-05-08T21:20:21.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 123/162 2024-05-08T21:20:21.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 124/162 2024-05-08T21:20:21.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 125/162 2024-05-08T21:20:21.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 126/162 2024-05-08T21:20:21.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 127/162 2024-05-08T21:20:21.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-15.e 128/162 2024-05-08T21:20:21.465 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-devel-2.32.1-44.el8_9.1.x86_64 129/162 2024-05-08T21:20:21.465 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 130/162 2024-05-08T21:20:21.465 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 131/162 2024-05-08T21:20:21.465 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-05-08T21:20:21.465 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 133/162 2024-05-08T21:20:21.465 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-05-08T21:20:21.465 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-05-08T21:20:21.465 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-05-08T21:20:21.465 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-05-08T21:20:21.465 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-05-08T21:20:21.465 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 139/162 2024-05-08T21:20:21.465 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 140/162 2024-05-08T21:20:21.465 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 141/162 2024-05-08T21:20:21.465 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 142/162 2024-05-08T21:20:21.466 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 143/162 2024-05-08T21:20:21.466 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 144/162 2024-05-08T21:20:21.466 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 145/162 2024-05-08T21:20:21.466 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 146/162 2024-05-08T21:20:21.466 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 147/162 2024-05-08T21:20:21.466 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 148/162 2024-05-08T21:20:21.466 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libblkid-2.32.1-44.el8_9.1.x86_64 149/162 2024-05-08T21:20:21.466 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 150/162 2024-05-08T21:20:21.466 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libfdisk-2.32.1-44.el8_9.1.x86_64 151/162 2024-05-08T21:20:21.466 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 152/162 2024-05-08T21:20:21.466 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libmount-2.32.1-44.el8_9.1.x86_64 153/162 2024-05-08T21:20:21.466 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 154/162 2024-05-08T21:20:21.466 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libsmartcols-2.32.1-44.el8_9.1.x86_64 155/162 2024-05-08T21:20:21.466 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 156/162 2024-05-08T21:20:21.467 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-2.32.1-44.el8_9.1.x86_64 157/162 2024-05-08T21:20:21.467 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 158/162 2024-05-08T21:20:21.467 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : util-linux-2.32.1-44.el8_9.1.x86_64 159/162 2024-05-08T21:20:21.467 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 160/162 2024-05-08T21:20:21.467 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : util-linux-user-2.32.1-44.el8_9.1.x86_64 161/162 2024-05-08T21:20:21.714 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 141/162 2024-05-08T21:20:22.011 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-05-08T21:20:22.167 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-15.e 143/162 2024-05-08T21:20:22.178 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-15.e 143/162 2024-05-08T21:20:22.383 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 144/162 2024-05-08T21:20:22.583 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flex-2.6.1-9.el8.x86_64 145/162 2024-05-08T21:20:22.670 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 145/162 2024-05-08T21:20:22.900 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : bison-3.0.4-10.el8.x86_64 146/162 2024-05-08T21:20:22.915 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 146/162 2024-05-08T21:20:23.128 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-05-08T21:20:23.311 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-05-08T21:20:23.463 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-05-08T21:20:23.463 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-05-08T21:20:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-05-08T21:20:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-05-08T21:20:23.996 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-05-08T21:20:23.996 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-05-08T21:20:24.266 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-05-08T21:20:24.502 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-05-08T21:20:24.672 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-05-08T21:20:24.927 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-05-08T21:20:24.927 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-05-08T21:20:25.144 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-05-08T21:20:25.145 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-05-08T21:20:25.369 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-05-08T21:20:25.369 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-05-08T21:20:25.611 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-05-08T21:20:25.611 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-05-08T21:20:25.827 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-05-08T21:20:25.827 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-05-08T21:20:26.086 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-05-08T21:20:26.236 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-05-08T21:20:26.236 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-05-08T21:20:27.229 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-05-08T21:20:27.230 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/162 2024-05-08T21:20:27.230 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 2/162 2024-05-08T21:20:27.230 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 3/162 2024-05-08T21:20:27.230 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 4/162 2024-05-08T21:20:27.230 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1322.gd21e4 5/162 2024-05-08T21:20:27.230 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-2:18.2.2-1322.gd21e4f96.el8.x86_64 6/162 2024-05-08T21:20:27.230 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 7/162 2024-05-08T21:20:27.230 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-2:18.2.2-1322.gd21e4f96.el8.x86_64 8/162 2024-05-08T21:20:27.230 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-2:18.2.2-1322.gd21e4f96.el8.x86_64 9/162 2024-05-08T21:20:27.230 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 10/162 2024-05-08T21:20:27.230 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-selinux-2:18.2.2-1322.gd21e4f96.el8.x86_64 11/162 2024-05-08T21:20:27.231 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-2:18.2.2-1322.gd21e4f96.el8.x86_64 12/162 2024-05-08T21:20:27.231 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephfs-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 13/162 2024-05-08T21:20:27.231 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs-devel-2:18.2.2-1322.gd21e4f96.el8.x86_ 14/162 2024-05-08T21:20:27.231 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 15/162 2024-05-08T21:20:27.231 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephsqlite-2:18.2.2-1322.gd21e4f96.el8.x86_64 16/162 2024-05-08T21:20:27.231 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-2:18.2.2-1322.gd21e4f96.el8.x86_6 17/162 2024-05-08T21:20:27.231 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libradosstriper1-2:18.2.2-1322.gd21e4f96.el8.x86 18/162 2024-05-08T21:20:27.231 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librgw2-2:18.2.2-1322.gd21e4f96.el8.x86_64 19/162 2024-05-08T21:20:27.231 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1322.gd21e4f96.el 20/162 2024-05-08T21:20:27.231 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-common-2:18.2.2-1322.gd21e4f96.el8. 21/162 2024-05-08T21:20:27.231 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-2:18.2.2-1322.gd21e4f96.el8.x86_6 22/162 2024-05-08T21:20:27.231 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-2:18.2.2-1322.gd21e4f96.el8.x86_64 23/162 2024-05-08T21:20:27.231 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 24/162 2024-05-08T21:20:27.231 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 25/162 2024-05-08T21:20:27.231 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 26/162 2024-05-08T21:20:27.232 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 27/162 2024-05-08T21:20:27.232 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 28/162 2024-05-08T21:20:27.232 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1322.gd21e4f96. 29/162 2024-05-08T21:20:27.232 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1322.gd21e4f96.el8.noa 30/162 2024-05-08T21:20:27.232 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1322.gd21e4f96.el8.n 31/162 2024-05-08T21:20:27.232 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1322.gd21 32/162 2024-05-08T21:20:27.232 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1322.gd21e4f96.el 33/162 2024-05-08T21:20:27.232 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1322.gd21e4f96.el8.noarch 34/162 2024-05-08T21:20:27.232 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1322.gd21e4f96.e 35/162 2024-05-08T21:20:27.232 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-volume-2:18.2.2-1322.gd21e4f96.el8.noarch 36/162 2024-05-08T21:20:27.232 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 37/162 2024-05-08T21:20:27.232 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephfs-top-2:18.2.2-1322.gd21e4f96.el8.noarch 38/162 2024-05-08T21:20:27.232 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-05-08T21:20:27.232 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-05-08T21:20:27.233 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-05-08T21:20:27.233 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-05-08T21:20:27.233 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-05-08T21:20:27.233 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-05-08T21:20:27.233 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-05-08T21:20:27.233 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-05-08T21:20:27.233 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-05-08T21:20:27.233 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-05-08T21:20:27.233 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-05-08T21:20:27.233 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-05-08T21:20:27.233 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-05-08T21:20:27.233 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-05-08T21:20:27.234 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-05-08T21:20:27.234 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-05-08T21:20:27.234 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-05-08T21:20:27.234 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-05-08T21:20:27.234 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-05-08T21:20:27.234 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-05-08T21:20:27.234 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-05-08T21:20:27.234 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-05-08T21:20:27.234 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-05-08T21:20:27.234 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-05-08T21:20:27.234 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-05-08T21:20:27.234 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-05-08T21:20:27.235 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-05-08T21:20:27.235 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-05-08T21:20:27.235 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-05-08T21:20:27.235 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-05-08T21:20:27.235 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-05-08T21:20:27.235 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-05-08T21:20:27.235 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-05-08T21:20:27.235 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 72/162 2024-05-08T21:20:27.235 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/162 2024-05-08T21:20:27.235 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 74/162 2024-05-08T21:20:27.235 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 75/162 2024-05-08T21:20:27.235 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 76/162 2024-05-08T21:20:27.235 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/162 2024-05-08T21:20:27.235 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/162 2024-05-08T21:20:27.236 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flex-2.6.1-9.el8.x86_64 79/162 2024-05-08T21:20:27.236 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 80/162 2024-05-08T21:20:27.236 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 81/162 2024-05-08T21:20:27.236 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 82/162 2024-05-08T21:20:27.236 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 83/162 2024-05-08T21:20:27.236 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 84/162 2024-05-08T21:20:27.236 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 85/162 2024-05-08T21:20:27.236 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 86/162 2024-05-08T21:20:27.236 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/162 2024-05-08T21:20:27.236 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 88/162 2024-05-08T21:20:27.236 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/162 2024-05-08T21:20:27.236 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 90/162 2024-05-08T21:20:27.237 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 91/162 2024-05-08T21:20:27.237 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 92/162 2024-05-08T21:20:27.237 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 93/162 2024-05-08T21:20:27.237 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : automake-1.16.1-8.el8.noarch 94/162 2024-05-08T21:20:27.237 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libpq-13.11-1.el8.x86_64 95/162 2024-05-08T21:20:27.237 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : jq-1.6-7.el8.x86_64 96/162 2024-05-08T21:20:27.237 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 97/162 2024-05-08T21:20:27.237 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 98/162 2024-05-08T21:20:27.237 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 99/162 2024-05-08T21:20:27.237 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 100/162 2024-05-08T21:20:27.237 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 101/162 2024-05-08T21:20:27.237 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 102/162 2024-05-08T21:20:27.237 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 103/162 2024-05-08T21:20:27.237 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 104/162 2024-05-08T21:20:27.238 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/162 2024-05-08T21:20:27.238 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 106/162 2024-05-08T21:20:27.238 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 107/162 2024-05-08T21:20:27.238 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 108/162 2024-05-08T21:20:27.238 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 109/162 2024-05-08T21:20:27.238 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 110/162 2024-05-08T21:20:27.238 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 111/162 2024-05-08T21:20:27.238 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 112/162 2024-05-08T21:20:27.238 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 113/162 2024-05-08T21:20:27.238 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 114/162 2024-05-08T21:20:27.238 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 115/162 2024-05-08T21:20:27.238 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 116/162 2024-05-08T21:20:27.239 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 117/162 2024-05-08T21:20:27.239 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 118/162 2024-05-08T21:20:27.239 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 119/162 2024-05-08T21:20:27.239 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 120/162 2024-05-08T21:20:27.239 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 121/162 2024-05-08T21:20:27.239 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 122/162 2024-05-08T21:20:27.239 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 123/162 2024-05-08T21:20:27.239 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 124/162 2024-05-08T21:20:27.239 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 125/162 2024-05-08T21:20:27.239 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 126/162 2024-05-08T21:20:27.239 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 127/162 2024-05-08T21:20:27.239 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 128/162 2024-05-08T21:20:27.239 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-15.e 129/162 2024-05-08T21:20:27.239 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libuuid-devel-2.32.1-44.el8_9.1.x86_64 130/162 2024-05-08T21:20:27.239 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 131/162 2024-05-08T21:20:27.240 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-05-08T21:20:27.240 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 133/162 2024-05-08T21:20:27.240 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-05-08T21:20:27.240 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-05-08T21:20:27.240 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-05-08T21:20:27.240 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-05-08T21:20:27.240 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-05-08T21:20:27.240 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 139/162 2024-05-08T21:20:27.240 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 140/162 2024-05-08T21:20:27.240 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 141/162 2024-05-08T21:20:27.240 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 142/162 2024-05-08T21:20:27.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 143/162 2024-05-08T21:20:27.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 144/162 2024-05-08T21:20:27.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 145/162 2024-05-08T21:20:27.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 146/162 2024-05-08T21:20:27.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 147/162 2024-05-08T21:20:27.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 148/162 2024-05-08T21:20:27.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libblkid-2.32.1-44.el8_9.1.x86_64 149/162 2024-05-08T21:20:27.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 150/162 2024-05-08T21:20:27.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libfdisk-2.32.1-44.el8_9.1.x86_64 151/162 2024-05-08T21:20:27.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 152/162 2024-05-08T21:20:27.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libmount-2.32.1-44.el8_9.1.x86_64 153/162 2024-05-08T21:20:27.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 154/162 2024-05-08T21:20:27.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libsmartcols-2.32.1-44.el8_9.1.x86_64 155/162 2024-05-08T21:20:27.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 156/162 2024-05-08T21:20:27.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libuuid-2.32.1-44.el8_9.1.x86_64 157/162 2024-05-08T21:20:27.242 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 158/162 2024-05-08T21:20:27.242 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : util-linux-2.32.1-44.el8_9.1.x86_64 159/162 2024-05-08T21:20:27.242 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 160/162 2024-05-08T21:20:27.242 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : util-linux-user-2.32.1-44.el8_9.1.x86_64 161/162 2024-05-08T21:20:34.512 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 162/162 2024-05-08T21:20:34.512 INFO:teuthology.orchestra.run.smithi186.stdout:Installed products updated. 2024-05-08T21:20:35.144 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:35.144 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2024-05-08T21:20:35.144 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-05-08T21:20:35.145 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-05-08T21:20:35.145 INFO:teuthology.orchestra.run.smithi186.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-05-08T21:20:35.145 INFO:teuthology.orchestra.run.smithi186.stdout: libblkid-2.32.1-44.el8_9.1.x86_64 2024-05-08T21:20:35.145 INFO:teuthology.orchestra.run.smithi186.stdout: libfdisk-2.32.1-44.el8_9.1.x86_64 2024-05-08T21:20:35.145 INFO:teuthology.orchestra.run.smithi186.stdout: libmount-2.32.1-44.el8_9.1.x86_64 2024-05-08T21:20:35.145 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.145 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.145 INFO:teuthology.orchestra.run.smithi186.stdout: libsmartcols-2.32.1-44.el8_9.1.x86_64 2024-05-08T21:20:35.145 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-2.32.1-44.el8_9.1.x86_64 2024-05-08T21:20:35.145 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-05-08T21:20:35.145 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-05-08T21:20:35.145 INFO:teuthology.orchestra.run.smithi186.stdout: util-linux-2.32.1-44.el8_9.1.x86_64 2024-05-08T21:20:35.145 INFO:teuthology.orchestra.run.smithi186.stdout: util-linux-user-2.32.1-44.el8_9.1.x86_64 2024-05-08T21:20:35.146 INFO:teuthology.orchestra.run.smithi186.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-05-08T21:20:35.146 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-05-08T21:20:35.146 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-08T21:20:35.146 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-initscripts-updown-1:1.40.16-15.el8_9.noarch 2024-05-08T21:20:35.146 INFO:teuthology.orchestra.run.smithi186.stdout: automake-1.16.1-8.el8.noarch 2024-05-08T21:20:35.146 INFO:teuthology.orchestra.run.smithi186.stdout: bison-3.0.4-10.el8.x86_64 2024-05-08T21:20:35.146 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.146 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.146 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.146 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.146 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:35.146 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.146 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.147 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.147 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:35.147 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:35.147 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:35.147 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:35.147 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:35.147 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.147 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.147 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:35.147 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.147 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.147 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.147 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:35.148 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:35.148 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.148 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:35.148 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-05-08T21:20:35.148 INFO:teuthology.orchestra.run.smithi186.stdout: flex-2.6.1-9.el8.x86_64 2024-05-08T21:20:35.148 INFO:teuthology.orchestra.run.smithi186.stdout: fuse-2.9.7-17.el8.x86_64 2024-05-08T21:20:35.148 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-05-08T21:20:35.148 INFO:teuthology.orchestra.run.smithi186.stdout: jq-1.6-7.el8.x86_64 2024-05-08T21:20:35.148 INFO:teuthology.orchestra.run.smithi186.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-05-08T21:20:35.148 INFO:teuthology.orchestra.run.smithi186.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-05-08T21:20:35.148 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-05-08T21:20:35.148 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.148 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.148 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.149 INFO:teuthology.orchestra.run.smithi186.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-05-08T21:20:35.149 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-05-08T21:20:35.149 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.2-3.el8.x86_64 2024-05-08T21:20:35.149 INFO:teuthology.orchestra.run.smithi186.stdout: libpq-13.11-1.el8.x86_64 2024-05-08T21:20:35.149 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-05-08T21:20:35.149 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-05-08T21:20:35.149 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.149 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.149 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-05-08T21:20:35.149 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.149 INFO:teuthology.orchestra.run.smithi186.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-05-08T21:20:35.149 INFO:teuthology.orchestra.run.smithi186.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-05-08T21:20:35.149 INFO:teuthology.orchestra.run.smithi186.stdout: libtool-2.4.6-25.el8.x86_64 2024-05-08T21:20:35.149 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.3.1-3.el8.x86_64 2024-05-08T21:20:35.150 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel-2.32.1-44.el8_9.1.x86_64 2024-05-08T21:20:35.150 INFO:teuthology.orchestra.run.smithi186.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-05-08T21:20:35.150 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-05-08T21:20:35.150 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-05-08T21:20:35.150 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-05-08T21:20:35.150 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.15-6.el8.x86_64 2024-05-08T21:20:35.150 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-05-08T21:20:35.150 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-05-08T21:20:35.150 INFO:teuthology.orchestra.run.smithi186.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-05-08T21:20:35.150 INFO:teuthology.orchestra.run.smithi186.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-05-08T21:20:35.150 INFO:teuthology.orchestra.run.smithi186.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-05-08T21:20:35.150 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-05-08T21:20:35.150 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-05-08T21:20:35.150 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-05-08T21:20:35.151 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-05-08T21:20:35.151 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.5.1-7.el8.noarch 2024-05-08T21:20:35.151 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-05-08T21:20:35.151 INFO:teuthology.orchestra.run.smithi186.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-05-08T21:20:35.151 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-05-08T21:20:35.151 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.151 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.151 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.151 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-05-08T21:20:35.151 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-05-08T21:20:35.151 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-05-08T21:20:35.151 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-05-08T21:20:35.151 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-05-08T21:20:35.151 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-05-08T21:20:35.151 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-6.2-6.el8.noarch 2024-05-08T21:20:35.152 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-05-08T21:20:35.152 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-05-08T21:20:35.152 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-05-08T21:20:35.152 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-05-08T21:20:35.152 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-05-08T21:20:35.152 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-05-08T21:20:35.152 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-05-08T21:20:35.152 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.0.6-14.el8.noarch 2024-05-08T21:20:35.152 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-05-08T21:20:35.152 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-05-08T21:20:35.152 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-05-08T21:20:35.152 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-05-08T21:20:35.152 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-05-08T21:20:35.152 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-05-08T21:20:35.153 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging-16.8-9.el8.noarch 2024-05-08T21:20:35.153 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-05-08T21:20:35.153 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-2.6-1.el8.noarch 2024-05-08T21:20:35.153 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-05-08T21:20:35.153 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-05-08T21:20:35.153 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-05-08T21:20:35.153 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-05-08T21:20:35.153 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.14-14.el8.noarch 2024-05-08T21:20:35.153 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2017.2-11.el8.noarch 2024-05-08T21:20:35.153 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.153 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.153 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-05-08T21:20:35.153 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-05-08T21:20:35.153 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.153 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.4.1-12.el8.noarch 2024-05-08T21:20:35.154 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el8.noarch 2024-05-08T21:20:35.154 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-05-08T21:20:35.154 INFO:teuthology.orchestra.run.smithi186.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-05-08T21:20:35.154 INFO:teuthology.orchestra.run.smithi186.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-05-08T21:20:35.154 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-05-08T21:20:35.154 INFO:teuthology.orchestra.run.smithi186.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-05-08T21:20:35.154 INFO:teuthology.orchestra.run.smithi186.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-05-08T21:20:35.154 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-05-08T21:20:35.154 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-05-08T21:20:35.154 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-05-08T21:20:35.154 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-05-08T21:20:35.154 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-05-08T21:20:35.154 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.154 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.154 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:35.155 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-1.el8.x86_64 2024-05-08T21:20:35.155 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.13.0-2.el8.x86_64 2024-05-08T21:20:35.155 INFO:teuthology.orchestra.run.smithi186.stdout: uuid-1.6.2-43.el8.x86_64 2024-05-08T21:20:35.155 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-05-08T21:20:35.155 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-05-08T21:20:35.155 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-05-08T21:20:35.155 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:35.156 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-08T21:20:37.753 DEBUG:teuthology.parallel:result is None 2024-05-08T21:20:38.730 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 162/162 2024-05-08T21:20:38.730 INFO:teuthology.orchestra.run.smithi003.stdout:Installed products updated. 2024-05-08T21:20:39.525 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:39.525 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2024-05-08T21:20:39.525 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-05-08T21:20:39.525 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-05-08T21:20:39.525 INFO:teuthology.orchestra.run.smithi003.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-05-08T21:20:39.525 INFO:teuthology.orchestra.run.smithi003.stdout: libblkid-2.32.1-44.el8_9.1.x86_64 2024-05-08T21:20:39.526 INFO:teuthology.orchestra.run.smithi003.stdout: libfdisk-2.32.1-44.el8_9.1.x86_64 2024-05-08T21:20:39.526 INFO:teuthology.orchestra.run.smithi003.stdout: libmount-2.32.1-44.el8_9.1.x86_64 2024-05-08T21:20:39.526 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.526 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.526 INFO:teuthology.orchestra.run.smithi003.stdout: libsmartcols-2.32.1-44.el8_9.1.x86_64 2024-05-08T21:20:39.526 INFO:teuthology.orchestra.run.smithi003.stdout: libuuid-2.32.1-44.el8_9.1.x86_64 2024-05-08T21:20:39.526 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-05-08T21:20:39.526 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-05-08T21:20:39.526 INFO:teuthology.orchestra.run.smithi003.stdout: util-linux-2.32.1-44.el8_9.1.x86_64 2024-05-08T21:20:39.526 INFO:teuthology.orchestra.run.smithi003.stdout: util-linux-user-2.32.1-44.el8_9.1.x86_64 2024-05-08T21:20:39.526 INFO:teuthology.orchestra.run.smithi003.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-05-08T21:20:39.526 INFO:teuthology.orchestra.run.smithi003.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-05-08T21:20:39.526 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-05-08T21:20:39.527 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-initscripts-updown-1:1.40.16-15.el8_9.noarch 2024-05-08T21:20:39.527 INFO:teuthology.orchestra.run.smithi003.stdout: automake-1.16.1-8.el8.noarch 2024-05-08T21:20:39.527 INFO:teuthology.orchestra.run.smithi003.stdout: bison-3.0.4-10.el8.x86_64 2024-05-08T21:20:39.527 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.527 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.527 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.527 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.527 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:39.527 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.527 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.527 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.527 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:39.527 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:39.528 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:39.528 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:39.528 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:39.528 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.528 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.528 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:39.528 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.528 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.528 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.528 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:39.528 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:39.528 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.528 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-top-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T21:20:39.529 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-05-08T21:20:39.529 INFO:teuthology.orchestra.run.smithi003.stdout: flex-2.6.1-9.el8.x86_64 2024-05-08T21:20:39.529 INFO:teuthology.orchestra.run.smithi003.stdout: fuse-2.9.7-17.el8.x86_64 2024-05-08T21:20:39.529 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-05-08T21:20:39.529 INFO:teuthology.orchestra.run.smithi003.stdout: jq-1.6-7.el8.x86_64 2024-05-08T21:20:39.529 INFO:teuthology.orchestra.run.smithi003.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-05-08T21:20:39.529 INFO:teuthology.orchestra.run.smithi003.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-05-08T21:20:39.529 INFO:teuthology.orchestra.run.smithi003.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-05-08T21:20:39.529 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.529 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.529 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.529 INFO:teuthology.orchestra.run.smithi003.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-05-08T21:20:39.529 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-05-08T21:20:39.529 INFO:teuthology.orchestra.run.smithi003.stdout: liboath-2.6.2-3.el8.x86_64 2024-05-08T21:20:39.530 INFO:teuthology.orchestra.run.smithi003.stdout: libpq-13.11-1.el8.x86_64 2024-05-08T21:20:39.530 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-05-08T21:20:39.530 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-05-08T21:20:39.530 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.530 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.530 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-05-08T21:20:39.530 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.530 INFO:teuthology.orchestra.run.smithi003.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-05-08T21:20:39.530 INFO:teuthology.orchestra.run.smithi003.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-05-08T21:20:39.530 INFO:teuthology.orchestra.run.smithi003.stdout: libtool-2.4.6-25.el8.x86_64 2024-05-08T21:20:39.530 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind-1.3.1-3.el8.x86_64 2024-05-08T21:20:39.530 INFO:teuthology.orchestra.run.smithi003.stdout: libuuid-devel-2.32.1-44.el8_9.1.x86_64 2024-05-08T21:20:39.530 INFO:teuthology.orchestra.run.smithi003.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-05-08T21:20:39.531 INFO:teuthology.orchestra.run.smithi003.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-05-08T21:20:39.531 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-05-08T21:20:39.531 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-05-08T21:20:39.531 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-0.3.15-6.el8.x86_64 2024-05-08T21:20:39.531 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-05-08T21:20:39.531 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-05-08T21:20:39.531 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-05-08T21:20:39.531 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-05-08T21:20:39.531 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-05-08T21:20:39.531 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-05-08T21:20:39.531 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-05-08T21:20:39.531 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-05-08T21:20:39.531 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-05-08T21:20:39.531 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel-2.5.1-7.el8.noarch 2024-05-08T21:20:39.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-05-08T21:20:39.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-05-08T21:20:39.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-05-08T21:20:39.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-05-08T21:20:39.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-05-08T21:20:39.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-05-08T21:20:39.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-05-08T21:20:39.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-05-08T21:20:39.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-05-08T21:20:39.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-6.2-6.el8.noarch 2024-05-08T21:20:39.532 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-05-08T21:20:39.533 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-05-08T21:20:39.533 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-05-08T21:20:39.533 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-05-08T21:20:39.533 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-05-08T21:20:39.533 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-05-08T21:20:39.533 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-05-08T21:20:39.533 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako-1.0.6-14.el8.noarch 2024-05-08T21:20:39.533 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-05-08T21:20:39.533 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-05-08T21:20:39.533 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-05-08T21:20:39.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-05-08T21:20:39.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-05-08T21:20:39.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-05-08T21:20:39.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging-16.8-9.el8.noarch 2024-05-08T21:20:39.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-05-08T21:20:39.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend-2.6-1.el8.noarch 2024-05-08T21:20:39.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-05-08T21:20:39.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-05-08T21:20:39.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-05-08T21:20:39.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-05-08T21:20:39.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser-2.14-14.el8.noarch 2024-05-08T21:20:39.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz-2017.2-11.el8.noarch 2024-05-08T21:20:39.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-05-08T21:20:39.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-05-08T21:20:39.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes-2.4.1-12.el8.noarch 2024-05-08T21:20:39.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa-4.9-2.el8.noarch 2024-05-08T21:20:39.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-05-08T21:20:39.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-05-08T21:20:39.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-05-08T21:20:39.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-05-08T21:20:39.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-05-08T21:20:39.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-05-08T21:20:39.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-05-08T21:20:39.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-05-08T21:20:39.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-05-08T21:20:39.536 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-05-08T21:20:39.536 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-05-08T21:20:39.536 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.536 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.536 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T21:20:39.536 INFO:teuthology.orchestra.run.smithi003.stdout: socat-1.7.4.1-1.el8.x86_64 2024-05-08T21:20:39.536 INFO:teuthology.orchestra.run.smithi003.stdout: thrift-0.13.0-2.el8.x86_64 2024-05-08T21:20:39.536 INFO:teuthology.orchestra.run.smithi003.stdout: uuid-1.6.2-43.el8.x86_64 2024-05-08T21:20:39.536 INFO:teuthology.orchestra.run.smithi003.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-05-08T21:20:39.536 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-05-08T21:20:39.537 INFO:teuthology.orchestra.run.smithi003.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-05-08T21:20:39.537 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:39.537 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-05-08T21:20:39.762 DEBUG:teuthology.parallel:result is None 2024-05-08T21:20:39.764 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d21e4f961318b56f0d915caa2490c3711bf13c34 2024-05-08T21:20:39.985 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-08T21:20:40.016 INFO:teuthology.orchestra.run.smithi003.stdout:18.2.2-1322.gd21e4f96.el8 2024-05-08T21:20:40.016 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1322.gd21e4f96.el8 2024-05-08T21:20:40.017 INFO:teuthology.task.install:The correct ceph version 18.2.2-1322.gd21e4f96 is installed. 2024-05-08T21:20:40.019 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d21e4f961318b56f0d915caa2490c3711bf13c34 2024-05-08T21:20:40.244 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-08T21:20:40.272 INFO:teuthology.orchestra.run.smithi186.stdout:18.2.2-1322.gd21e4f96.el8 2024-05-08T21:20:40.272 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1322.gd21e4f96.el8 2024-05-08T21:20:40.272 INFO:teuthology.task.install:The correct ceph version 18.2.2-1322.gd21e4f96 is installed. 2024-05-08T21:20:40.274 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-08T21:20:40.274 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:20:40.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-08T21:20:40.328 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:20:40.328 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-08T21:20:40.375 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-08T21:20:40.375 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:20:40.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2024-05-08T21:20:40.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-08T21:20:40.484 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:20:40.484 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-05-08T21:20:40.515 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-08T21:20:40.585 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-08T21:20:40.585 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:20:40.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-08T21:20:40.619 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-08T21:20:40.694 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:20:40.694 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-08T21:20:40.722 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-08T21:20:40.791 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-08T21:20:40.791 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:20:40.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2024-05-08T21:20:40.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-08T21:20:40.900 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:20:40.900 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-05-08T21:20:40.927 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-08T21:20:40.995 INFO:teuthology.run_tasks:Running task ceph... 2024-05-08T21:20:41.076 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-08T21:20:41.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /var/log/ceph 2024-05-08T21:20:41.079 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /var/log/ceph 2024-05-08T21:20:41.110 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-08T21:20:41.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-08T21:20:41.153 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-08T21:20:41.184 INFO:tasks.ceph:Creating extra log directories... 2024-05-08T21:20:41.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-08T21:20:41.227 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-08T21:20:41.265 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-05-08T21:20:41.265 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-05-08T21:20:41.266 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-05-08T21:20:41.266 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:20:41.266 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-05-08T21:20:41.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-05-08T21:20:41.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-05-08T21:20:41.484 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:20:41.484 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-05-08T21:20:41.501 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-05-08T21:20:41.617 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-05-08T21:20:41.650 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-08T21:20:41.650 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'unmatched fragstat', 'unmatched rstat', 'was unreadable, recreating it now', 'Scrub error on inode', 'Metadata damage detected', 'MDS_FAILED', 'MDS_DAMAGE'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-08T21:20:41.650 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7698047', 'branch': 'wip-pdonnell-testing-20240508.150423-reef', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/alternate-pool}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7698047', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20240508.150423-reef'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'unmatched fragstat', 'unmatched rstat', 'was unreadable, recreating it now', 'Scrub error on inode', 'Metadata damage detected', 'MDS_FAILED', 'MDS_DAMAGE'], 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20240508.150423-reef', 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 65, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 2738, 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34', 'sleep_before_teardown': 0, 'subset': '17/512', 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20240508.150423-reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d21e4f961318b56f0d915caa2490c3711bf13c34/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi003.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4gFNQ1ey7Dp5Fx416rLYsi8Dgi1Kok8hXob6AtIr+XKEJVSs/karzoZzdqjR4wbzZbCvO4bFrFgFY6Pr/A5A0=', 'smithi186.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNCJ9V9wH96GjNq0MhtgUxT+2pzTTqQNcfJr+gi2hpna4lGM81wHR8Gmgv+/DuVpEXBdopANQ7gkmGZ8nL0zhlE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'unmatched fragstat', 'unmatched rstat', 'was unreadable, recreating it now', 'Scrub error on inode', 'Metadata damage detected', 'MDS_FAILED', 'MDS_DAMAGE'], 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_recovery_pool']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-08_19:22:21', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-08T21:20:41.650 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-08T21:20:41.653 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-08T21:20:41.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/run/ceph 2024-05-08T21:20:41.716 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/run/ceph 2024-05-08T21:20:41.768 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:20:41.768 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2024-05-08T21:20:41.809 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-08T21:20:41.809 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 2024-05-08T21:20:41.871 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-08T21:20:41.871 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T21:20:41.871 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 6h/6d Inode: 181584 Links: 1 2024-05-08T21:20:41.871 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T21:20:41.871 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T21:20:41.871 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-05-08 21:20:26.777749936 +0000 2024-05-08T21:20:41.871 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-05-08 21:16:53.316766489 +0000 2024-05-08T21:20:41.871 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-05-08 21:16:53.316766489 +0000 2024-05-08T21:20:41.871 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2024-05-08T21:20:41.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-08T21:20:41.943 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-05-08T21:20:41.943 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-05-08T21:20:41.943 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000243643 s, 2.1 MB/s 2024-05-08T21:20:41.944 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-08T21:20:42.007 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 2024-05-08T21:20:42.070 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-08T21:20:42.070 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T21:20:42.070 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 6h/6d Inode: 185538 Links: 1 2024-05-08T21:20:42.070 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T21:20:42.070 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T21:20:42.070 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-05-08 21:20:26.777749936 +0000 2024-05-08T21:20:42.070 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-05-08 21:16:53.681762033 +0000 2024-05-08T21:20:42.070 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-05-08 21:16:53.681762033 +0000 2024-05-08T21:20:42.070 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2024-05-08T21:20:42.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-08T21:20:42.142 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-05-08T21:20:42.142 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-05-08T21:20:42.142 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000229627 s, 2.2 MB/s 2024-05-08T21:20:42.143 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-08T21:20:42.206 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 2024-05-08T21:20:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-08T21:20:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T21:20:42.269 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 6h/6d Inode: 184659 Links: 1 2024-05-08T21:20:42.269 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T21:20:42.269 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T21:20:42.269 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-05-08 21:20:26.777749936 +0000 2024-05-08T21:20:42.269 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-05-08 21:16:54.072757259 +0000 2024-05-08T21:20:42.269 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-05-08 21:16:54.072757259 +0000 2024-05-08T21:20:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2024-05-08T21:20:42.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-08T21:20:42.341 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-05-08T21:20:42.341 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-05-08T21:20:42.341 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000189351 s, 2.7 MB/s 2024-05-08T21:20:42.343 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-08T21:20:42.406 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 2024-05-08T21:20:42.469 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-08T21:20:42.469 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T21:20:42.469 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 6h/6d Inode: 178822 Links: 1 2024-05-08T21:20:42.469 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T21:20:42.469 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T21:20:42.469 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-05-08 21:20:26.777749936 +0000 2024-05-08T21:20:42.469 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-05-08 21:16:54.482752252 +0000 2024-05-08T21:20:42.469 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-05-08 21:16:54.482752252 +0000 2024-05-08T21:20:42.470 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2024-05-08T21:20:42.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-08T21:20:42.541 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-05-08T21:20:42.541 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-05-08T21:20:42.541 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000250192 s, 2.0 MB/s 2024-05-08T21:20:42.542 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-08T21:20:42.606 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-08T21:20:42.606 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:20:42.606 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-05-08T21:20:42.627 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-08T21:20:42.627 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-05-08T21:20:42.687 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-08T21:20:42.687 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T21:20:42.687 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 6h/6d Inode: 180952 Links: 1 2024-05-08T21:20:42.688 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T21:20:42.688 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T21:20:42.688 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-08 21:20:21.047336042 +0000 2024-05-08T21:20:42.688 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-08 21:16:00.742133085 +0000 2024-05-08T21:20:42.688 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-08 21:16:00.742133085 +0000 2024-05-08T21:20:42.688 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2024-05-08T21:20:42.688 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-08T21:20:42.757 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-08T21:20:42.757 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-08T21:20:42.757 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000276395 s, 1.9 MB/s 2024-05-08T21:20:42.758 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-08T21:20:42.819 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-05-08T21:20:42.878 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-08T21:20:42.879 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T21:20:42.879 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 6h/6d Inode: 178813 Links: 1 2024-05-08T21:20:42.879 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T21:20:42.879 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T21:20:42.879 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-08 21:20:21.048336024 +0000 2024-05-08T21:20:42.879 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-08 21:16:01.087127143 +0000 2024-05-08T21:20:42.879 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-08 21:16:01.087127143 +0000 2024-05-08T21:20:42.879 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2024-05-08T21:20:42.879 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-08T21:20:42.946 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-08T21:20:42.946 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-08T21:20:42.946 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000239942 s, 2.1 MB/s 2024-05-08T21:20:42.947 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-08T21:20:43.006 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-05-08T21:20:43.064 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-08T21:20:43.064 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T21:20:43.064 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 6h/6d Inode: 177826 Links: 1 2024-05-08T21:20:43.064 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T21:20:43.065 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T21:20:43.065 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-08 21:20:21.048336024 +0000 2024-05-08T21:20:43.065 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-08 21:16:01.432121200 +0000 2024-05-08T21:20:43.065 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-08 21:16:01.432121200 +0000 2024-05-08T21:20:43.065 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2024-05-08T21:20:43.065 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-08T21:20:43.131 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-08T21:20:43.131 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-08T21:20:43.131 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000193492 s, 2.6 MB/s 2024-05-08T21:20:43.132 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-08T21:20:43.191 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-05-08T21:20:43.303 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-08T21:20:43.303 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T21:20:43.303 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 6h/6d Inode: 178907 Links: 1 2024-05-08T21:20:43.303 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T21:20:43.303 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T21:20:43.303 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-08 21:20:21.048336024 +0000 2024-05-08T21:20:43.303 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-08 21:16:01.805114775 +0000 2024-05-08T21:20:43.303 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-08 21:16:01.805114775 +0000 2024-05-08T21:20:43.303 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2024-05-08T21:20:43.304 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-08T21:20:43.329 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-08T21:20:43.330 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-08T21:20:43.330 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000249562 s, 2.1 MB/s 2024-05-08T21:20:43.331 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-08T21:20:43.388 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-08T21:20:43.388 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-08T21:20:43.388 INFO:tasks.ceph:Generating config... 2024-05-08T21:20:43.389 INFO:tasks.ceph:[client] client mount timeout = 600 2024-05-08T21:20:43.390 INFO:tasks.ceph:[client] debug client = 20 2024-05-08T21:20:43.390 INFO:tasks.ceph:[client] debug ms = 1 2024-05-08T21:20:43.390 INFO:tasks.ceph:[client] pid file = 2024-05-08T21:20:43.390 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-05-08T21:20:43.390 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-05-08T21:20:43.390 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-05-08T21:20:43.390 INFO:tasks.ceph:[mds] debug mds = 20 2024-05-08T21:20:43.390 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-05-08T21:20:43.390 INFO:tasks.ceph:[mds] debug ms = 1 2024-05-08T21:20:43.390 INFO:tasks.ceph:[mds] mds debug frag = True 2024-05-08T21:20:43.390 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-05-08T21:20:43.390 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-05-08T21:20:43.390 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-05-08T21:20:43.390 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-05-08T21:20:43.391 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-05-08T21:20:43.391 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-05-08T21:20:43.391 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-08T21:20:43.391 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-08T21:20:43.391 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-08T21:20:43.391 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-08T21:20:43.391 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-08T21:20:43.391 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-05-08T21:20:43.391 INFO:tasks.ceph:[osd] bdev async discard = True 2024-05-08T21:20:43.391 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-05-08T21:20:43.391 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-05-08T21:20:43.391 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-08T21:20:43.391 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-08T21:20:43.391 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-05-08T21:20:43.392 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-05-08T21:20:43.392 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-08T21:20:43.392 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-08T21:20:43.392 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-05-08T21:20:43.392 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-08T21:20:43.392 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-08T21:20:43.392 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-08T21:20:43.392 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-08T21:20:43.392 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-08T21:20:43.392 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-05-08T21:20:43.392 INFO:tasks.ceph:Setting up mon.a... 2024-05-08T21:20:43.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-08T21:20:43.459 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.keyring 2024-05-08T21:20:43.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-08T21:20:43.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-08T21:20:43.548 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.3'), ('mon.b', '172.21.15.186'), ('mon.c', '[v2:172.21.15.186:3301,v1:172.21.15.186:6790]')] 2024-05-08T21:20:43.548 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.3,172.21.15.186,[v2:172.21.15.186:3301,v1:172.21.15.186:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-05-08T21:20:43.549 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:20:43.549 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-08T21:20:43.611 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.3 --add b 172.21.15.186 --addv c '[v2:172.21.15.186:3301,v1:172.21.15.186:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-05-08T21:20:43.708 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-08T21:20:43.708 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: generated fsid d0f58275-758f-4470-b003-84d6abe25489 2024-05-08T21:20:43.708 INFO:teuthology.orchestra.run.smithi003.stdout:setting min_mon_release = pacific 2024-05-08T21:20:43.708 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 0 2024-05-08T21:20:43.709 INFO:teuthology.orchestra.run.smithi003.stdout:fsid d0f58275-758f-4470-b003-84d6abe25489 2024-05-08T21:20:43.709 INFO:teuthology.orchestra.run.smithi003.stdout:last_changed 2024-05-08T21:20:43.707509+0000 2024-05-08T21:20:43.709 INFO:teuthology.orchestra.run.smithi003.stdout:created 2024-05-08T21:20:43.707509+0000 2024-05-08T21:20:43.709 INFO:teuthology.orchestra.run.smithi003.stdout:min_mon_release 16 (pacific) 2024-05-08T21:20:43.709 INFO:teuthology.orchestra.run.smithi003.stdout:election_strategy: 1 2024-05-08T21:20:43.709 INFO:teuthology.orchestra.run.smithi003.stdout:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-05-08T21:20:43.709 INFO:teuthology.orchestra.run.smithi003.stdout:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-08T21:20:43.709 INFO:teuthology.orchestra.run.smithi003.stdout:2: [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] mon.c 2024-05-08T21:20:43.709 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-05-08T21:20:43.710 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-08T21:20:43.729 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID d0f58275-758f-4470-b003-84d6abe25489... 2024-05-08T21:20:43.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-08T21:20:43.772 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-08T21:20:43.821 INFO:teuthology.orchestra.run.smithi186.stdout:[global] 2024-05-08T21:20:43.821 INFO:teuthology.orchestra.run.smithi186.stdout: chdir = "" 2024-05-08T21:20:43.821 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-08T21:20:43.821 INFO:teuthology.orchestra.run.smithi186.stdout: auth supported = cephx 2024-05-08T21:20:43.821 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.821 INFO:teuthology.orchestra.run.smithi186.stdout: filestore xattr use omap = true 2024-05-08T21:20:43.821 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.821 INFO:teuthology.orchestra.run.smithi186.stdout: mon clock drift allowed = 1.000 2024-05-08T21:20:43.822 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.822 INFO:teuthology.orchestra.run.smithi186.stdout: osd crush chooseleaf type = 0 2024-05-08T21:20:43.822 INFO:teuthology.orchestra.run.smithi186.stdout: auth debug = true 2024-05-08T21:20:43.822 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.822 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on old message = true 2024-05-08T21:20:43.822 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on bug = true 2024-05-08T21:20:43.822 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.822 INFO:teuthology.orchestra.run.smithi186.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-08T21:20:43.822 INFO:teuthology.orchestra.run.smithi186.stdout: mon pg warn max object skew = 0 2024-05-08T21:20:43.822 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.822 INFO:teuthology.orchestra.run.smithi186.stdout: # disable pg_autoscaler by default for new pools 2024-05-08T21:20:43.822 INFO:teuthology.orchestra.run.smithi186.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-08T21:20:43.822 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.823 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default size = 2 2024-05-08T21:20:43.823 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.823 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow primary affinity = true 2024-05-08T21:20:43.823 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow pg remap = true 2024-05-08T21:20:43.823 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on legacy crush tunables = false 2024-05-08T21:20:43.823 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on crush straw calc version zero = false 2024-05-08T21:20:43.823 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on no sortbitwise = false 2024-05-08T21:20:43.823 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on osd down out interval zero = false 2024-05-08T21:20:43.823 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on too few osds = false 2024-05-08T21:20:43.823 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-08T21:20:43.823 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-08T21:20:43.823 INFO:teuthology.orchestra.run.smithi186.stdout: mon_allow_pool_size_one = true 2024-05-08T21:20:43.823 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.823 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-08T21:20:43.824 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.824 INFO:teuthology.orchestra.run.smithi186.stdout: osd default data pool replay window = 5 2024-05-08T21:20:43.824 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.824 INFO:teuthology.orchestra.run.smithi186.stdout: mon allow pool delete = true 2024-05-08T21:20:43.824 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.824 INFO:teuthology.orchestra.run.smithi186.stdout: mon cluster log file level = debug 2024-05-08T21:20:43.824 INFO:teuthology.orchestra.run.smithi186.stdout: debug asserts on shutdown = true 2024-05-08T21:20:43.824 INFO:teuthology.orchestra.run.smithi186.stdout: mon health detail to clog = false 2024-05-08T21:20:43.824 INFO:teuthology.orchestra.run.smithi186.stdout: mon host = "172.21.15.3,172.21.15.186,[v2:172.21.15.186:3301,v1:172.21.15.186:6790]" 2024-05-08T21:20:43.824 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on skipped message = False 2024-05-08T21:20:43.824 INFO:teuthology.orchestra.run.smithi186.stdout: fsid = d0f58275-758f-4470-b003-84d6abe25489 2024-05-08T21:20:43.824 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.824 INFO:teuthology.orchestra.run.smithi186.stdout:[osd] 2024-05-08T21:20:43.824 INFO:teuthology.orchestra.run.smithi186.stdout: osd journal size = 100 2024-05-08T21:20:43.824 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.824 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub load threshold = 5.0 2024-05-08T21:20:43.825 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub max interval = 600 2024-05-08T21:20:43.825 INFO:teuthology.orchestra.run.smithi186.stdout: osd mclock profile = high_recovery_ops 2024-05-08T21:20:43.825 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.825 INFO:teuthology.orchestra.run.smithi186.stdout: osd recover clone overlap = true 2024-05-08T21:20:43.825 INFO:teuthology.orchestra.run.smithi186.stdout: osd recovery max chunk = 1048576 2024-05-08T21:20:43.825 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.825 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug shutdown = true 2024-05-08T21:20:43.825 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug op order = true 2024-05-08T21:20:43.825 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify stray on activate = true 2024-05-08T21:20:43.825 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.825 INFO:teuthology.orchestra.run.smithi186.stdout: osd open classes on start = true 2024-05-08T21:20:43.825 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug pg log writeout = true 2024-05-08T21:20:43.825 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.825 INFO:teuthology.orchestra.run.smithi186.stdout: osd deep scrub update digest min age = 30 2024-05-08T21:20:43.825 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.826 INFO:teuthology.orchestra.run.smithi186.stdout: osd map max advance = 10 2024-05-08T21:20:43.826 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.826 INFO:teuthology.orchestra.run.smithi186.stdout: journal zero on create = true 2024-05-08T21:20:43.826 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.826 INFO:teuthology.orchestra.run.smithi186.stdout: filestore ondisk finisher threads = 3 2024-05-08T21:20:43.826 INFO:teuthology.orchestra.run.smithi186.stdout: filestore apply finisher threads = 3 2024-05-08T21:20:43.826 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.826 INFO:teuthology.orchestra.run.smithi186.stdout: bdev debug aio = true 2024-05-08T21:20:43.826 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug misdirected ops = true 2024-05-08T21:20:43.826 INFO:teuthology.orchestra.run.smithi186.stdout: bdev async discard = True 2024-05-08T21:20:43.826 INFO:teuthology.orchestra.run.smithi186.stdout: bdev enable discard = True 2024-05-08T21:20:43.826 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore allocator = bitmap 2024-05-08T21:20:43.826 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore block size = 96636764160 2024-05-08T21:20:43.826 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore fsck on mount = True 2024-05-08T21:20:43.827 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluefs = 1/20 2024-05-08T21:20:43.827 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluestore = 1/20 2024-05-08T21:20:43.827 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-08T21:20:43.827 INFO:teuthology.orchestra.run.smithi186.stdout: debug osd = 20 2024-05-08T21:20:43.827 INFO:teuthology.orchestra.run.smithi186.stdout: debug rocksdb = 4/10 2024-05-08T21:20:43.827 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd backfillfull_ratio = 0.85 2024-05-08T21:20:43.827 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd full ratio = 0.9 2024-05-08T21:20:43.827 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd nearfull ratio = 0.8 2024-05-08T21:20:43.827 INFO:teuthology.orchestra.run.smithi186.stdout: osd failsafe full ratio = 0.95 2024-05-08T21:20:43.827 INFO:teuthology.orchestra.run.smithi186.stdout: osd objectstore = bluestore 2024-05-08T21:20:43.827 INFO:teuthology.orchestra.run.smithi186.stdout: osd op complaint time = 180 2024-05-08T21:20:43.827 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.827 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr] 2024-05-08T21:20:43.827 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-08T21:20:43.827 INFO:teuthology.orchestra.run.smithi186.stdout: debug mgr = 20 2024-05-08T21:20:43.827 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-05-08T21:20:43.828 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-05-08T21:20:43.828 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-05-08T21:20:43.828 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-05-08T21:20:43.828 INFO:teuthology.orchestra.run.smithi186.stdout: mgr/telemetry/nag = false 2024-05-08T21:20:43.828 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.828 INFO:teuthology.orchestra.run.smithi186.stdout:[mon] 2024-05-08T21:20:43.828 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-08T21:20:43.828 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-05-08T21:20:43.828 INFO:teuthology.orchestra.run.smithi186.stdout: debug paxos = 20 2024-05-08T21:20:43.828 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-05-08T21:20:43.828 INFO:teuthology.orchestra.run.smithi186.stdout: mon data avail warn = 5 2024-05-08T21:20:43.828 INFO:teuthology.orchestra.run.smithi186.stdout: mon mgr mkfs grace = 240 2024-05-08T21:20:43.828 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-05-08T21:20:43.828 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd reporter subtree level = osd 2024-05-08T21:20:43.828 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd prime pg temp = true 2024-05-08T21:20:43.828 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-05-08T21:20:43.829 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.829 INFO:teuthology.orchestra.run.smithi186.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-08T21:20:43.829 INFO:teuthology.orchestra.run.smithi186.stdout: auth mon ticket ttl = 660 # 11m 2024-05-08T21:20:43.829 INFO:teuthology.orchestra.run.smithi186.stdout: auth service ticket ttl = 240 # 4m 2024-05-08T21:20:43.829 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.829 INFO:teuthology.orchestra.run.smithi186.stdout: # don't complain about insecure global_id in the test suite 2024-05-08T21:20:43.829 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-08T21:20:43.829 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-08T21:20:43.829 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.829 INFO:teuthology.orchestra.run.smithi186.stdout: # 1m isn't quite enough 2024-05-08T21:20:43.829 INFO:teuthology.orchestra.run.smithi186.stdout: mon_down_mkfs_grace = 2m 2024-05-08T21:20:43.829 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.829 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_filestore_osds = false 2024-05-08T21:20:43.829 INFO:teuthology.orchestra.run.smithi186.stdout: mon op complaint time = 120 2024-05-08T21:20:43.829 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:20:43.830 INFO:teuthology.orchestra.run.smithi186.stdout:[client] 2024-05-08T21:20:43.830 INFO:teuthology.orchestra.run.smithi186.stdout: rgw cache enabled = true 2024-05-08T21:20:43.830 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable ops log = true 2024-05-08T21:20:43.830 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable usage log = true 2024-05-08T21:20:43.830 INFO:teuthology.orchestra.run.smithi186.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-08T21:20:43.830 INFO:teuthology.orchestra.run.smithi186.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-08T21:20:43.830 INFO:teuthology.orchestra.run.smithi186.stdout: client mount timeout = 600 2024-05-08T21:20:43.830 INFO:teuthology.orchestra.run.smithi186.stdout: debug client = 20 2024-05-08T21:20:43.830 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-08T21:20:43.830 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = "" 2024-05-08T21:20:43.830 INFO:teuthology.orchestra.run.smithi186.stdout: rados mon op timeout = 900 2024-05-08T21:20:43.830 INFO:teuthology.orchestra.run.smithi186.stdout: rados osd op timeout = 900 2024-05-08T21:20:43.830 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.a] 2024-05-08T21:20:43.830 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.b] 2024-05-08T21:20:43.831 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.c] 2024-05-08T21:20:43.831 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.a] 2024-05-08T21:20:43.831 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.c] 2024-05-08T21:20:43.831 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.b] 2024-05-08T21:20:43.831 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.d] 2024-05-08T21:20:43.831 INFO:teuthology.orchestra.run.smithi186.stdout:[mds] 2024-05-08T21:20:43.831 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds = 20 2024-05-08T21:20:43.831 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds balancer = 20 2024-05-08T21:20:43.831 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-08T21:20:43.831 INFO:teuthology.orchestra.run.smithi186.stdout: mds debug frag = True 2024-05-08T21:20:43.831 INFO:teuthology.orchestra.run.smithi186.stdout: mds debug scatterstat = True 2024-05-08T21:20:43.831 INFO:teuthology.orchestra.run.smithi186.stdout: mds op complaint time = 180 2024-05-08T21:20:43.831 INFO:teuthology.orchestra.run.smithi186.stdout: mds verify scatter = True 2024-05-08T21:20:43.831 INFO:teuthology.orchestra.run.smithi186.stdout: osd op complaint time = 180 2024-05-08T21:20:43.831 INFO:teuthology.orchestra.run.smithi186.stdout: rados mon op timeout = 900 2024-05-08T21:20:43.831 INFO:teuthology.orchestra.run.smithi186.stdout: rados osd op timeout = 900 2024-05-08T21:20:43.832 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2024-05-08T21:20:43.832 INFO:teuthology.orchestra.run.smithi003.stdout: chdir = "" 2024-05-08T21:20:43.832 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-08T21:20:43.832 INFO:teuthology.orchestra.run.smithi003.stdout: auth supported = cephx 2024-05-08T21:20:43.832 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.832 INFO:teuthology.orchestra.run.smithi003.stdout: filestore xattr use omap = true 2024-05-08T21:20:43.832 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.832 INFO:teuthology.orchestra.run.smithi003.stdout: mon clock drift allowed = 1.000 2024-05-08T21:20:43.832 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.832 INFO:teuthology.orchestra.run.smithi003.stdout: osd crush chooseleaf type = 0 2024-05-08T21:20:43.833 INFO:teuthology.orchestra.run.smithi003.stdout: auth debug = true 2024-05-08T21:20:43.833 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.833 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on old message = true 2024-05-08T21:20:43.833 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on bug = true 2024-05-08T21:20:43.833 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.833 INFO:teuthology.orchestra.run.smithi003.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-08T21:20:43.833 INFO:teuthology.orchestra.run.smithi003.stdout: mon pg warn max object skew = 0 2024-05-08T21:20:43.833 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.833 INFO:teuthology.orchestra.run.smithi003.stdout: # disable pg_autoscaler by default for new pools 2024-05-08T21:20:43.833 INFO:teuthology.orchestra.run.smithi003.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-08T21:20:43.833 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.833 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default size = 2 2024-05-08T21:20:43.834 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.834 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow primary affinity = true 2024-05-08T21:20:43.834 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow pg remap = true 2024-05-08T21:20:43.834 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on legacy crush tunables = false 2024-05-08T21:20:43.834 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on crush straw calc version zero = false 2024-05-08T21:20:43.834 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on no sortbitwise = false 2024-05-08T21:20:43.834 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on osd down out interval zero = false 2024-05-08T21:20:43.834 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on too few osds = false 2024-05-08T21:20:43.834 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-08T21:20:43.834 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-08T21:20:43.834 INFO:teuthology.orchestra.run.smithi003.stdout: mon_allow_pool_size_one = true 2024-05-08T21:20:43.834 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.834 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-08T21:20:43.834 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.834 INFO:teuthology.orchestra.run.smithi003.stdout: osd default data pool replay window = 5 2024-05-08T21:20:43.835 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.835 INFO:teuthology.orchestra.run.smithi003.stdout: mon allow pool delete = true 2024-05-08T21:20:43.835 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.835 INFO:teuthology.orchestra.run.smithi003.stdout: mon cluster log file level = debug 2024-05-08T21:20:43.835 INFO:teuthology.orchestra.run.smithi003.stdout: debug asserts on shutdown = true 2024-05-08T21:20:43.835 INFO:teuthology.orchestra.run.smithi003.stdout: mon health detail to clog = false 2024-05-08T21:20:43.835 INFO:teuthology.orchestra.run.smithi003.stdout: mon host = "172.21.15.3,172.21.15.186,[v2:172.21.15.186:3301,v1:172.21.15.186:6790]" 2024-05-08T21:20:43.835 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on skipped message = False 2024-05-08T21:20:43.835 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = d0f58275-758f-4470-b003-84d6abe25489 2024-05-08T21:20:43.835 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.835 INFO:teuthology.orchestra.run.smithi003.stdout:[osd] 2024-05-08T21:20:43.835 INFO:teuthology.orchestra.run.smithi003.stdout: osd journal size = 100 2024-05-08T21:20:43.835 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.835 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub load threshold = 5.0 2024-05-08T21:20:43.835 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub max interval = 600 2024-05-08T21:20:43.836 INFO:teuthology.orchestra.run.smithi003.stdout: osd mclock profile = high_recovery_ops 2024-05-08T21:20:43.836 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.836 INFO:teuthology.orchestra.run.smithi003.stdout: osd recover clone overlap = true 2024-05-08T21:20:43.836 INFO:teuthology.orchestra.run.smithi003.stdout: osd recovery max chunk = 1048576 2024-05-08T21:20:43.836 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.836 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug shutdown = true 2024-05-08T21:20:43.836 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug op order = true 2024-05-08T21:20:43.836 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify stray on activate = true 2024-05-08T21:20:43.836 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.836 INFO:teuthology.orchestra.run.smithi003.stdout: osd open classes on start = true 2024-05-08T21:20:43.836 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug pg log writeout = true 2024-05-08T21:20:43.836 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.836 INFO:teuthology.orchestra.run.smithi003.stdout: osd deep scrub update digest min age = 30 2024-05-08T21:20:43.836 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.836 INFO:teuthology.orchestra.run.smithi003.stdout: osd map max advance = 10 2024-05-08T21:20:43.836 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.837 INFO:teuthology.orchestra.run.smithi003.stdout: journal zero on create = true 2024-05-08T21:20:43.837 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.837 INFO:teuthology.orchestra.run.smithi003.stdout: filestore ondisk finisher threads = 3 2024-05-08T21:20:43.837 INFO:teuthology.orchestra.run.smithi003.stdout: filestore apply finisher threads = 3 2024-05-08T21:20:43.837 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.837 INFO:teuthology.orchestra.run.smithi003.stdout: bdev debug aio = true 2024-05-08T21:20:43.837 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug misdirected ops = true 2024-05-08T21:20:43.837 INFO:teuthology.orchestra.run.smithi003.stdout: bdev async discard = True 2024-05-08T21:20:43.837 INFO:teuthology.orchestra.run.smithi003.stdout: bdev enable discard = True 2024-05-08T21:20:43.837 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore allocator = bitmap 2024-05-08T21:20:43.837 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore block size = 96636764160 2024-05-08T21:20:43.837 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore fsck on mount = True 2024-05-08T21:20:43.837 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluefs = 1/20 2024-05-08T21:20:43.837 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluestore = 1/20 2024-05-08T21:20:43.837 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-08T21:20:43.838 INFO:teuthology.orchestra.run.smithi003.stdout: debug osd = 20 2024-05-08T21:20:43.838 INFO:teuthology.orchestra.run.smithi003.stdout: debug rocksdb = 4/10 2024-05-08T21:20:43.838 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd backfillfull_ratio = 0.85 2024-05-08T21:20:43.838 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd full ratio = 0.9 2024-05-08T21:20:43.838 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd nearfull ratio = 0.8 2024-05-08T21:20:43.838 INFO:teuthology.orchestra.run.smithi003.stdout: osd failsafe full ratio = 0.95 2024-05-08T21:20:43.838 INFO:teuthology.orchestra.run.smithi003.stdout: osd objectstore = bluestore 2024-05-08T21:20:43.838 INFO:teuthology.orchestra.run.smithi003.stdout: osd op complaint time = 180 2024-05-08T21:20:43.839 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.839 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr] 2024-05-08T21:20:43.839 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-08T21:20:43.839 INFO:teuthology.orchestra.run.smithi003.stdout: debug mgr = 20 2024-05-08T21:20:43.839 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2024-05-08T21:20:43.839 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2024-05-08T21:20:43.839 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2024-05-08T21:20:43.839 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2024-05-08T21:20:43.839 INFO:teuthology.orchestra.run.smithi003.stdout: mgr/telemetry/nag = false 2024-05-08T21:20:43.839 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.839 INFO:teuthology.orchestra.run.smithi003.stdout:[mon] 2024-05-08T21:20:43.839 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-08T21:20:43.839 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2024-05-08T21:20:43.839 INFO:teuthology.orchestra.run.smithi003.stdout: debug paxos = 20 2024-05-08T21:20:43.839 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2024-05-08T21:20:43.840 INFO:teuthology.orchestra.run.smithi003.stdout: mon data avail warn = 5 2024-05-08T21:20:43.840 INFO:teuthology.orchestra.run.smithi003.stdout: mon mgr mkfs grace = 240 2024-05-08T21:20:43.840 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2024-05-08T21:20:43.840 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd reporter subtree level = osd 2024-05-08T21:20:43.840 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd prime pg temp = true 2024-05-08T21:20:43.840 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2024-05-08T21:20:43.840 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.840 INFO:teuthology.orchestra.run.smithi003.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-08T21:20:43.840 INFO:teuthology.orchestra.run.smithi003.stdout: auth mon ticket ttl = 660 # 11m 2024-05-08T21:20:43.840 INFO:teuthology.orchestra.run.smithi003.stdout: auth service ticket ttl = 240 # 4m 2024-05-08T21:20:43.840 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.840 INFO:teuthology.orchestra.run.smithi003.stdout: # don't complain about insecure global_id in the test suite 2024-05-08T21:20:43.841 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-08T21:20:43.841 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-08T21:20:43.841 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.841 INFO:teuthology.orchestra.run.smithi003.stdout: # 1m isn't quite enough 2024-05-08T21:20:43.841 INFO:teuthology.orchestra.run.smithi003.stdout: mon_down_mkfs_grace = 2m 2024-05-08T21:20:43.841 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.841 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_filestore_osds = false 2024-05-08T21:20:43.841 INFO:teuthology.orchestra.run.smithi003.stdout: mon op complaint time = 120 2024-05-08T21:20:43.841 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:20:43.841 INFO:teuthology.orchestra.run.smithi003.stdout:[client] 2024-05-08T21:20:43.841 INFO:teuthology.orchestra.run.smithi003.stdout: rgw cache enabled = true 2024-05-08T21:20:43.841 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable ops log = true 2024-05-08T21:20:43.841 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable usage log = true 2024-05-08T21:20:43.841 INFO:teuthology.orchestra.run.smithi003.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-08T21:20:43.842 INFO:teuthology.orchestra.run.smithi003.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-08T21:20:43.842 INFO:teuthology.orchestra.run.smithi003.stdout: client mount timeout = 600 2024-05-08T21:20:43.842 INFO:teuthology.orchestra.run.smithi003.stdout: debug client = 20 2024-05-08T21:20:43.842 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-08T21:20:43.842 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = "" 2024-05-08T21:20:43.842 INFO:teuthology.orchestra.run.smithi003.stdout: rados mon op timeout = 900 2024-05-08T21:20:43.842 INFO:teuthology.orchestra.run.smithi003.stdout: rados osd op timeout = 900 2024-05-08T21:20:43.842 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.a] 2024-05-08T21:20:43.842 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.b] 2024-05-08T21:20:43.842 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.c] 2024-05-08T21:20:43.842 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.a] 2024-05-08T21:20:43.842 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.c] 2024-05-08T21:20:43.843 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.b] 2024-05-08T21:20:43.843 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.d] 2024-05-08T21:20:43.843 INFO:teuthology.orchestra.run.smithi003.stdout:[mds] 2024-05-08T21:20:43.843 INFO:teuthology.orchestra.run.smithi003.stdout: debug mds = 20 2024-05-08T21:20:43.843 INFO:teuthology.orchestra.run.smithi003.stdout: debug mds balancer = 20 2024-05-08T21:20:43.843 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-08T21:20:43.843 INFO:teuthology.orchestra.run.smithi003.stdout: mds debug frag = True 2024-05-08T21:20:43.843 INFO:teuthology.orchestra.run.smithi003.stdout: mds debug scatterstat = True 2024-05-08T21:20:43.843 INFO:teuthology.orchestra.run.smithi003.stdout: mds op complaint time = 180 2024-05-08T21:20:43.843 INFO:teuthology.orchestra.run.smithi003.stdout: mds verify scatter = True 2024-05-08T21:20:43.843 INFO:teuthology.orchestra.run.smithi003.stdout: osd op complaint time = 180 2024-05-08T21:20:43.843 INFO:teuthology.orchestra.run.smithi003.stdout: rados mon op timeout = 900 2024-05-08T21:20:43.843 INFO:teuthology.orchestra.run.smithi003.stdout: rados osd op timeout = 900 2024-05-08T21:20:43.845 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-08T21:20:43.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-08T21:20:43.956 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-08T21:20:43.956 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:20:43.956 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-08T21:20:43.975 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:20:43.975 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-08T21:20:44.037 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi003.front.sepia.ceph.com 2024-05-08T21:20:44.037 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:20:44.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-08T21:20:44.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-08T21:20:44.168 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:20:44.168 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-08T21:20:44.189 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi186.front.sepia.ceph.com 2024-05-08T21:20:44.189 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:20:44.189 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-08T21:20:44.189 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-08T21:20:44.230 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:20:44.231 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-08T21:20:44.288 INFO:tasks.ceph:Setting up mon nodes... 2024-05-08T21:20:44.288 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-08T21:20:44.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-05-08T21:20:44.356 INFO:teuthology.orchestra.run.smithi003.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-05-08T21:20:44.359 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-08T21:20:44.426 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-08T21:20:44.429 INFO:tasks.ceph:Setting up mds nodes... 2024-05-08T21:20:44.429 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-05-08T21:20:44.498 INFO:teuthology.orchestra.run.smithi003.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-05-08T21:20:44.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-05-08T21:20:44.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-05-08T21:20:44.639 INFO:teuthology.orchestra.run.smithi003.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-05-08T21:20:44.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-05-08T21:20:44.672 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-05-08T21:20:44.729 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-05-08T21:20:44.731 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-05-08T21:20:44.758 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-05-08T21:20:44.852 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-05-08T21:20:44.855 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-05-08T21:20:44.881 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-08T21:20:44.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-08T21:20:44.936 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-08T21:20:44.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-05-08T21:20:45.022 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-05-08T21:20:45.038 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-08T21:20:45.082 INFO:teuthology.orchestra.run.smithi186.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-05-08T21:20:45.095 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-05-08T21:20:45.138 INFO:teuthology.orchestra.run.smithi186.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-05-08T21:20:45.151 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-08T21:20:45.151 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-08T21:20:45.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-08T21:20:45.186 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-08T21:20:45.186 INFO:tasks.ceph:role: osd.0 2024-05-08T21:20:45.186 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com 2024-05-08T21:20:45.186 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-08T21:20:45.268 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T21:20:45.268 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T21:20:45.269 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T21:20:45.269 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-08T21:20:45.269 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T21:20:45.269 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-05-08T21:20:45.269 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T21:20:45.269 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-08T21:20:45.269 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T21:20:45.269 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T21:20:45.700 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-05-08T21:20:45.703 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-05-08T21:20:45.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-08T21:20:45.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-08T21:20:45.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-08T21:20:45.984 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-08T21:20:45.985 INFO:tasks.ceph:role: osd.1 2024-05-08T21:20:45.985 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com 2024-05-08T21:20:45.985 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-08T21:20:46.066 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T21:20:46.066 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T21:20:46.066 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T21:20:46.067 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-08T21:20:46.067 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T21:20:46.067 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-05-08T21:20:46.067 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T21:20:46.067 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-08T21:20:46.067 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T21:20:46.067 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T21:20:46.486 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-05-08T21:20:46.488 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-05-08T21:20:46.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-08T21:20:46.602 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-08T21:20:46.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-08T21:20:46.715 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-08T21:20:46.715 INFO:tasks.ceph:role: osd.2 2024-05-08T21:20:46.715 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com 2024-05-08T21:20:46.715 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-08T21:20:46.797 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T21:20:46.797 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T21:20:46.797 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T21:20:46.798 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-08T21:20:46.798 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T21:20:46.798 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-05-08T21:20:46.798 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T21:20:46.798 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-08T21:20:46.798 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T21:20:46.798 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T21:20:47.247 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-05-08T21:20:47.249 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-05-08T21:20:47.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-08T21:20:47.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-08T21:20:47.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-08T21:20:47.525 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-08T21:20:47.525 INFO:tasks.ceph:role: osd.3 2024-05-08T21:20:47.525 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi003.front.sepia.ceph.com 2024-05-08T21:20:47.525 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-08T21:20:47.607 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T21:20:47.607 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T21:20:47.607 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T21:20:47.607 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-08T21:20:47.607 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T21:20:47.607 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-05-08T21:20:47.607 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T21:20:47.607 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-08T21:20:47.608 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T21:20:47.608 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T21:20:48.037 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-05-08T21:20:48.040 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-05-08T21:20:48.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-05-08T21:20:48.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-08T21:20:48.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T21:20:48.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:48.335+0000 7f434367b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-08T21:20:48.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:48.335+0000 7f434367b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-08T21:20:48.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:48.335+0000 7f434367b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-08T21:20:48.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:48.335+0000 7f434367b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-08T21:20:48.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:48.335+0000 7f434367b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-08T21:20:48.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:48.335+0000 7f434367b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-08T21:20:51.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-08T21:20:51.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T21:20:51.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:51.341+0000 7f406863d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-08T21:20:51.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:51.341+0000 7f406863d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-08T21:20:51.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:51.341+0000 7f406863d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-08T21:20:51.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:51.341+0000 7f406863d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-08T21:20:51.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:51.341+0000 7f406863d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-08T21:20:51.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:51.342+0000 7f406863d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-08T21:20:54.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-08T21:20:54.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T21:20:54.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:54.335+0000 7f82d44827c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-08T21:20:54.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:54.335+0000 7f82d44827c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-08T21:20:54.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:54.335+0000 7f82d44827c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-08T21:20:54.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:54.335+0000 7f82d44827c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-08T21:20:54.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:54.335+0000 7f82d44827c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-08T21:20:54.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:54.335+0000 7f82d44827c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-08T21:20:57.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-08T21:20:57.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T21:20:57.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:57.329+0000 7f130e4407c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-08T21:20:57.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:57.329+0000 7f130e4407c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-08T21:20:57.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:57.329+0000 7f130e4407c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-08T21:20:57.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:57.329+0000 7f130e4407c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-08T21:20:57.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:57.329+0000 7f130e4407c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-08T21:20:57.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:20:57.330+0000 7f130e4407c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-08T21:21:00.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-08T21:21:00.223 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-08T21:21:00.257 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-08T21:21:00.257 INFO:tasks.ceph:role: osd.4 2024-05-08T21:21:00.257 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com 2024-05-08T21:21:00.258 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-08T21:21:00.337 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T21:21:00.337 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T21:21:00.337 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T21:21:00.337 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-08T21:21:00.337 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T21:21:00.337 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-05-08T21:21:00.337 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T21:21:00.337 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-08T21:21:00.337 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T21:21:00.338 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T21:21:00.781 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-05-08T21:21:00.783 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-05-08T21:21:00.784 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-05-08T21:21:00.971 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-08T21:21:01.007 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-08T21:21:01.076 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-08T21:21:01.076 INFO:tasks.ceph:role: osd.5 2024-05-08T21:21:01.076 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com 2024-05-08T21:21:01.076 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-08T21:21:01.152 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T21:21:01.152 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T21:21:01.152 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T21:21:01.152 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-08T21:21:01.152 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T21:21:01.153 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-05-08T21:21:01.153 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T21:21:01.153 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-08T21:21:01.153 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T21:21:01.153 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T21:21:01.566 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-05-08T21:21:01.569 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-05-08T21:21:01.569 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-05-08T21:21:01.677 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-08T21:21:01.711 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-05-08T21:21:01.780 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-08T21:21:01.780 INFO:tasks.ceph:role: osd.6 2024-05-08T21:21:01.780 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com 2024-05-08T21:21:01.780 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-08T21:21:01.855 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T21:21:01.855 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T21:21:01.855 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T21:21:01.856 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-08T21:21:01.856 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T21:21:01.856 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-05-08T21:21:01.856 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T21:21:01.856 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-08T21:21:01.856 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T21:21:01.856 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T21:21:02.304 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-05-08T21:21:02.306 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-05-08T21:21:02.306 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-05-08T21:21:02.480 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-05-08T21:21:02.515 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-05-08T21:21:02.585 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-08T21:21:02.585 INFO:tasks.ceph:role: osd.7 2024-05-08T21:21:02.585 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi186.front.sepia.ceph.com 2024-05-08T21:21:02.585 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-08T21:21:02.661 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T21:21:02.662 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T21:21:02.662 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T21:21:02.662 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-08T21:21:02.662 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T21:21:02.662 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-05-08T21:21:02.662 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T21:21:02.662 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-08T21:21:02.662 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T21:21:02.662 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T21:21:03.095 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-05-08T21:21:03.097 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-05-08T21:21:03.097 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-05-08T21:21:03.287 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-05-08T21:21:03.325 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T21:21:03.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:03.413+0000 7fb1d43527c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-08T21:21:03.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:03.414+0000 7fb1d43527c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-08T21:21:03.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:03.414+0000 7fb1d43527c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-08T21:21:03.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:03.414+0000 7fb1d43527c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-08T21:21:03.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:03.414+0000 7fb1d43527c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-08T21:21:03.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:03.414+0000 7fb1d43527c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-08T21:21:06.275 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-08T21:21:06.307 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T21:21:06.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:06.399+0000 7f5281e107c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-08T21:21:06.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:06.399+0000 7f5281e107c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-08T21:21:06.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:06.400+0000 7f5281e107c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-08T21:21:06.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:06.400+0000 7f5281e107c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-08T21:21:06.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:06.400+0000 7f5281e107c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-08T21:21:06.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:06.400+0000 7f5281e107c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-08T21:21:09.251 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-08T21:21:09.283 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T21:21:09.376 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:09.374+0000 7fe152f587c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-05-08T21:21:09.376 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:09.374+0000 7fe152f587c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-05-08T21:21:09.376 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:09.374+0000 7fe152f587c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-08T21:21:09.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:09.374+0000 7fe152f587c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-08T21:21:09.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:09.374+0000 7fe152f587c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-08T21:21:09.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:09.375+0000 7fe152f587c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-05-08T21:21:11.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-08T21:21:11.655 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-08T21:21:12.248 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-08T21:21:12.278 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T21:21:12.370 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:12.368+0000 7fdd436dd7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-05-08T21:21:12.370 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:12.368+0000 7fdd436dd7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-05-08T21:21:12.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:12.368+0000 7fdd436dd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-08T21:21:12.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:12.368+0000 7fdd436dd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-08T21:21:12.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:12.368+0000 7fdd436dd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-08T21:21:12.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:12.369+0000 7fdd436dd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-05-08T21:21:15.226 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-08T21:21:15.256 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-08T21:21:15.256 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:21:15.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-05-08T21:21:15.291 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:21:15.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-05-08T21:21:15.366 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:21:15.367 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-05-08T21:21:15.441 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:21:15.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-08T21:21:15.515 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:21:15.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-08T21:21:15.590 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:21:15.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-08T21:21:15.665 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:21:15.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-08T21:21:15.739 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:21:15.739 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-08T21:21:15.770 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:21:15.771 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-05-08T21:21:15.841 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:21:15.841 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-05-08T21:21:15.911 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:21:15.911 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-08T21:21:15.979 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:21:15.979 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-08T21:21:16.047 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:21:16.047 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-05-08T21:21:16.115 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:21:16.115 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-05-08T21:21:16.181 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:21:16.181 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-08T21:21:16.203 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:21:16.203 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-05-08T21:21:16.265 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:21:16.265 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-05-08T21:21:16.282 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:21:16.282 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-05-08T21:21:16.339 INFO:tasks.ceph:Adding keys to all mons... 2024-05-08T21:21:16.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-08T21:21:16.342 DEBUG:teuthology.orchestra.run.smithi186:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-08T21:21:16.382 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.y] 2024-05-08T21:21:16.382 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCs7DtmF9wvFRAArRqZRUxiZFut4m22n8Vw+A== 2024-05-08T21:21:16.382 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.a] 2024-05-08T21:21:16.383 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCs7Dtmcb2tHRAA2QzDU8WLD64M0ONuKl5Iog== 2024-05-08T21:21:16.383 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.c] 2024-05-08T21:21:16.383 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCs7Dtmaw4NJhAAWdciLu9JWl39vhcVAdnBHw== 2024-05-08T21:21:16.383 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.0] 2024-05-08T21:21:16.383 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCw7Dtm250HFBAAS7AJ95J560z3oTTa9CFbOw== 2024-05-08T21:21:16.383 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.1] 2024-05-08T21:21:16.383 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCz7DtmiR9oFBAAyJXjJxS/AtGBybRyKVoSAQ== 2024-05-08T21:21:16.383 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.2] 2024-05-08T21:21:16.383 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQC27Dtm8XYHFBAAhVst/mX9SsJAEA5fFUA6Kw== 2024-05-08T21:21:16.383 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.3] 2024-05-08T21:21:16.383 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQC57Dtm0rizExAA68YZqUs2zq1tBwVb8Q+tvw== 2024-05-08T21:21:16.383 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.x] 2024-05-08T21:21:16.383 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCs7DtmYF5dGRAAmBIxydvBNs7D+BQ5Gn2pSw== 2024-05-08T21:21:16.383 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.b] 2024-05-08T21:21:16.384 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCs7DtmVPZqKxAAp4iAjTWYy+DPBtuOQFNIXw== 2024-05-08T21:21:16.384 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.d] 2024-05-08T21:21:16.384 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCs7Dtmw3nKMhAAGtM2BT5AWuVFdFTRwte3IQ== 2024-05-08T21:21:16.384 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.4] 2024-05-08T21:21:16.384 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQC/7DtmA3+4GBAAaye0ak/pWNIdDz+UnzNvLA== 2024-05-08T21:21:16.384 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.5] 2024-05-08T21:21:16.384 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDC7DtmRpbhFxAAIKofN/JzliaxKIXZLVGw5A== 2024-05-08T21:21:16.384 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.6] 2024-05-08T21:21:16.384 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDF7DtmvB1hFhAAQowb3SH+rt7zIMK4Dz8FdA== 2024-05-08T21:21:16.384 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.7] 2024-05-08T21:21:16.384 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDI7Dtm/8cFFhAAe+fAJVIoH1I9pwwnrqRssg== 2024-05-08T21:21:16.384 INFO:teuthology.orchestra.run.smithi003.stdout:[client.0] 2024-05-08T21:21:16.384 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCs7DtmLtjINxAA2457rrNLNP06VzQmA55MMA== 2024-05-08T21:21:16.384 INFO:teuthology.orchestra.run.smithi003.stdout:[client.2] 2024-05-08T21:21:16.385 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCt7Dtm3idSARAAdMThdnq8J9CBIBB4Ja0zgQ== 2024-05-08T21:21:16.385 INFO:teuthology.orchestra.run.smithi003.stdout:[client.1] 2024-05-08T21:21:16.385 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCt7DtmqYziBBAAOgnWbP2XqOMqQDiranNyjQ== 2024-05-08T21:21:16.385 INFO:teuthology.orchestra.run.smithi003.stdout:[client.3] 2024-05-08T21:21:16.385 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCt7DtmLyo2CBAAy/Em6jfx4vDVKXAeEhAO/Q== 2024-05-08T21:21:16.405 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr.y] 2024-05-08T21:21:16.405 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQCs7DtmF9wvFRAArRqZRUxiZFut4m22n8Vw+A== 2024-05-08T21:21:16.405 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.a] 2024-05-08T21:21:16.405 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQCs7Dtmcb2tHRAA2QzDU8WLD64M0ONuKl5Iog== 2024-05-08T21:21:16.405 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.c] 2024-05-08T21:21:16.405 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQCs7Dtmaw4NJhAAWdciLu9JWl39vhcVAdnBHw== 2024-05-08T21:21:16.406 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.0] 2024-05-08T21:21:16.406 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQCw7Dtm250HFBAAS7AJ95J560z3oTTa9CFbOw== 2024-05-08T21:21:16.406 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.1] 2024-05-08T21:21:16.406 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQCz7DtmiR9oFBAAyJXjJxS/AtGBybRyKVoSAQ== 2024-05-08T21:21:16.406 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.2] 2024-05-08T21:21:16.406 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQC27Dtm8XYHFBAAhVst/mX9SsJAEA5fFUA6Kw== 2024-05-08T21:21:16.406 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.3] 2024-05-08T21:21:16.406 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQC57Dtm0rizExAA68YZqUs2zq1tBwVb8Q+tvw== 2024-05-08T21:21:16.406 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr.x] 2024-05-08T21:21:16.406 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQCs7DtmYF5dGRAAmBIxydvBNs7D+BQ5Gn2pSw== 2024-05-08T21:21:16.406 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.b] 2024-05-08T21:21:16.406 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQCs7DtmVPZqKxAAp4iAjTWYy+DPBtuOQFNIXw== 2024-05-08T21:21:16.406 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.d] 2024-05-08T21:21:16.407 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQCs7Dtmw3nKMhAAGtM2BT5AWuVFdFTRwte3IQ== 2024-05-08T21:21:16.407 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.4] 2024-05-08T21:21:16.407 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQC/7DtmA3+4GBAAaye0ak/pWNIdDz+UnzNvLA== 2024-05-08T21:21:16.407 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.5] 2024-05-08T21:21:16.407 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDC7DtmRpbhFxAAIKofN/JzliaxKIXZLVGw5A== 2024-05-08T21:21:16.407 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.6] 2024-05-08T21:21:16.407 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDF7DtmvB1hFhAAQowb3SH+rt7zIMK4Dz8FdA== 2024-05-08T21:21:16.407 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.7] 2024-05-08T21:21:16.407 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDI7Dtm/8cFFhAAe+fAJVIoH1I9pwwnrqRssg== 2024-05-08T21:21:16.407 INFO:teuthology.orchestra.run.smithi186.stdout:[client.0] 2024-05-08T21:21:16.407 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQCs7DtmLtjINxAA2457rrNLNP06VzQmA55MMA== 2024-05-08T21:21:16.407 INFO:teuthology.orchestra.run.smithi186.stdout:[client.2] 2024-05-08T21:21:16.407 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQCt7Dtm3idSARAAdMThdnq8J9CBIBB4Ja0zgQ== 2024-05-08T21:21:16.407 INFO:teuthology.orchestra.run.smithi186.stdout:[client.1] 2024-05-08T21:21:16.407 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQCt7DtmqYziBBAAOgnWbP2XqOMqQDiranNyjQ== 2024-05-08T21:21:16.408 INFO:teuthology.orchestra.run.smithi186.stdout:[client.3] 2024-05-08T21:21:16.408 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQCt7DtmLyo2CBAAy/Em6jfx4vDVKXAeEhAO/Q== 2024-05-08T21:21:16.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-08T21:21:16.428 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-08T21:21:16.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-08T21:21:16.504 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-08T21:21:16.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-08T21:21:16.570 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-08T21:21:16.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T21:21:16.634 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T21:21:16.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T21:21:16.694 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T21:21:16.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T21:21:16.754 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T21:21:16.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T21:21:16.813 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T21:21:16.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-08T21:21:16.872 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-08T21:21:16.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-08T21:21:16.931 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-08T21:21:16.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-08T21:21:16.994 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-08T21:21:17.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T21:21:17.057 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T21:21:17.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T21:21:17.117 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T21:21:17.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T21:21:17.177 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T21:21:17.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T21:21:17.236 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T21:21:17.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T21:21:17.297 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T21:21:17.358 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T21:21:17.360 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T21:21:17.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T21:21:17.423 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T21:21:17.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T21:21:17.487 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T21:21:17.548 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-08T21:21:17.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-08T21:21:17.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-08T21:21:17.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-08T21:21:17.717 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-08T21:21:17.745 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-08T21:21:17.846 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-08T21:21:17.873 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-08T21:21:17.940 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-08T21:21:18.039 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-08T21:21:18.066 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-08T21:21:18.068 DEBUG:teuthology.orchestra.run.smithi186:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-08T21:21:18.123 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-08T21:21:18.123 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-08T21:21:18.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-08T21:21:18.130 INFO:tasks.ceph.mon.a:Started 2024-05-08T21:21:18.130 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-08T21:21:18.130 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-08T21:21:18.165 INFO:tasks.ceph.mon.b:Started 2024-05-08T21:21:18.165 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-08T21:21:18.166 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-08T21:21:18.167 INFO:tasks.ceph.mon.c:Started 2024-05-08T21:21:18.167 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-08T21:21:18.167 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-05-08T21:21:18.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-05-08T21:21:18.170 INFO:tasks.ceph.mgr.y:Started 2024-05-08T21:21:18.170 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-08T21:21:18.170 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-05-08T21:21:18.172 INFO:tasks.ceph.mgr.x:Started 2024-05-08T21:21:18.172 INFO:tasks.ceph:Setting crush tunables to default 2024-05-08T21:21:18.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd crush tunables default 2024-05-08T21:21:18.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.311+0000 7f86d5f4a700 1 Processor -- start 2024-05-08T21:21:18.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.311+0000 7f86d5f4a700 1 -- start start 2024-05-08T21:21:18.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86d5f4a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 0x7f86d006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:18.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86d5f4a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 0x7f86d01614f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:18.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86d5f4a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d006d7f0 con 0x7f86d0161a30 2024-05-08T21:21:18.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86d5f4a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d00ce850 con 0x7f86d015d770 2024-05-08T21:21:18.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86d5f4a700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d00ce240 con 0x7f86d006c770 2024-05-08T21:21:18.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86cf7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 0x7f86d006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:18.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86cf7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 0x7f86d006c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49744/0 (socket says 172.21.15.3:49744) 2024-05-08T21:21:18.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86cffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f86d0161a30 0x7f86d00cece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:36200/0 (socket says 172.21.15.3:36200) 2024-05-08T21:21:18.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 learned_addr learned my addr 172.21.15.3:0/1933138900 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:18.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 0x7f86d01614f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:18.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 msgr2=0x7f86d006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-08T21:21:18.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 msgr2=0x7f86d006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:18.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 0x7f86d006c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:18.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 0x7f86d006c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-08T21:21:18.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86ceffd700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 msgr2=0x7f86d01614f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-08T21:21:18.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86ceffd700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 msgr2=0x7f86d01614f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:18.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 0x7f86d01614f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:18.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.312+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 0x7f86d01614f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-08T21:21:18.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.513+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 0x7f86d006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:18.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.513+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 msgr2=0x7f86d006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-08T21:21:18.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.513+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 msgr2=0x7f86d006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:18.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.513+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 0x7f86d006c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:18.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.513+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 0x7f86d006c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-08T21:21:18.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.513+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 0x7f86d01614f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:18.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.513+0000 7f86ceffd700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 msgr2=0x7f86d01614f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-08T21:21:18.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.513+0000 7f86ceffd700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 msgr2=0x7f86d01614f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:18.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.513+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 0x7f86d01614f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:18.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.513+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 0x7f86d01614f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-08T21:21:18.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.914+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 0x7f86d006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:18.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.914+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 0x7f86d01614f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:18.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.914+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 msgr2=0x7f86d006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-08T21:21:18.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.914+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 msgr2=0x7f86d006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:18.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.914+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 0x7f86d006c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:18.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.914+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 0x7f86d006c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-08T21:21:18.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.914+0000 7f86ceffd700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 msgr2=0x7f86d01614f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-08T21:21:18.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.914+0000 7f86ceffd700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 msgr2=0x7f86d01614f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:18.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.914+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 0x7f86d01614f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:18.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:18.914+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 0x7f86d01614f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-08T21:21:19.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:19.716+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 0x7f86d006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:19.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:19.716+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 msgr2=0x7f86d006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-08T21:21:19.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:19.716+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 msgr2=0x7f86d006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:19.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:19.716+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 0x7f86d006c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:19.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:19.716+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 0x7f86d006c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-08T21:21:19.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:19.716+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 0x7f86d01614f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:19.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:19.716+0000 7f86ceffd700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 msgr2=0x7f86d01614f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-08T21:21:19.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:19.716+0000 7f86ceffd700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 msgr2=0x7f86d01614f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:19.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:19.716+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 0x7f86d01614f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:19.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:19.716+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 0x7f86d01614f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-08T21:21:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.312+0000 7f86cd7fa700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 msgr2=0x7f86d01614f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-08T21:21:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.312+0000 7f86cd7fa700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86d006c770 0x7f86d01614f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.312+0000 7f86cd7fa700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 msgr2=0x7f86d006c230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-08T21:21:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.312+0000 7f86cd7fa700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86d015d770 0x7f86d006c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.312+0000 7f86cd7fa700 1 -- 172.21.15.3:0/1933138900 >> v1:172.21.15.3:6789/0 conn(0x7f86d0161a30 legacy=0x7f86d00cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.312+0000 7f86cd7fa700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 0x7f86b0005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.312+0000 7f86cd7fa700 1 -- 172.21.15.3:0/1933138900 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d0161f40 con 0x7f86b0006200 2024-05-08T21:21:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.312+0000 7f86cd7fa700 1 -- 172.21.15.3:0/1933138900 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d00ce850 con 0x7f86b0003740 2024-05-08T21:21:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.313+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 0x7f86b0005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.313+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-08T21:21:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.313+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:21.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.313+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 0x7f86b0005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:21.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.313+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 0x7f86b0005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-08T21:21:21.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.514+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 0x7f86b0005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:21.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.514+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-08T21:21:21.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.514+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:21.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.514+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 0x7f86b0005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:21.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.514+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 0x7f86b0005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-08T21:21:21.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.915+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 0x7f86b0005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:21.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.915+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-08T21:21:21.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.915+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:21.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.915+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 0x7f86b0005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:21.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:21.915+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 0x7f86b0005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-08T21:21:22.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:22.716+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 0x7f86b0005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:22.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:22.717+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-08T21:21:22.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:22.717+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:22.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:22.717+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 0x7f86b0005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:22.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:22.717+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 0x7f86b0005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-08T21:21:24.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.312+0000 7f86cd7fa700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-08T21:21:24.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.312+0000 7f86cd7fa700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0003740 0x7f86b0005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:24.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.312+0000 7f86cd7fa700 1 -- 172.21.15.3:0/1933138900 >> v1:172.21.15.186:6789/0 conn(0x7f86b0006200 legacy=0x7f86b00086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:24.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.313+0000 7f86cffff700 1 -- 172.21.15.3:0/1933138900 reap_dead start 2024-05-08T21:21:24.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.313+0000 7f86cd7fa700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 0x7f86b0005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:24.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.313+0000 7f86cd7fa700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 0x7f86b00086a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:24.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.313+0000 7f86cd7fa700 1 -- 172.21.15.3:0/1933138900 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b000b030 con 0x7f86d0161a30 2024-05-08T21:21:24.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.313+0000 7f86cd7fa700 1 -- 172.21.15.3:0/1933138900 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d00ce850 con 0x7f86b000a780 2024-05-08T21:21:24.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.313+0000 7f86cd7fa700 1 -- 172.21.15.3:0/1933138900 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d00ce240 con 0x7f86b0008dd0 2024-05-08T21:21:24.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.313+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 0x7f86b0005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:24.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.313+0000 7f86cffff700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 0x7f86b00086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:24.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.313+0000 7f86ceffd700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-08T21:21:24.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.313+0000 7f86ceffd700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:24.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.313+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 0x7f86b0005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:24.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.313+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 0x7f86b0005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-08T21:21:24.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.313+0000 7f86cffff700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 msgr2=0x7f86b00086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-08T21:21:24.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.313+0000 7f86cffff700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 msgr2=0x7f86b00086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:24.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.313+0000 7f86cffff700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 0x7f86b00086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:24.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.313+0000 7f86cffff700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 0x7f86b00086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-08T21:21:24.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.514+0000 7f86cffff700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 0x7f86b00086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:24.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.514+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 0x7f86b0005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:24.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.514+0000 7f86cffff700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 msgr2=0x7f86b00086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-08T21:21:24.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.514+0000 7f86cffff700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 msgr2=0x7f86b00086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:24.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.514+0000 7f86cffff700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 0x7f86b00086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:24.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.514+0000 7f86ceffd700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-08T21:21:24.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.514+0000 7f86ceffd700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:24.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.514+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 0x7f86b0005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:24.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.514+0000 7f86cffff700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 0x7f86b00086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-08T21:21:24.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.514+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 0x7f86b0005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-08T21:21:24.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.915+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 0x7f86b0005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:24.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.915+0000 7f86cffff700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 0x7f86b00086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:24.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.915+0000 7f86ceffd700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-08T21:21:24.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.915+0000 7f86ceffd700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:24.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.915+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 0x7f86b0005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:24.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.915+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 0x7f86b0005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-08T21:21:24.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.915+0000 7f86cffff700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 msgr2=0x7f86b00086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-08T21:21:24.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.915+0000 7f86cffff700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 msgr2=0x7f86b00086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:24.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.915+0000 7f86cffff700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 0x7f86b00086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:24.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:24.915+0000 7f86cffff700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 0x7f86b00086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-08T21:21:25.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:25.717+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 0x7f86b0005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:25.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:25.717+0000 7f86cffff700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 0x7f86b00086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:25.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:25.717+0000 7f86ceffd700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-08T21:21:25.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:25.717+0000 7f86ceffd700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:25.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:25.717+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 0x7f86b0005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:25.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:25.717+0000 7f86cffff700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 msgr2=0x7f86b00086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-08T21:21:25.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:25.717+0000 7f86cffff700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 msgr2=0x7f86b00086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:25.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:25.717+0000 7f86cffff700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 0x7f86b00086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:25.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:25.717+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 0x7f86b0005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-08T21:21:25.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:25.717+0000 7f86cffff700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 0x7f86b00086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-08T21:21:27.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.312+0000 7f86cd7fa700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 msgr2=0x7f86b00086a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-08T21:21:27.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.312+0000 7f86cd7fa700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b0008dd0 0x7f86b00086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.312+0000 7f86cd7fa700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 msgr2=0x7f86b0005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-08T21:21:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.312+0000 7f86cd7fa700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.186:3300/0 conn(0x7f86b000a780 0x7f86b0005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.312+0000 7f86cd7fa700 1 -- 172.21.15.3:0/1933138900 >> v1:172.21.15.3:6789/0 conn(0x7f86d0161a30 legacy=0x7f86b000a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.312+0000 7f86cd7fa700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 0x7f86b00098d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.312+0000 7f86cd7fa700 1 -- 172.21.15.3:0/1933138900 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b0009280 con 0x7f86b0010430 2024-05-08T21:21:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.312+0000 7f86cd7fa700 1 -- 172.21.15.3:0/1933138900 --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d00ce850 con 0x7f86b00094c0 2024-05-08T21:21:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.312+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 0x7f86b00098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.312+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 msgr2=0x7f86b00098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-08T21:21:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.312+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 msgr2=0x7f86b00098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.312+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 0x7f86b00098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.312+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 0x7f86b00098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-08T21:21:27.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.513+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 0x7f86b00098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:27.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.513+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 msgr2=0x7f86b00098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-08T21:21:27.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.513+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 msgr2=0x7f86b00098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:27.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.513+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 0x7f86b00098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:27.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.513+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 0x7f86b00098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-08T21:21:27.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.914+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 0x7f86b00098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:27.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.914+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 msgr2=0x7f86b00098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-08T21:21:27.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.914+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 msgr2=0x7f86b00098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-08T21:21:27.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.914+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 0x7f86b00098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-08T21:21:27.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:27.914+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 0x7f86b00098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-08T21:21:28.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.570+0000 7f86ce7fc700 1 -- 172.21.15.3:0/1933138900 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667861602 0 0) 0x7f86b0009280 con 0x7f86b0010430 2024-05-08T21:21:28.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.570+0000 7f86ce7fc700 1 -- 172.21.15.3:0/1933138900 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86bc0035b0 con 0x7f86b0010430 2024-05-08T21:21:28.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.573+0000 7f86ce7fc700 1 -- 172.21.15.3:0/1933138900 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 540448583 0 0) 0x7f86bc0035b0 con 0x7f86b0010430 2024-05-08T21:21:28.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.573+0000 7f86ce7fc700 1 -- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 msgr2=0x7f86b00098d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-08T21:21:28.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.573+0000 7f86ce7fc700 1 --2- 172.21.15.3:0/1933138900 >> v2:172.21.15.3:3300/0 conn(0x7f86b00094c0 0x7f86b00098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:28.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86ce7fc700 1 -- 172.21.15.3:0/1933138900 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b0015180 con 0x7f86b0010430 2024-05-08T21:21:28.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86ce7fc700 1 -- 172.21.15.3:0/1933138900 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f86b8016710 con 0x7f86b0010430 2024-05-08T21:21:28.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86ce7fc700 1 -- 172.21.15.3:0/1933138900 >> v1:172.21.15.186:6789/0 conn(0x7f86b0010430 legacy=0x7f86b0012890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:28.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86cffff700 1 -- 172.21.15.3:0/1933138900 reap_dead start 2024-05-08T21:21:28.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86ce7fc700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86bc003a00 0x7f86bc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:28.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86ce7fc700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86bc004850 0x7f86bc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:28.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86ce7fc700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86bc0052e0 0x7f86bc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:28.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86ce7fc700 1 -- 172.21.15.3:0/1933138900 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d00ce850 con 0x7f86bc004850 2024-05-08T21:21:28.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86ce7fc700 1 -- 172.21.15.3:0/1933138900 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc00c230 con 0x7f86bc003a00 2024-05-08T21:21:28.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86ce7fc700 1 -- 172.21.15.3:0/1933138900 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc00c3f0 con 0x7f86bc0052e0 2024-05-08T21:21:28.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86ceffd700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86bc004850 0x7f86bc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:28.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86bc003a00 0x7f86bc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:28.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86cffff700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86bc0052e0 0x7f86bc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:28.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86bc0052e0 msgr2=0x7f86bc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:28.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86bc0052e0 0x7f86bc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:28.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86bc004850 msgr2=0x7f86bc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:28.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86bc004850 0x7f86bc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:28.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.574+0000 7f86cf7fe700 1 -- 172.21.15.3:0/1933138900 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86bc0035b0 con 0x7f86bc003a00 2024-05-08T21:21:28.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.575+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86bc003a00 0x7f86bc003e00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f86c0006fa0 tx=0x7f86bc005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=413d4aa0b8e8fd69 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:28.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.575+0000 7f86ce7fc700 1 -- 172.21.15.3:0/1933138900 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0018850 con 0x7f86bc003a00 2024-05-08T21:21:28.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.575+0000 7f86ce7fc700 1 -- 172.21.15.3:0/1933138900 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c001e070 con 0x7f86bc003a00 2024-05-08T21:21:28.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.575+0000 7f86ce7fc700 1 -- 172.21.15.3:0/1933138900 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0018bf0 con 0x7f86bc003a00 2024-05-08T21:21:28.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.576+0000 7f86d5f4a700 1 -- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86bc003a00 msgr2=0x7f86bc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:28.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.576+0000 7f86d5f4a700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86bc003a00 0x7f86bc003e00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f86c0006fa0 tx=0x7f86bc005880 comp rx=0 tx=0).stop 2024-05-08T21:21:28.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.576+0000 7f86d5f4a700 1 -- 172.21.15.3:0/1933138900 shutdown_connections 2024-05-08T21:21:28.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.576+0000 7f86d5f4a700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86bc0052e0 0x7f86bc00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:28.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.576+0000 7f86d5f4a700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86bc004850 0x7f86bc004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:28.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.577+0000 7f86d5f4a700 1 --2- 172.21.15.3:0/1933138900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86bc003a00 0x7f86bc003e00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:28.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.577+0000 7f86d5f4a700 1 -- 172.21.15.3:0/1933138900 >> 172.21.15.3:0/1933138900 conn(0x7f86d008d070 msgr2=0x7f86d009c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:28.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.577+0000 7f86d5f4a700 1 -- 172.21.15.3:0/1933138900 shutdown_connections 2024-05-08T21:21:28.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.577+0000 7f86d5f4a700 1 -- 172.21.15.3:0/1933138900 wait complete. 2024-05-08T21:21:28.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.578+0000 7f86d5f4a700 1 Processor -- start 2024-05-08T21:21:28.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.579+0000 7f86d5f4a700 1 -- start start 2024-05-08T21:21:28.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.580+0000 7f86d5f4a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86bc003a00 0x7f86d00b8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:28.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.580+0000 7f86d5f4a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86bc004850 0x7f86d00b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:28.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.580+0000 7f86d5f4a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86bc0052e0 0x7f86d00b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:28.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.580+0000 7f86d5f4a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d01f4180 con 0x7f86bc003a00 2024-05-08T21:21:28.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.580+0000 7f86d5f4a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d00b9510 con 0x7f86bc004850 2024-05-08T21:21:28.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.580+0000 7f86d5f4a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d01f3ea0 con 0x7f86bc0052e0 2024-05-08T21:21:28.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.580+0000 7f86cf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86bc003a00 0x7f86d00b8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:28.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.580+0000 7f86cf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86bc003a00 0x7f86d00b8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45852/0 (socket says 172.21.15.3:45852) 2024-05-08T21:21:28.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.580+0000 7f86cf7fe700 1 -- 172.21.15.3:0/4018030971 learned_addr learned my addr 172.21.15.3:0/4018030971 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:28.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.580+0000 7f86cffff700 1 --2- 172.21.15.3:0/4018030971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86bc0052e0 0x7f86d00b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:28.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.580+0000 7f86ceffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86bc004850 0x7f86d00b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:28.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.581+0000 7f86cf7fe700 1 -- 172.21.15.3:0/4018030971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86bc0052e0 msgr2=0x7f86d00b56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:28.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.581+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/4018030971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86bc0052e0 0x7f86d00b56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:28.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.581+0000 7f86cf7fe700 1 -- 172.21.15.3:0/4018030971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86bc004850 msgr2=0x7f86d00b51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:28.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.581+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/4018030971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86bc004850 0x7f86d00b51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:28.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.581+0000 7f86cf7fe700 1 -- 172.21.15.3:0/4018030971 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d00b0390 con 0x7f86bc003a00 2024-05-08T21:21:28.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.581+0000 7f86ceffd700 1 --2- 172.21.15.3:0/4018030971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86bc004850 0x7f86d00b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:21:28.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.581+0000 7f86cf7fe700 1 --2- 172.21.15.3:0/4018030971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86bc003a00 0x7f86d00b8fd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f86c00040b0 tx=0x7f86c00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:28.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.581+0000 7f86c7fff700 1 -- 172.21.15.3:0/4018030971 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0006240 con 0x7f86bc003a00 2024-05-08T21:21:28.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.581+0000 7f86c7fff700 1 -- 172.21.15.3:0/4018030971 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c000c040 con 0x7f86bc003a00 2024-05-08T21:21:28.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.581+0000 7f86c7fff700 1 -- 172.21.15.3:0/4018030971 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0006600 con 0x7f86bc003a00 2024-05-08T21:21:28.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.581+0000 7f86d5f4a700 1 -- 172.21.15.3:0/4018030971 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d00b0670 con 0x7f86bc003a00 2024-05-08T21:21:28.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.581+0000 7f86d5f4a700 1 -- 172.21.15.3:0/4018030971 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d00b0e70 con 0x7f86bc003a00 2024-05-08T21:21:28.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.581+0000 7f86c7fff700 1 -- 172.21.15.3:0/4018030971 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f86c0006760 con 0x7f86bc003a00 2024-05-08T21:21:28.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.581+0000 7f86c7fff700 1 -- 172.21.15.3:0/4018030971 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f86c0000e40 con 0x7f86bc003a00 2024-05-08T21:21:28.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.582+0000 7f86d5f4a700 1 -- 172.21.15.3:0/4018030971 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d00b01f0 con 0x7f86bc003a00 2024-05-08T21:21:28.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.587+0000 7f86c7fff700 1 -- 172.21.15.3:0/4018030971 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f86d00b01f0 con 0x7f86bc003a00 2024-05-08T21:21:28.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.669+0000 7f86d5f4a700 1 -- 172.21.15.3:0/4018030971 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f86d0074bc0 con 0x7f86bc003a00 2024-05-08T21:21:28.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.672+0000 7f86c7fff700 1 -- 172.21.15.3:0/4018030971 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f86c0047090 con 0x7f86bc003a00 2024-05-08T21:21:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.673+0000 7f86d5f4a700 1 -- 172.21.15.3:0/4018030971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86bc003a00 msgr2=0x7f86d00b8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.673+0000 7f86d5f4a700 1 --2- 172.21.15.3:0/4018030971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86bc003a00 0x7f86d00b8fd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f86c00040b0 tx=0x7f86c00047c0 comp rx=0 tx=0).stop 2024-05-08T21:21:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.673+0000 7f86d5f4a700 1 -- 172.21.15.3:0/4018030971 shutdown_connections 2024-05-08T21:21:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.673+0000 7f86d5f4a700 1 --2- 172.21.15.3:0/4018030971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86bc0052e0 0x7f86d00b56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.673+0000 7f86d5f4a700 1 --2- 172.21.15.3:0/4018030971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86bc003a00 0x7f86d00b8fd0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.673+0000 7f86d5f4a700 1 --2- 172.21.15.3:0/4018030971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86bc004850 0x7f86d00b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.673+0000 7f86d5f4a700 1 -- 172.21.15.3:0/4018030971 >> 172.21.15.3:0/4018030971 conn(0x7f86d008d070 msgr2=0x7f86d015ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.673+0000 7f86d5f4a700 1 -- 172.21.15.3:0/4018030971 shutdown_connections 2024-05-08T21:21:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:28.673+0000 7f86d5f4a700 1 -- 172.21.15.3:0/4018030971 wait complete. 2024-05-08T21:21:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2024-05-08T21:21:28.695 INFO:tasks.ceph:check_enable_crimson: False 2024-05-08T21:21:28.695 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-08T21:21:28.695 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:21:28.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-08T21:21:28.725 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:21:28.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-08T21:21:28.795 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:21:28.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-08T21:21:28.865 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:21:28.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-08T21:21:28.935 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:21:28.935 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-08T21:21:28.961 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:21:28.961 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-08T21:21:29.028 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:21:29.028 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-05-08T21:21:29.057 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:29.055+0000 7fb71cb9a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-08T21:21:29.095 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:21:29.095 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-05-08T21:21:29.102 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:29.101+0000 7f1f5b89f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-08T21:21:29.128 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new d4e85793-ae1b-4a72-a1a8-2f1e299373ea 0 2024-05-08T21:21:29.147 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:29.144+0000 7fb71cb9a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-08T21:21:29.186 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:29.185+0000 7f1f5b89f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-08T21:21:29.230 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:29.228+0000 7fb71cb9a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-08T21:21:29.265 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:29.263+0000 7f1f5b89f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-08T21:21:29.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.288+0000 7fcf3436a700 1 Processor -- start 2024-05-08T21:21:29.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.289+0000 7fcf3436a700 1 -- start start 2024-05-08T21:21:29.293 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.289+0000 7fcf3436a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcf2c15dc60 0x7fcf2c0ad9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:29.293 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.289+0000 7fcf3436a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf2c072170 0x7fcf2c0adf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:29.293 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.289+0000 7fcf3436a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcf2c0ae450 0x7fcf2c0b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:29.293 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.289+0000 7fcf3436a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2c0ae8e0 con 0x7fcf2c15dc60 2024-05-08T21:21:29.293 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.289+0000 7fcf3436a700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2c0aed90 con 0x7fcf2c0ae450 2024-05-08T21:21:29.293 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.289+0000 7fcf3436a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c06dcd0 con 0x7fcf2c072170 2024-05-08T21:21:29.293 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.290+0000 7fcf32907700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcf2c0ae450 0x7fcf2c0b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.290+0000 7fcf31905700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf2c072170 0x7fcf2c0adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.290+0000 7fcf32907700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcf2c0ae450 0x7fcf2c0b3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:35530/0 (socket says 172.21.15.186:35530) 2024-05-08T21:21:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.290+0000 7fcf31905700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf2c072170 0x7fcf2c0adf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:34606/0 (socket says 172.21.15.186:34606) 2024-05-08T21:21:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.290+0000 7fcf32907700 1 -- 172.21.15.186:0/695500458 learned_addr learned my addr 172.21.15.186:0/695500458 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-08T21:21:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.290+0000 7fcf32106700 1 --2- 172.21.15.186:0/695500458 >> v2:172.21.15.3:3300/0 conn(0x7fcf2c15dc60 0x7fcf2c0ad9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.290+0000 7fcf32907700 1 -- 172.21.15.186:0/695500458 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf2c072170 msgr2=0x7fcf2c0adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.290+0000 7fcf32907700 1 --2- 172.21.15.186:0/695500458 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf2c072170 0x7fcf2c0adf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.290+0000 7fcf32907700 1 -- 172.21.15.186:0/695500458 >> v2:172.21.15.3:3300/0 conn(0x7fcf2c15dc60 msgr2=0x7fcf2c0ad9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.290+0000 7fcf32907700 1 --2- 172.21.15.186:0/695500458 >> v2:172.21.15.3:3300/0 conn(0x7fcf2c15dc60 0x7fcf2c0ad9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.290+0000 7fcf32907700 1 -- 172.21.15.186:0/695500458 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2c06e290 con 0x7fcf2c0ae450 2024-05-08T21:21:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.290+0000 7fcf31905700 1 --2- 172.21.15.186:0/695500458 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf2c072170 0x7fcf2c0adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:21:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.290+0000 7fcf32907700 1 --2- 172.21.15.186:0/695500458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf2c0ae450 0x7fcf2c0b3be0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c071bc0 tx=0x7fcf2400d430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1892e29dc46cb688 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.290+0000 7fcf31104700 1 -- 172.21.15.186:0/695500458 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf24010070 con 0x7fcf2c0ae450 2024-05-08T21:21:29.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.290+0000 7fcf31104700 1 -- 172.21.15.186:0/695500458 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2400dce0 con 0x7fcf2c0ae450 2024-05-08T21:21:29.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.290+0000 7fcf31104700 1 -- 172.21.15.186:0/695500458 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2400b310 con 0x7fcf2c0ae450 2024-05-08T21:21:29.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.291+0000 7fcf3436a700 1 -- 172.21.15.186:0/695500458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf2c0ae450 msgr2=0x7fcf2c0b3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:29.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.291+0000 7fcf3436a700 1 --2- 172.21.15.186:0/695500458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf2c0ae450 0x7fcf2c0b3be0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c071bc0 tx=0x7fcf2400d430 comp rx=0 tx=0).stop 2024-05-08T21:21:29.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.291+0000 7fcf3436a700 1 -- 172.21.15.186:0/695500458 shutdown_connections 2024-05-08T21:21:29.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.291+0000 7fcf3436a700 1 --2- 172.21.15.186:0/695500458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf2c0ae450 0x7fcf2c0b3be0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.291+0000 7fcf3436a700 1 --2- 172.21.15.186:0/695500458 >> v2:172.21.15.3:3300/0 conn(0x7fcf2c15dc60 0x7fcf2c0ad9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.291+0000 7fcf3436a700 1 --2- 172.21.15.186:0/695500458 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf2c072170 0x7fcf2c0adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.291+0000 7fcf3436a700 1 -- 172.21.15.186:0/695500458 >> 172.21.15.186:0/695500458 conn(0x7fcf2c0b5a30 msgr2=0x7fcf2c0b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:29.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.291+0000 7fcf3436a700 1 -- 172.21.15.186:0/695500458 shutdown_connections 2024-05-08T21:21:29.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.291+0000 7fcf3436a700 1 -- 172.21.15.186:0/695500458 wait complete. 2024-05-08T21:21:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.292+0000 7fcf3436a700 1 Processor -- start 2024-05-08T21:21:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.292+0000 7fcf3436a700 1 -- start start 2024-05-08T21:21:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf3436a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf2c072170 0x7fcf2c0cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf3436a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf2c0ae450 0x7fcf2c153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf3436a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf2c15dc60 0x7fcf2c154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf3436a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c1f4a80 con 0x7fcf2c072170 2024-05-08T21:21:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf3436a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c1f3eb0 con 0x7fcf2c0ae450 2024-05-08T21:21:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf3436a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c1f4150 con 0x7fcf2c15dc60 2024-05-08T21:21:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf31905700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf2c0ae450 0x7fcf2c153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf32907700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf2c15dc60 0x7fcf2c154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf31905700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf2c0ae450 0x7fcf2c153bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:35536/0 (socket says 172.21.15.186:35536) 2024-05-08T21:21:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf31905700 1 -- 172.21.15.186:0/734921808 learned_addr learned my addr 172.21.15.186:0/734921808 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-08T21:21:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf32106700 1 --2- 172.21.15.186:0/734921808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf2c072170 0x7fcf2c0cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf31905700 1 -- 172.21.15.186:0/734921808 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf2c15dc60 msgr2=0x7fcf2c154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf31905700 1 --2- 172.21.15.186:0/734921808 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf2c15dc60 0x7fcf2c154100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf31905700 1 -- 172.21.15.186:0/734921808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf2c072170 msgr2=0x7fcf2c0cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf31905700 1 --2- 172.21.15.186:0/734921808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf2c072170 0x7fcf2c0cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf31905700 1 -- 172.21.15.186:0/734921808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c0094a0 con 0x7fcf2c0ae450 2024-05-08T21:21:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.293+0000 7fcf32907700 1 --2- 172.21.15.186:0/734921808 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf2c15dc60 0x7fcf2c154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:21:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.294+0000 7fcf31905700 1 --2- 172.21.15.186:0/734921808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf2c0ae450 0x7fcf2c153bc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00ebe0 tx=0x7fcf1c00eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.294+0000 7fcf22ffd700 1 -- 172.21.15.186:0/734921808 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c009780 con 0x7fcf2c0ae450 2024-05-08T21:21:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.294+0000 7fcf22ffd700 1 -- 172.21.15.186:0/734921808 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1c00c850 con 0x7fcf2c0ae450 2024-05-08T21:21:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.294+0000 7fcf22ffd700 1 -- 172.21.15.186:0/734921808 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c00cbe0 con 0x7fcf2c0ae450 2024-05-08T21:21:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.294+0000 7fcf3436a700 1 -- 172.21.15.186:0/734921808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2400bd40 con 0x7fcf2c0ae450 2024-05-08T21:21:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.294+0000 7fcf3436a700 1 -- 172.21.15.186:0/734921808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2c1f4710 con 0x7fcf2c0ae450 2024-05-08T21:21:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.294+0000 7fcf22ffd700 1 -- 172.21.15.186:0/734921808 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fcf1c005a20 con 0x7fcf2c0ae450 2024-05-08T21:21:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.294+0000 7fcf22ffd700 1 -- 172.21.15.186:0/734921808 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcf1c0137b0 con 0x7fcf2c0ae450 2024-05-08T21:21:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.295+0000 7fcf3436a700 1 -- 172.21.15.186:0/734921808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2c1f4bf0 con 0x7fcf2c0ae450 2024-05-08T21:21:29.300 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.297+0000 7fcf22ffd700 1 -- 172.21.15.186:0/734921808 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fcf1c021060 con 0x7fcf2c0ae450 2024-05-08T21:21:29.310 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:29.307+0000 7fb71cb9a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-08T21:21:29.341 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:29.339+0000 7f1f5b89f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-08T21:21:29.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.383+0000 7fcf3436a700 1 -- 172.21.15.186:0/734921808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d4e85793-ae1b-4a72-a1a8-2f1e299373ea", "id": 0} v 0) v1 -- 0x7fcf2c074bc0 con 0x7fcf2c0ae450 2024-05-08T21:21:29.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.388+0000 7fcf22ffd700 1 -- 172.21.15.186:0/734921808 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d4e85793-ae1b-4a72-a1a8-2f1e299373ea", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcf1c017070 con 0x7fcf2c0ae450 2024-05-08T21:21:29.390 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-05-08T21:21:29.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.389+0000 7fcf3436a700 1 -- 172.21.15.186:0/734921808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf2c0ae450 msgr2=0x7fcf2c153bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:29.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.389+0000 7fcf3436a700 1 --2- 172.21.15.186:0/734921808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf2c0ae450 0x7fcf2c153bc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00ebe0 tx=0x7fcf1c00eef0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.389+0000 7fcf3436a700 1 -- 172.21.15.186:0/734921808 shutdown_connections 2024-05-08T21:21:29.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.389+0000 7fcf3436a700 1 --2- 172.21.15.186:0/734921808 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf2c15dc60 0x7fcf2c154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.389+0000 7fcf3436a700 1 --2- 172.21.15.186:0/734921808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf2c072170 0x7fcf2c0cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.389+0000 7fcf3436a700 1 --2- 172.21.15.186:0/734921808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf2c0ae450 0x7fcf2c153bc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.389+0000 7fcf3436a700 1 -- 172.21.15.186:0/734921808 >> 172.21.15.186:0/734921808 conn(0x7fcf2c0b5a30 msgr2=0x7fcf2c15cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:29.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.389+0000 7fcf3436a700 1 -- 172.21.15.186:0/734921808 shutdown_connections 2024-05-08T21:21:29.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.390+0000 7fcf3436a700 1 -- 172.21.15.186:0/734921808 wait complete. 2024-05-08T21:21:29.403 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 05adc715-3b7c-4b51-966c-b3c4c4269ffc 1 2024-05-08T21:21:29.505 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:29.503+0000 7fb71cb9a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-08T21:21:29.524 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:29.522+0000 7f1f5b89f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-08T21:21:29.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.564+0000 7fda19faf700 1 Processor -- start 2024-05-08T21:21:29.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.564+0000 7fda19faf700 1 -- start start 2024-05-08T21:21:29.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.565+0000 7fda19faf700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda140b50d0 0x7fda1406a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:29.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.565+0000 7fda19faf700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fda140b5550 0x7fda1406aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:29.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.565+0000 7fda19faf700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda14099ce0 con 0x7fda1415d8c0 2024-05-08T21:21:29.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.565+0000 7fda19faf700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda1406fde0 con 0x7fda140b5550 2024-05-08T21:21:29.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.565+0000 7fda19faf700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda1406f260 con 0x7fda140b50d0 2024-05-08T21:21:29.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.565+0000 7fda12ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda140b50d0 0x7fda1406a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:29.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.565+0000 7fda12ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda140b50d0 0x7fda1406a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:34618/0 (socket says 172.21.15.186:34618) 2024-05-08T21:21:29.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.565+0000 7fda137fe700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fda1415d8c0 0x7fda140ad260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:53008/0 (socket says 172.21.15.186:53008) 2024-05-08T21:21:29.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.565+0000 7fda12ffd700 1 -- 172.21.15.186:0/624152240 learned_addr learned my addr 172.21.15.186:0/624152240 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-08T21:21:29.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.565+0000 7fda13fff700 1 --2- 172.21.15.186:0/624152240 >> v2:172.21.15.3:3300/0 conn(0x7fda140b5550 0x7fda1406aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:29.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.565+0000 7fda127fc700 1 -- 172.21.15.186:0/624152240 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402966590 0 0) 0x7fda14099ce0 con 0x7fda1415d8c0 2024-05-08T21:21:29.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.565+0000 7fda12ffd700 1 -- 172.21.15.186:0/624152240 >> v2:172.21.15.3:3300/0 conn(0x7fda140b5550 msgr2=0x7fda1406aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:29.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.565+0000 7fda12ffd700 1 --2- 172.21.15.186:0/624152240 >> v2:172.21.15.3:3300/0 conn(0x7fda140b5550 0x7fda1406aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.565+0000 7fda12ffd700 1 -- 172.21.15.186:0/624152240 >> v1:172.21.15.186:6789/0 conn(0x7fda1415d8c0 legacy=0x7fda140ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:29.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.565+0000 7fda12ffd700 1 -- 172.21.15.186:0/624152240 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1406f540 con 0x7fda140b50d0 2024-05-08T21:21:29.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.566+0000 7fda12ffd700 1 --2- 172.21.15.186:0/624152240 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda140b50d0 0x7fda1406a960 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fda080098f0 tx=0x7fda08009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8672502b5c45846 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:29.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.566+0000 7fda127fc700 1 -- 172.21.15.186:0/624152240 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda08011830 con 0x7fda140b50d0 2024-05-08T21:21:29.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.566+0000 7fda127fc700 1 -- 172.21.15.186:0/624152240 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda08011990 con 0x7fda140b50d0 2024-05-08T21:21:29.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.566+0000 7fda127fc700 1 -- 172.21.15.186:0/624152240 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda08011d20 con 0x7fda140b50d0 2024-05-08T21:21:29.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.566+0000 7fda19faf700 1 -- 172.21.15.186:0/624152240 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda140b50d0 msgr2=0x7fda1406a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:29.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.566+0000 7fda19faf700 1 --2- 172.21.15.186:0/624152240 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda140b50d0 0x7fda1406a960 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fda080098f0 tx=0x7fda08009c00 comp rx=0 tx=0).stop 2024-05-08T21:21:29.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.566+0000 7fda19faf700 1 -- 172.21.15.186:0/624152240 shutdown_connections 2024-05-08T21:21:29.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.566+0000 7fda19faf700 1 --2- 172.21.15.186:0/624152240 >> v2:172.21.15.3:3300/0 conn(0x7fda140b5550 0x7fda1406aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.566+0000 7fda19faf700 1 --2- 172.21.15.186:0/624152240 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda140b50d0 0x7fda1406a960 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.566+0000 7fda19faf700 1 -- 172.21.15.186:0/624152240 >> 172.21.15.186:0/624152240 conn(0x7fda140b5a30 msgr2=0x7fda140b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:29.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.566+0000 7fda19faf700 1 -- 172.21.15.186:0/624152240 shutdown_connections 2024-05-08T21:21:29.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.566+0000 7fda19faf700 1 -- 172.21.15.186:0/624152240 wait complete. 2024-05-08T21:21:29.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.568+0000 7fda19faf700 1 Processor -- start 2024-05-08T21:21:29.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.568+0000 7fda19faf700 1 -- start start 2024-05-08T21:21:29.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda19faf700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda140b50d0 0x7fda1415f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:29.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda19faf700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda140b5550 0x7fda1415f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:29.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda19faf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda1415d8c0 0x7fda14155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:29.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda19faf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda14160540 con 0x7fda1415d8c0 2024-05-08T21:21:29.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda19faf700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1415fc50 con 0x7fda140b50d0 2024-05-08T21:21:29.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda19faf700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda1415ff30 con 0x7fda140b5550 2024-05-08T21:21:29.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda12ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda140b5550 0x7fda1415f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:29.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda137fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda140b50d0 0x7fda1415f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:29.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda12ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda140b5550 0x7fda1415f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:34628/0 (socket says 172.21.15.186:34628) 2024-05-08T21:21:29.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda12ffd700 1 -- 172.21.15.186:0/3968331790 learned_addr learned my addr 172.21.15.186:0/3968331790 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-08T21:21:29.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda13fff700 1 --2- 172.21.15.186:0/3968331790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda1415d8c0 0x7fda14155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:29.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda137fe700 1 -- 172.21.15.186:0/3968331790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda140b5550 msgr2=0x7fda1415f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:29.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda137fe700 1 --2- 172.21.15.186:0/3968331790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda140b5550 0x7fda1415f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda137fe700 1 -- 172.21.15.186:0/3968331790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda1415d8c0 msgr2=0x7fda14155230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:29.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda137fe700 1 --2- 172.21.15.186:0/3968331790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda1415d8c0 0x7fda14155230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda137fe700 1 -- 172.21.15.186:0/3968331790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda08009550 con 0x7fda140b50d0 2024-05-08T21:21:29.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda12ffd700 1 --2- 172.21.15.186:0/3968331790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda140b5550 0x7fda1415f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:21:29.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda13fff700 1 --2- 172.21.15.186:0/3968331790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda1415d8c0 0x7fda14155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:21:29.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda137fe700 1 --2- 172.21.15.186:0/3968331790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda140b50d0 0x7fda1415f070 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fda0400c410 tx=0x7fda0400c720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:29.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fd9fbfff700 1 -- 172.21.15.186:0/3968331790 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda04004c80 con 0x7fda140b50d0 2024-05-08T21:21:29.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fd9fbfff700 1 -- 172.21.15.186:0/3968331790 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda04004de0 con 0x7fda140b50d0 2024-05-08T21:21:29.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fd9fbfff700 1 -- 172.21.15.186:0/3968331790 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda04007310 con 0x7fda140b50d0 2024-05-08T21:21:29.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda19faf700 1 -- 172.21.15.186:0/3968331790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda14154750 con 0x7fda140b50d0 2024-05-08T21:21:29.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.569+0000 7fda19faf700 1 -- 172.21.15.186:0/3968331790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda14154d10 con 0x7fda140b50d0 2024-05-08T21:21:29.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.570+0000 7fd9fbfff700 1 -- 172.21.15.186:0/3968331790 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fda0401d3f0 con 0x7fda140b50d0 2024-05-08T21:21:29.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.570+0000 7fd9fbfff700 1 -- 172.21.15.186:0/3968331790 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fda04013770 con 0x7fda140b50d0 2024-05-08T21:21:29.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.570+0000 7fda19faf700 1 -- 172.21.15.186:0/3968331790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda141545b0 con 0x7fda140b50d0 2024-05-08T21:21:29.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.572+0000 7fd9fbfff700 1 -- 172.21.15.186:0/3968331790 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fda141545b0 con 0x7fda140b50d0 2024-05-08T21:21:29.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.656+0000 7fda19faf700 1 -- 172.21.15.186:0/3968331790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "05adc715-3b7c-4b51-966c-b3c4c4269ffc", "id": 1} v 0) v1 -- 0x7fda14074bc0 con 0x7fda140b50d0 2024-05-08T21:21:29.665 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.664+0000 7fd9fbfff700 1 -- 172.21.15.186:0/3968331790 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "05adc715-3b7c-4b51-966c-b3c4c4269ffc", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fda040139d0 con 0x7fda140b50d0 2024-05-08T21:21:29.666 INFO:teuthology.orchestra.run.smithi186.stdout:1 2024-05-08T21:21:29.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.665+0000 7fda19faf700 1 -- 172.21.15.186:0/3968331790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda140b50d0 msgr2=0x7fda1415f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:29.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.665+0000 7fda19faf700 1 --2- 172.21.15.186:0/3968331790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda140b50d0 0x7fda1415f070 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fda0400c410 tx=0x7fda0400c720 comp rx=0 tx=0).stop 2024-05-08T21:21:29.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.665+0000 7fda19faf700 1 -- 172.21.15.186:0/3968331790 shutdown_connections 2024-05-08T21:21:29.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.665+0000 7fda19faf700 1 --2- 172.21.15.186:0/3968331790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda1415d8c0 0x7fda14155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.665+0000 7fda19faf700 1 --2- 172.21.15.186:0/3968331790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda140b50d0 0x7fda1415f070 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.665+0000 7fda19faf700 1 --2- 172.21.15.186:0/3968331790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda140b5550 0x7fda1415f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.665+0000 7fda19faf700 1 -- 172.21.15.186:0/3968331790 >> 172.21.15.186:0/3968331790 conn(0x7fda140b5a30 msgr2=0x7fda140cf970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:29.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.665+0000 7fda19faf700 1 -- 172.21.15.186:0/3968331790 shutdown_connections 2024-05-08T21:21:29.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.665+0000 7fda19faf700 1 -- 172.21.15.186:0/3968331790 wait complete. 2024-05-08T21:21:29.679 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new d9717d7b-dfd4-4826-b6db-6bb3e61511fa 2 2024-05-08T21:21:29.692 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:29.689+0000 7fb71cb9a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-08T21:21:29.702 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:29.701+0000 7f1f5b89f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-08T21:21:29.801 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.799+0000 7fea1bc1f700 1 Processor -- start 2024-05-08T21:21:29.801 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.800+0000 7fea1bc1f700 1 -- start start 2024-05-08T21:21:29.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.801+0000 7fea1bc1f700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fea140cc590 0x7fea1406f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:29.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.801+0000 7fea1bc1f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea140cb160 con 0x7fea1415db00 2024-05-08T21:21:29.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.801+0000 7fea1bc1f700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1406ad20 con 0x7fea1406f7c0 2024-05-08T21:21:29.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.801+0000 7fea1bc1f700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea1406fc50 con 0x7fea140cc590 2024-05-08T21:21:29.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.801+0000 7fea191ba700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fea140cc590 0x7fea1406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:29.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.801+0000 7fea1a1bc700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fea1406f7c0 0x7fea1406e320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:53012/0 (socket says 172.21.15.186:53012) 2024-05-08T21:21:29.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.801+0000 7fea1a1bc700 1 -- 172.21.15.186:0/1868048210 learned_addr learned my addr 172.21.15.186:0/1868048210 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-05-08T21:21:29.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.801+0000 7fea191ba700 1 -- 172.21.15.186:0/1868048210 >> v1:172.21.15.186:6789/0 conn(0x7fea1406f7c0 legacy=0x7fea1406e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:29.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.801+0000 7fea191ba700 1 -- 172.21.15.186:0/1868048210 >> v1:172.21.15.3:6789/0 conn(0x7fea1415db00 legacy=0x7fea140caa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:29.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.801+0000 7fea189b9700 1 -- 172.21.15.186:0/1868048210 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559517893 0 0) 0x7fea1406ad20 con 0x7fea1406f7c0 2024-05-08T21:21:29.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.801+0000 7fea191ba700 1 -- 172.21.15.186:0/1868048210 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1406b000 con 0x7fea140cc590 2024-05-08T21:21:29.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.801+0000 7fea191ba700 1 --2- 172.21.15.186:0/1868048210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea140cc590 0x7fea1406f280 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fea04009840 tx=0x7fea04009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4720b49a4be05212 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:29.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.801+0000 7fea189b9700 1 -- 172.21.15.186:0/1868048210 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea04011830 con 0x7fea140cc590 2024-05-08T21:21:29.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.801+0000 7fea189b9700 1 -- 172.21.15.186:0/1868048210 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea04011990 con 0x7fea140cc590 2024-05-08T21:21:29.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.801+0000 7fea189b9700 1 -- 172.21.15.186:0/1868048210 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea04011d20 con 0x7fea140cc590 2024-05-08T21:21:29.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.802+0000 7fea1bc1f700 1 -- 172.21.15.186:0/1868048210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea140cc590 msgr2=0x7fea1406f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:29.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.802+0000 7fea1bc1f700 1 --2- 172.21.15.186:0/1868048210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea140cc590 0x7fea1406f280 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fea04009840 tx=0x7fea04009b50 comp rx=0 tx=0).stop 2024-05-08T21:21:29.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.802+0000 7fea1bc1f700 1 -- 172.21.15.186:0/1868048210 shutdown_connections 2024-05-08T21:21:29.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.802+0000 7fea1bc1f700 1 --2- 172.21.15.186:0/1868048210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea140cc590 0x7fea1406f280 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.802+0000 7fea1bc1f700 1 -- 172.21.15.186:0/1868048210 >> 172.21.15.186:0/1868048210 conn(0x7fea14099a50 msgr2=0x7fea140d0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:29.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.802+0000 7fea1bc1f700 1 -- 172.21.15.186:0/1868048210 shutdown_connections 2024-05-08T21:21:29.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.802+0000 7fea1bc1f700 1 -- 172.21.15.186:0/1868048210 wait complete. 2024-05-08T21:21:29.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.803+0000 7fea1bc1f700 1 Processor -- start 2024-05-08T21:21:29.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.803+0000 7fea1bc1f700 1 -- start start 2024-05-08T21:21:29.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.804+0000 7fea1bc1f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea1406f7c0 0x7fea140b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:29.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.804+0000 7fea1bc1f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea140cc590 0x7fea14159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:29.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.804+0000 7fea1bc1f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea1415db00 0x7fea140b0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:29.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.804+0000 7fea1bc1f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea140b3ab0 con 0x7fea1415db00 2024-05-08T21:21:29.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.804+0000 7fea1bc1f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea141f40a0 con 0x7fea140cc590 2024-05-08T21:21:29.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.804+0000 7fea1bc1f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea140b3110 con 0x7fea1406f7c0 2024-05-08T21:21:29.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.804+0000 7fea191ba700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea140cc590 0x7fea14159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:29.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.804+0000 7fea199bb700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea1406f7c0 0x7fea140b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:29.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.804+0000 7fea191ba700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea140cc590 0x7fea14159a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:35564/0 (socket says 172.21.15.186:35564) 2024-05-08T21:21:29.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.804+0000 7fea191ba700 1 -- 172.21.15.186:0/1139479656 learned_addr learned my addr 172.21.15.186:0/1139479656 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-08T21:21:29.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.804+0000 7fea1a1bc700 1 --2- 172.21.15.186:0/1139479656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea1415db00 0x7fea140b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:29.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.805+0000 7fea191ba700 1 -- 172.21.15.186:0/1139479656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea1406f7c0 msgr2=0x7fea140b3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:29.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.805+0000 7fea191ba700 1 --2- 172.21.15.186:0/1139479656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea1406f7c0 0x7fea140b3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.805+0000 7fea191ba700 1 -- 172.21.15.186:0/1139479656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea1415db00 msgr2=0x7fea140b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:29.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.805+0000 7fea191ba700 1 --2- 172.21.15.186:0/1139479656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea1415db00 0x7fea140b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.805+0000 7fea191ba700 1 -- 172.21.15.186:0/1139479656 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea040094a0 con 0x7fea140cc590 2024-05-08T21:21:29.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.805+0000 7fea199bb700 1 --2- 172.21.15.186:0/1139479656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea1406f7c0 0x7fea140b3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:21:29.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.805+0000 7fea191ba700 1 --2- 172.21.15.186:0/1139479656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea140cc590 0x7fea14159a30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fea040090b0 tx=0x7fea04004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:29.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.805+0000 7fea0a7fc700 1 -- 172.21.15.186:0/1139479656 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0401a7e0 con 0x7fea140cc590 2024-05-08T21:21:29.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.805+0000 7fea0a7fc700 1 -- 172.21.15.186:0/1139479656 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0401a940 con 0x7fea140cc590 2024-05-08T21:21:29.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.805+0000 7fea1bc1f700 1 -- 172.21.15.186:0/1139479656 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea140b33f0 con 0x7fea140cc590 2024-05-08T21:21:29.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.805+0000 7fea0a7fc700 1 -- 172.21.15.186:0/1139479656 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0401acb0 con 0x7fea140cc590 2024-05-08T21:21:29.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.805+0000 7fea1bc1f700 1 -- 172.21.15.186:0/1139479656 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea140b52b0 con 0x7fea140cc590 2024-05-08T21:21:29.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.805+0000 7fea0a7fc700 1 -- 172.21.15.186:0/1139479656 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fea04022070 con 0x7fea140cc590 2024-05-08T21:21:29.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.805+0000 7fea0a7fc700 1 -- 172.21.15.186:0/1139479656 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fea04024840 con 0x7fea140cc590 2024-05-08T21:21:29.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.806+0000 7fea1bc1f700 1 -- 172.21.15.186:0/1139479656 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea140b3bf0 con 0x7fea140cc590 2024-05-08T21:21:29.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.808+0000 7fea0a7fc700 1 -- 172.21.15.186:0/1139479656 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fea140b3bf0 con 0x7fea140cc590 2024-05-08T21:21:29.817 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:29.814+0000 7fb71cb9a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-08T21:21:29.819 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:29.817+0000 7f1f5b89f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-08T21:21:29.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.893+0000 7fea1bc1f700 1 -- 172.21.15.186:0/1139479656 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d9717d7b-dfd4-4826-b6db-6bb3e61511fa", "id": 2} v 0) v1 -- 0x7fea140c73a0 con 0x7fea140cc590 2024-05-08T21:21:29.902 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.901+0000 7fea0a7fc700 1 -- 172.21.15.186:0/1139479656 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d9717d7b-dfd4-4826-b6db-6bb3e61511fa", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fea04033060 con 0x7fea140cc590 2024-05-08T21:21:29.903 INFO:teuthology.orchestra.run.smithi186.stdout:2 2024-05-08T21:21:29.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.902+0000 7fea1bc1f700 1 -- 172.21.15.186:0/1139479656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea140cc590 msgr2=0x7fea14159a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:29.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.902+0000 7fea1bc1f700 1 --2- 172.21.15.186:0/1139479656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea140cc590 0x7fea14159a30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fea040090b0 tx=0x7fea04004750 comp rx=0 tx=0).stop 2024-05-08T21:21:29.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.902+0000 7fea1bc1f700 1 -- 172.21.15.186:0/1139479656 shutdown_connections 2024-05-08T21:21:29.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.902+0000 7fea1bc1f700 1 --2- 172.21.15.186:0/1139479656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea1406f7c0 0x7fea140b3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.902+0000 7fea1bc1f700 1 --2- 172.21.15.186:0/1139479656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea1415db00 0x7fea140b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.902+0000 7fea1bc1f700 1 --2- 172.21.15.186:0/1139479656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea140cc590 0x7fea14159a30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:29.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.902+0000 7fea1bc1f700 1 -- 172.21.15.186:0/1139479656 >> 172.21.15.186:0/1139479656 conn(0x7fea14099a50 msgr2=0x7fea140caa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:29.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.902+0000 7fea1bc1f700 1 -- 172.21.15.186:0/1139479656 shutdown_connections 2024-05-08T21:21:29.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:29.902+0000 7fea1bc1f700 1 -- 172.21.15.186:0/1139479656 wait complete. 2024-05-08T21:21:29.915 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 1a904ea7-5acf-4af7-b73d-c4f0070ad767 3 2024-05-08T21:21:29.918 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:29.917+0000 7f1f5b89f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-08T21:21:29.923 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:29.921+0000 7fb71cb9a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-08T21:21:30.000 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:29.998+0000 7f1f5b89f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-08T21:21:30.009 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:30.007+0000 7fb71cb9a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-08T21:21:30.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.038+0000 7fcb6775d700 1 Processor -- start 2024-05-08T21:21:30.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.038+0000 7fcb6775d700 1 -- start start 2024-05-08T21:21:30.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.039+0000 7fcb6775d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcb6015dc60 0x7fcb600ad9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.039+0000 7fcb6775d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcb60072170 0x7fcb600adf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.039+0000 7fcb6775d700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb600aed90 con 0x7fcb600ae450 2024-05-08T21:21:30.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.039+0000 7fcb6775d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb6006dcd0 con 0x7fcb60072170 2024-05-08T21:21:30.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.039+0000 7fcb6775d700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb6006e290 con 0x7fcb6015dc60 2024-05-08T21:21:30.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.039+0000 7fcb654f9700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcb6015dc60 0x7fcb600ad9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.039+0000 7fcb65cfa700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fcb600ae450 0x7fcb600b3be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:53018/0 (socket says 172.21.15.186:53018) 2024-05-08T21:21:30.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.039+0000 7fcb65cfa700 1 -- 172.21.15.186:0/1965589139 learned_addr learned my addr 172.21.15.186:0/1965589139 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-05-08T21:21:30.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.039+0000 7fcb64cf8700 1 --2- 172.21.15.186:0/1965589139 >> v2:172.21.15.3:3300/0 conn(0x7fcb60072170 0x7fcb600adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.040+0000 7fcb654f9700 1 -- 172.21.15.186:0/1965589139 >> v2:172.21.15.3:3300/0 conn(0x7fcb60072170 msgr2=0x7fcb600adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:30.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.040+0000 7fcb654f9700 1 --2- 172.21.15.186:0/1965589139 >> v2:172.21.15.3:3300/0 conn(0x7fcb60072170 0x7fcb600adf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.040+0000 7fcb654f9700 1 -- 172.21.15.186:0/1965589139 >> v1:172.21.15.186:6789/0 conn(0x7fcb600ae450 legacy=0x7fcb600b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.040+0000 7fcb654f9700 1 -- 172.21.15.186:0/1965589139 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb600ae8c0 con 0x7fcb6015dc60 2024-05-08T21:21:30.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.040+0000 7fcb64cf8700 1 --2- 172.21.15.186:0/1965589139 >> v2:172.21.15.3:3300/0 conn(0x7fcb60072170 0x7fcb600adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:21:30.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.040+0000 7fcb654f9700 1 --2- 172.21.15.186:0/1965589139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb6015dc60 0x7fcb600ad9d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcb6006db20 tx=0x7fcb5c00bdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9336d8721e9f4980 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:30.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.040+0000 7fcb57fff700 1 -- 172.21.15.186:0/1965589139 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c00dc70 con 0x7fcb6015dc60 2024-05-08T21:21:30.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.040+0000 7fcb57fff700 1 -- 172.21.15.186:0/1965589139 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c00ddd0 con 0x7fcb6015dc60 2024-05-08T21:21:30.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.040+0000 7fcb57fff700 1 -- 172.21.15.186:0/1965589139 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c010690 con 0x7fcb6015dc60 2024-05-08T21:21:30.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.040+0000 7fcb6775d700 1 -- 172.21.15.186:0/1965589139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb6015dc60 msgr2=0x7fcb600ad9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.040+0000 7fcb6775d700 1 --2- 172.21.15.186:0/1965589139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb6015dc60 0x7fcb600ad9d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcb6006db20 tx=0x7fcb5c00bdc0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.041+0000 7fcb6775d700 1 -- 172.21.15.186:0/1965589139 shutdown_connections 2024-05-08T21:21:30.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.041+0000 7fcb6775d700 1 --2- 172.21.15.186:0/1965589139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb6015dc60 0x7fcb600ad9d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.041+0000 7fcb6775d700 1 --2- 172.21.15.186:0/1965589139 >> v2:172.21.15.3:3300/0 conn(0x7fcb60072170 0x7fcb600adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.041+0000 7fcb6775d700 1 -- 172.21.15.186:0/1965589139 >> 172.21.15.186:0/1965589139 conn(0x7fcb600b5a30 msgr2=0x7fcb600b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:30.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.041+0000 7fcb6775d700 1 -- 172.21.15.186:0/1965589139 shutdown_connections 2024-05-08T21:21:30.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.041+0000 7fcb6775d700 1 -- 172.21.15.186:0/1965589139 wait complete. 2024-05-08T21:21:30.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.042+0000 7fcb6775d700 1 Processor -- start 2024-05-08T21:21:30.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.042+0000 7fcb6775d700 1 -- start start 2024-05-08T21:21:30.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb6775d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb60072170 0x7fcb601541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb6775d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb600ae450 0x7fcb6014ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb6775d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb6015dc60 0x7fcb601f7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb6775d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb601f6b00 con 0x7fcb60072170 2024-05-08T21:21:30.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb6775d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb60156350 con 0x7fcb600ae450 2024-05-08T21:21:30.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb6775d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb60156630 con 0x7fcb6015dc60 2024-05-08T21:21:30.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb64cf8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb600ae450 0x7fcb6014ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb64cf8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb600ae450 0x7fcb6014ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:35588/0 (socket says 172.21.15.186:35588) 2024-05-08T21:21:30.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb64cf8700 1 -- 172.21.15.186:0/2503842048 learned_addr learned my addr 172.21.15.186:0/2503842048 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-08T21:21:30.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb654f9700 1 --2- 172.21.15.186:0/2503842048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb60072170 0x7fcb601541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb65cfa700 1 --2- 172.21.15.186:0/2503842048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb6015dc60 0x7fcb601f7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb64cf8700 1 -- 172.21.15.186:0/2503842048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb6015dc60 msgr2=0x7fcb601f7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb64cf8700 1 --2- 172.21.15.186:0/2503842048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb6015dc60 0x7fcb601f7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb64cf8700 1 -- 172.21.15.186:0/2503842048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb60072170 msgr2=0x7fcb601541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb64cf8700 1 --2- 172.21.15.186:0/2503842048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb60072170 0x7fcb601541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb64cf8700 1 -- 172.21.15.186:0/2503842048 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c00b740 con 0x7fcb600ae450 2024-05-08T21:21:30.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb64cf8700 1 --2- 172.21.15.186:0/2503842048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb600ae450 0x7fcb6014ee50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000e810 tx=0x7fcb5000eb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:30.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb55ffb700 1 -- 172.21.15.186:0/2503842048 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5000c070 con 0x7fcb600ae450 2024-05-08T21:21:30.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb55ffb700 1 -- 172.21.15.186:0/2503842048 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb50004230 con 0x7fcb600ae450 2024-05-08T21:21:30.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb55ffb700 1 -- 172.21.15.186:0/2503842048 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb500036a0 con 0x7fcb600ae450 2024-05-08T21:21:30.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.043+0000 7fcb6775d700 1 -- 172.21.15.186:0/2503842048 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb60156910 con 0x7fcb600ae450 2024-05-08T21:21:30.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.044+0000 7fcb55ffb700 1 -- 172.21.15.186:0/2503842048 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fcb50003b00 con 0x7fcb600ae450 2024-05-08T21:21:30.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.044+0000 7fcb6775d700 1 -- 172.21.15.186:0/2503842048 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb60154710 con 0x7fcb600ae450 2024-05-08T21:21:30.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.044+0000 7fcb55ffb700 1 -- 172.21.15.186:0/2503842048 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fcb50013b90 con 0x7fcb600ae450 2024-05-08T21:21:30.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.045+0000 7fcb6775d700 1 -- 172.21.15.186:0/2503842048 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb60156b00 con 0x7fcb600ae450 2024-05-08T21:21:30.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.047+0000 7fcb55ffb700 1 -- 172.21.15.186:0/2503842048 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fcb5000f040 con 0x7fcb600ae450 2024-05-08T21:21:30.134 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.133+0000 7fcb6775d700 1 -- 172.21.15.186:0/2503842048 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1a904ea7-5acf-4af7-b73d-c4f0070ad767", "id": 3} v 0) v1 -- 0x7fcb60074bc0 con 0x7fcb600ae450 2024-05-08T21:21:30.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.140+0000 7fcb55ffb700 1 -- 172.21.15.186:0/2503842048 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1a904ea7-5acf-4af7-b73d-c4f0070ad767", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcb50013d50 con 0x7fcb600ae450 2024-05-08T21:21:30.142 INFO:teuthology.orchestra.run.smithi186.stdout:3 2024-05-08T21:21:30.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.141+0000 7fcb6775d700 1 -- 172.21.15.186:0/2503842048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb600ae450 msgr2=0x7fcb6014ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.141+0000 7fcb6775d700 1 --2- 172.21.15.186:0/2503842048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb600ae450 0x7fcb6014ee50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000e810 tx=0x7fcb5000eb20 comp rx=0 tx=0).stop 2024-05-08T21:21:30.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.141+0000 7fcb6775d700 1 -- 172.21.15.186:0/2503842048 shutdown_connections 2024-05-08T21:21:30.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.141+0000 7fcb6775d700 1 --2- 172.21.15.186:0/2503842048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb6015dc60 0x7fcb601f7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.141+0000 7fcb6775d700 1 --2- 172.21.15.186:0/2503842048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb60072170 0x7fcb601541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.141+0000 7fcb6775d700 1 --2- 172.21.15.186:0/2503842048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb600ae450 0x7fcb6014ee50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.141+0000 7fcb6775d700 1 -- 172.21.15.186:0/2503842048 >> 172.21.15.186:0/2503842048 conn(0x7fcb600b5a30 msgr2=0x7fcb600b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:30.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.141+0000 7fcb6775d700 1 -- 172.21.15.186:0/2503842048 shutdown_connections 2024-05-08T21:21:30.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.142+0000 7fcb6775d700 1 -- 172.21.15.186:0/2503842048 wait complete. 2024-05-08T21:21:30.155 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 3f01306b-2f27-4185-b978-bff25e857c7c 4 2024-05-08T21:21:30.235 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:30.233+0000 7f1f5b89f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-08T21:21:30.270 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:30.267+0000 7fb71cb9a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-08T21:21:30.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.296+0000 7f734a44c700 1 Processor -- start 2024-05-08T21:21:30.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.296+0000 7f734a44c700 1 -- start start 2024-05-08T21:21:30.300 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.297+0000 7f734a44c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f73440b50d0 0x7f734406a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.300 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.297+0000 7f734a44c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f73440b5550 0x7f734406aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.297+0000 7f734a44c700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7344099ce0 con 0x7f734415d8c0 2024-05-08T21:21:30.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.297+0000 7f734a44c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f734406fde0 con 0x7f73440b50d0 2024-05-08T21:21:30.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.297+0000 7f734a44c700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f734406f260 con 0x7f73440b5550 2024-05-08T21:21:30.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.297+0000 7f73489e9700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f73440b5550 0x7f734406aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.297+0000 7f7343fff700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f734415d8c0 0x7f73440ad260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:53022/0 (socket says 172.21.15.186:53022) 2024-05-08T21:21:30.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.297+0000 7f7343fff700 1 -- 172.21.15.186:0/3710222784 learned_addr learned my addr 172.21.15.186:0/3710222784 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-05-08T21:21:30.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.297+0000 7f733bfff700 1 --2- 172.21.15.186:0/3710222784 >> v2:172.21.15.3:3300/0 conn(0x7f73440b50d0 0x7f734406a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.298+0000 7f73489e9700 1 -- 172.21.15.186:0/3710222784 >> v2:172.21.15.3:3300/0 conn(0x7f73440b50d0 msgr2=0x7f734406a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:30.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.298+0000 7f73489e9700 1 --2- 172.21.15.186:0/3710222784 >> v2:172.21.15.3:3300/0 conn(0x7f73440b50d0 0x7f734406a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.298+0000 7f73489e9700 1 -- 172.21.15.186:0/3710222784 >> v1:172.21.15.186:6789/0 conn(0x7f734415d8c0 legacy=0x7f73440ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.298+0000 7f73489e9700 1 -- 172.21.15.186:0/3710222784 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734406f540 con 0x7f73440b5550 2024-05-08T21:21:30.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.298+0000 7f73437fe700 1 -- 172.21.15.186:0/3710222784 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215668460 0 0) 0x7f7344099ce0 con 0x7f734415d8c0 2024-05-08T21:21:30.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.298+0000 7f733bfff700 1 --2- 172.21.15.186:0/3710222784 >> v2:172.21.15.3:3300/0 conn(0x7f73440b50d0 0x7f734406a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:21:30.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.298+0000 7f73489e9700 1 --2- 172.21.15.186:0/3710222784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73440b5550 0x7f734406aea0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f73440b4650 tx=0x7f733c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a5e25294c3d6462 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:30.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.298+0000 7f73437fe700 1 -- 172.21.15.186:0/3710222784 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c00e070 con 0x7f73440b5550 2024-05-08T21:21:30.302 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.298+0000 7f73437fe700 1 -- 172.21.15.186:0/3710222784 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733c00bbd0 con 0x7f73440b5550 2024-05-08T21:21:30.302 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.298+0000 7f73437fe700 1 -- 172.21.15.186:0/3710222784 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c012430 con 0x7f73440b5550 2024-05-08T21:21:30.302 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.298+0000 7f734a44c700 1 -- 172.21.15.186:0/3710222784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73440b5550 msgr2=0x7f734406aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.302 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.298+0000 7f734a44c700 1 --2- 172.21.15.186:0/3710222784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73440b5550 0x7f734406aea0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f73440b4650 tx=0x7f733c00b330 comp rx=0 tx=0).stop 2024-05-08T21:21:30.302 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.299+0000 7f734a44c700 1 -- 172.21.15.186:0/3710222784 shutdown_connections 2024-05-08T21:21:30.302 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.299+0000 7f734a44c700 1 --2- 172.21.15.186:0/3710222784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73440b5550 0x7f734406aea0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.302 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.299+0000 7f734a44c700 1 --2- 172.21.15.186:0/3710222784 >> v2:172.21.15.3:3300/0 conn(0x7f73440b50d0 0x7f734406a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.302 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.299+0000 7f734a44c700 1 -- 172.21.15.186:0/3710222784 >> 172.21.15.186:0/3710222784 conn(0x7f73440b5a30 msgr2=0x7f73440b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:30.302 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.299+0000 7f734a44c700 1 -- 172.21.15.186:0/3710222784 shutdown_connections 2024-05-08T21:21:30.302 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.299+0000 7f734a44c700 1 -- 172.21.15.186:0/3710222784 wait complete. 2024-05-08T21:21:30.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.300+0000 7f734a44c700 1 Processor -- start 2024-05-08T21:21:30.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.300+0000 7f734a44c700 1 -- start start 2024-05-08T21:21:30.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f734a44c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73440b50d0 0x7f73441f31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f734a44c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73440b5550 0x7f7344155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f734a44c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f734415d8c0 0x7f734414ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f734a44c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734415e3e0 con 0x7f73440b5550 2024-05-08T21:21:30.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f734a44c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734414faf0 con 0x7f73440b50d0 2024-05-08T21:21:30.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f734a44c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734415e630 con 0x7f734415d8c0 2024-05-08T21:21:30.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f73489e9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f734415d8c0 0x7f734414ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f73489e9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f734415d8c0 0x7f734414ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:34662/0 (socket says 172.21.15.186:34662) 2024-05-08T21:21:30.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f73489e9700 1 -- 172.21.15.186:0/2260031786 learned_addr learned my addr 172.21.15.186:0/2260031786 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-08T21:21:30.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f733bfff700 1 --2- 172.21.15.186:0/2260031786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73440b5550 0x7f7344155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f7343fff700 1 --2- 172.21.15.186:0/2260031786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73440b50d0 0x7f73441f31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f73489e9700 1 -- 172.21.15.186:0/2260031786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73440b50d0 msgr2=0x7f73441f31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f73489e9700 1 --2- 172.21.15.186:0/2260031786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73440b50d0 0x7f73441f31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f73489e9700 1 -- 172.21.15.186:0/2260031786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73440b5550 msgr2=0x7f7344155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f73489e9700 1 --2- 172.21.15.186:0/2260031786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73440b5550 0x7f7344155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f73489e9700 1 -- 172.21.15.186:0/2260031786 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c009a70 con 0x7f734415d8c0 2024-05-08T21:21:30.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f73489e9700 1 --2- 172.21.15.186:0/2260031786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f734415d8c0 0x7f734414ee70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f734406ff20 tx=0x7f733c004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:30.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f73417fa700 1 -- 172.21.15.186:0/2260031786 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c012430 con 0x7f734415d8c0 2024-05-08T21:21:30.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f73417fa700 1 -- 172.21.15.186:0/2260031786 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733c020070 con 0x7f734415d8c0 2024-05-08T21:21:30.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f734a44c700 1 -- 172.21.15.186:0/2260031786 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f734415e910 con 0x7f734415d8c0 2024-05-08T21:21:30.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f73417fa700 1 -- 172.21.15.186:0/2260031786 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c004280 con 0x7f734415d8c0 2024-05-08T21:21:30.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.301+0000 7f734a44c700 1 -- 172.21.15.186:0/2260031786 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73441561d0 con 0x7f734415d8c0 2024-05-08T21:21:30.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.302+0000 7f73417fa700 1 -- 172.21.15.186:0/2260031786 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f733c0046e0 con 0x7f734415d8c0 2024-05-08T21:21:30.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.302+0000 7f73417fa700 1 -- 172.21.15.186:0/2260031786 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f733c022070 con 0x7f734415d8c0 2024-05-08T21:21:30.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.302+0000 7f734a44c700 1 -- 172.21.15.186:0/2260031786 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f734415ee30 con 0x7f734415d8c0 2024-05-08T21:21:30.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.305+0000 7f73417fa700 1 -- 172.21.15.186:0/2260031786 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f733c02c060 con 0x7f734415d8c0 2024-05-08T21:21:30.312 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:30.311+0000 7f1f5b89f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-08T21:21:30.349 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:30.347+0000 7fb71cb9a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-08T21:21:30.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.389+0000 7f734a44c700 1 -- 172.21.15.186:0/2260031786 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3f01306b-2f27-4185-b978-bff25e857c7c", "id": 4} v 0) v1 -- 0x7f7344074bc0 con 0x7f734415d8c0 2024-05-08T21:21:30.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.397+0000 7f73417fa700 1 -- 172.21.15.186:0/2260031786 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3f01306b-2f27-4185-b978-bff25e857c7c", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f733c012e40 con 0x7f734415d8c0 2024-05-08T21:21:30.399 INFO:teuthology.orchestra.run.smithi186.stdout:4 2024-05-08T21:21:30.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.398+0000 7f734a44c700 1 -- 172.21.15.186:0/2260031786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f734415d8c0 msgr2=0x7f734414ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.398+0000 7f734a44c700 1 --2- 172.21.15.186:0/2260031786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f734415d8c0 0x7f734414ee70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f734406ff20 tx=0x7f733c004ad0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.398+0000 7f734a44c700 1 -- 172.21.15.186:0/2260031786 shutdown_connections 2024-05-08T21:21:30.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.398+0000 7f734a44c700 1 --2- 172.21.15.186:0/2260031786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f734415d8c0 0x7f734414ee70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.398+0000 7f734a44c700 1 --2- 172.21.15.186:0/2260031786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73440b50d0 0x7f73441f31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.398+0000 7f734a44c700 1 --2- 172.21.15.186:0/2260031786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73440b5550 0x7f7344155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.398+0000 7f734a44c700 1 -- 172.21.15.186:0/2260031786 >> 172.21.15.186:0/2260031786 conn(0x7f73440b5a30 msgr2=0x7f734415a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:30.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.398+0000 7f734a44c700 1 -- 172.21.15.186:0/2260031786 shutdown_connections 2024-05-08T21:21:30.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.398+0000 7f734a44c700 1 -- 172.21.15.186:0/2260031786 wait complete. 2024-05-08T21:21:30.411 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 36f775b1-4bc3-4889-958b-5d13b5855b4c 5 2024-05-08T21:21:30.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.531+0000 7f9178b5b700 1 Processor -- start 2024-05-08T21:21:30.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.531+0000 7f9178b5b700 1 -- start start 2024-05-08T21:21:30.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.532+0000 7f9178b5b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91740cbf50 0x7f9174161d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.532+0000 7f9178b5b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f917415e760 0x7f917406a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.532+0000 7f9178b5b700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91740cf970 con 0x7f9174162240 2024-05-08T21:21:30.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.532+0000 7f9178b5b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91740cfc50 con 0x7f917415e760 2024-05-08T21:21:30.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.532+0000 7f9178b5b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91740cff30 con 0x7f91740cbf50 2024-05-08T21:21:30.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.532+0000 7f917259c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91740cbf50 0x7f9174161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.532+0000 7f9171d9b700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f9174162240 0x7f917415e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:53024/0 (socket says 172.21.15.186:53024) 2024-05-08T21:21:30.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.532+0000 7f9171d9b700 1 -- 172.21.15.186:0/530150571 learned_addr learned my addr 172.21.15.186:0/530150571 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-05-08T21:21:30.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.532+0000 7f917259c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91740cbf50 0x7f9174161d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:34670/0 (socket says 172.21.15.186:34670) 2024-05-08T21:21:30.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.533+0000 7f9172d9d700 1 --2- 172.21.15.186:0/530150571 >> v2:172.21.15.3:3300/0 conn(0x7f917415e760 0x7f917406a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.533+0000 7f917159a700 1 -- 172.21.15.186:0/530150571 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661330035 0 0) 0x7f91740cf970 con 0x7f9174162240 2024-05-08T21:21:30.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.533+0000 7f917259c700 1 -- 172.21.15.186:0/530150571 >> v2:172.21.15.3:3300/0 conn(0x7f917415e760 msgr2=0x7f917406a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:30.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.533+0000 7f917259c700 1 --2- 172.21.15.186:0/530150571 >> v2:172.21.15.3:3300/0 conn(0x7f917415e760 0x7f917406a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.533+0000 7f917259c700 1 -- 172.21.15.186:0/530150571 >> v1:172.21.15.186:6789/0 conn(0x7f9174162240 legacy=0x7f917415e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.533+0000 7f917259c700 1 -- 172.21.15.186:0/530150571 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91741617e0 con 0x7f91740cbf50 2024-05-08T21:21:30.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.533+0000 7f9172d9d700 1 --2- 172.21.15.186:0/530150571 >> v2:172.21.15.3:3300/0 conn(0x7f917415e760 0x7f917406a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:21:30.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.533+0000 7f917259c700 1 --2- 172.21.15.186:0/530150571 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91740cbf50 0x7f9174161d00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9168009840 tx=0x7f9168009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a01ca8eb72bbc65a server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:30.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.533+0000 7f917159a700 1 -- 172.21.15.186:0/530150571 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9168011830 con 0x7f91740cbf50 2024-05-08T21:21:30.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.533+0000 7f917159a700 1 -- 172.21.15.186:0/530150571 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9168011990 con 0x7f91740cbf50 2024-05-08T21:21:30.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.533+0000 7f917159a700 1 -- 172.21.15.186:0/530150571 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9168011d20 con 0x7f91740cbf50 2024-05-08T21:21:30.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.534+0000 7f9178b5b700 1 -- 172.21.15.186:0/530150571 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91740cbf50 msgr2=0x7f9174161d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.534+0000 7f9178b5b700 1 --2- 172.21.15.186:0/530150571 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91740cbf50 0x7f9174161d00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9168009840 tx=0x7f9168009b50 comp rx=0 tx=0).stop 2024-05-08T21:21:30.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.534+0000 7f9178b5b700 1 -- 172.21.15.186:0/530150571 shutdown_connections 2024-05-08T21:21:30.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.534+0000 7f9178b5b700 1 --2- 172.21.15.186:0/530150571 >> v2:172.21.15.3:3300/0 conn(0x7f917415e760 0x7f917406a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.534+0000 7f9178b5b700 1 --2- 172.21.15.186:0/530150571 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91740cbf50 0x7f9174161d00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.534+0000 7f9178b5b700 1 -- 172.21.15.186:0/530150571 >> 172.21.15.186:0/530150571 conn(0x7f9174099a50 msgr2=0x7f91740d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:30.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.534+0000 7f9178b5b700 1 -- 172.21.15.186:0/530150571 shutdown_connections 2024-05-08T21:21:30.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.534+0000 7f9178b5b700 1 -- 172.21.15.186:0/530150571 wait complete. 2024-05-08T21:21:30.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.535+0000 7f9178b5b700 1 Processor -- start 2024-05-08T21:21:30.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.535+0000 7f9178b5b700 1 -- start start 2024-05-08T21:21:30.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.536+0000 7f9178b5b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91740cbf50 0x7f91740b2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.536+0000 7f9178b5b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9174162240 0x7f91741e7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.536+0000 7f9178b5b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91740b1670 0x7f9174156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.536+0000 7f9178b5b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91741e8f30 con 0x7f91740b1670 2024-05-08T21:21:30.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.536+0000 7f9178b5b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9174157300 con 0x7f9174162240 2024-05-08T21:21:30.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.536+0000 7f9178b5b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91741e8690 con 0x7f91740cbf50 2024-05-08T21:21:30.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.536+0000 7f917259c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91740cbf50 0x7f91740b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.536+0000 7f917259c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91740cbf50 0x7f91740b2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:34686/0 (socket says 172.21.15.186:34686) 2024-05-08T21:21:30.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.536+0000 7f917259c700 1 -- 172.21.15.186:0/979859465 learned_addr learned my addr 172.21.15.186:0/979859465 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-08T21:21:30.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.536+0000 7f9171d9b700 1 --2- 172.21.15.186:0/979859465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9174162240 0x7f91741e7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.536+0000 7f9172d9d700 1 --2- 172.21.15.186:0/979859465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91740b1670 0x7f9174156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.537+0000 7f917259c700 1 -- 172.21.15.186:0/979859465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9174162240 msgr2=0x7f91741e7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.537+0000 7f917259c700 1 --2- 172.21.15.186:0/979859465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9174162240 0x7f91741e7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.537+0000 7f917259c700 1 -- 172.21.15.186:0/979859465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91740b1670 msgr2=0x7f9174156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.537+0000 7f917259c700 1 --2- 172.21.15.186:0/979859465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91740b1670 0x7f9174156530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.537+0000 7f917259c700 1 -- 172.21.15.186:0/979859465 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91680094a0 con 0x7f91740cbf50 2024-05-08T21:21:30.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.537+0000 7f9171d9b700 1 --2- 172.21.15.186:0/979859465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9174162240 0x7f91741e7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:21:30.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.537+0000 7f917259c700 1 --2- 172.21.15.186:0/979859465 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91740cbf50 0x7f91740b2010 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f91740af550 tx=0x7f9168004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:30.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.537+0000 7f9162ffd700 1 -- 172.21.15.186:0/979859465 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9168004550 con 0x7f91740cbf50 2024-05-08T21:21:30.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.537+0000 7f9162ffd700 1 -- 172.21.15.186:0/979859465 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916801f070 con 0x7f91740cbf50 2024-05-08T21:21:30.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.537+0000 7f9178b5b700 1 -- 172.21.15.186:0/979859465 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91741e8970 con 0x7f91740cbf50 2024-05-08T21:21:30.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.537+0000 7f9162ffd700 1 -- 172.21.15.186:0/979859465 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91680038e0 con 0x7f91740cbf50 2024-05-08T21:21:30.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.537+0000 7f9178b5b700 1 -- 172.21.15.186:0/979859465 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91741e8380 con 0x7f91740cbf50 2024-05-08T21:21:30.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.537+0000 7f9162ffd700 1 -- 172.21.15.186:0/979859465 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f9168003d40 con 0x7f91740cbf50 2024-05-08T21:21:30.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.537+0000 7f9162ffd700 1 -- 172.21.15.186:0/979859465 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f9168021070 con 0x7f91740cbf50 2024-05-08T21:21:30.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.538+0000 7f9178b5b700 1 -- 172.21.15.186:0/979859465 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9174074bc0 con 0x7f91740cbf50 2024-05-08T21:21:30.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.540+0000 7f9162ffd700 1 -- 172.21.15.186:0/979859465 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f9168015020 con 0x7f91740cbf50 2024-05-08T21:21:30.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.628+0000 7f9178b5b700 1 -- 172.21.15.186:0/979859465 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd new", "uuid": "36f775b1-4bc3-4889-958b-5d13b5855b4c", "id": 5} v 0) v1 -- 0x7f91741570d0 con 0x7f91740cbf50 2024-05-08T21:21:30.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.635+0000 7f9162ffd700 1 -- 172.21.15.186:0/979859465 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "36f775b1-4bc3-4889-958b-5d13b5855b4c", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9168026550 con 0x7f91740cbf50 2024-05-08T21:21:30.637 INFO:teuthology.orchestra.run.smithi186.stdout:5 2024-05-08T21:21:30.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.636+0000 7f9178b5b700 1 -- 172.21.15.186:0/979859465 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91740cbf50 msgr2=0x7f91740b2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.636+0000 7f9178b5b700 1 --2- 172.21.15.186:0/979859465 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91740cbf50 0x7f91740b2010 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f91740af550 tx=0x7f9168004750 comp rx=0 tx=0).stop 2024-05-08T21:21:30.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.636+0000 7f9178b5b700 1 -- 172.21.15.186:0/979859465 shutdown_connections 2024-05-08T21:21:30.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.636+0000 7f9178b5b700 1 --2- 172.21.15.186:0/979859465 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91740cbf50 0x7f91740b2010 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.636+0000 7f9178b5b700 1 --2- 172.21.15.186:0/979859465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91740b1670 0x7f9174156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.636+0000 7f9178b5b700 1 --2- 172.21.15.186:0/979859465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9174162240 0x7f91741e7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.636+0000 7f9178b5b700 1 -- 172.21.15.186:0/979859465 >> 172.21.15.186:0/979859465 conn(0x7f9174099a50 msgr2=0x7f917406a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:30.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.636+0000 7f9178b5b700 1 -- 172.21.15.186:0/979859465 shutdown_connections 2024-05-08T21:21:30.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.636+0000 7f9178b5b700 1 -- 172.21.15.186:0/979859465 wait complete. 2024-05-08T21:21:30.650 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 2869e118-04fe-4577-91f4-80c935cd259e 6 2024-05-08T21:21:30.713 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:30.712+0000 7f1f5b89f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-08T21:21:30.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.770+0000 7f84a4c11700 1 Processor -- start 2024-05-08T21:21:30.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.770+0000 7f84a4c11700 1 -- start start 2024-05-08T21:21:30.774 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.771+0000 7f84a4c11700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f84a015d8c0 0x7f84a00ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.774 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.771+0000 7f84a4c11700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f84a00b5550 0x7f84a01618f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.774 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.771+0000 7f84a4c11700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a0099ce0 con 0x7f84a00b50d0 2024-05-08T21:21:30.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.771+0000 7f84a4c11700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a006fde0 con 0x7f84a00b5550 2024-05-08T21:21:30.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.771+0000 7f84a4c11700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a006f260 con 0x7f84a015d8c0 2024-05-08T21:21:30.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.771+0000 7f849e59c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f84a015d8c0 0x7f84a00ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.771+0000 7f849e59c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f84a015d8c0 0x7f84a00ad260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:35610/0 (socket says 172.21.15.186:35610) 2024-05-08T21:21:30.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.771+0000 7f849e59c700 1 -- 172.21.15.186:0/2604855279 learned_addr learned my addr 172.21.15.186:0/2604855279 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-08T21:21:30.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.771+0000 7f849ed9d700 1 --2- 172.21.15.186:0/2604855279 >> v2:172.21.15.3:3300/0 conn(0x7f84a00b5550 0x7f84a01618f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.771+0000 7f849e59c700 1 -- 172.21.15.186:0/2604855279 >> v2:172.21.15.3:3300/0 conn(0x7f84a00b5550 msgr2=0x7f84a01618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:30.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.771+0000 7f849e59c700 1 --2- 172.21.15.186:0/2604855279 >> v2:172.21.15.3:3300/0 conn(0x7f84a00b5550 0x7f84a01618f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.771+0000 7f849e59c700 1 -- 172.21.15.186:0/2604855279 >> v1:172.21.15.3:6789/0 conn(0x7f84a00b50d0 legacy=0x7f84a006a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.771+0000 7f849e59c700 1 -- 172.21.15.186:0/2604855279 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a006f540 con 0x7f84a015d8c0 2024-05-08T21:21:30.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.772+0000 7f849e59c700 1 --2- 172.21.15.186:0/2604855279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a015d8c0 0x7f84a00ad260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f84a006d730 tx=0x7f8490009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82400291e7d383b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:30.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.772+0000 7f849d59a700 1 -- 172.21.15.186:0/2604855279 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849000b870 con 0x7f84a015d8c0 2024-05-08T21:21:30.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.772+0000 7f849d59a700 1 -- 172.21.15.186:0/2604855279 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849000b9d0 con 0x7f84a015d8c0 2024-05-08T21:21:30.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.772+0000 7f849d59a700 1 -- 172.21.15.186:0/2604855279 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849000bd60 con 0x7f84a015d8c0 2024-05-08T21:21:30.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.772+0000 7f84a4c11700 1 -- 172.21.15.186:0/2604855279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a015d8c0 msgr2=0x7f84a00ad260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.772+0000 7f84a4c11700 1 --2- 172.21.15.186:0/2604855279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a015d8c0 0x7f84a00ad260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f84a006d730 tx=0x7f8490009bd0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.772+0000 7f84a4c11700 1 -- 172.21.15.186:0/2604855279 shutdown_connections 2024-05-08T21:21:30.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.772+0000 7f84a4c11700 1 --2- 172.21.15.186:0/2604855279 >> v2:172.21.15.3:3300/0 conn(0x7f84a00b5550 0x7f84a01618f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.772+0000 7f84a4c11700 1 --2- 172.21.15.186:0/2604855279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a015d8c0 0x7f84a00ad260 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.772+0000 7f84a4c11700 1 -- 172.21.15.186:0/2604855279 >> 172.21.15.186:0/2604855279 conn(0x7f84a00b5a30 msgr2=0x7f84a00b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:30.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.773+0000 7f84a4c11700 1 -- 172.21.15.186:0/2604855279 shutdown_connections 2024-05-08T21:21:30.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.773+0000 7f84a4c11700 1 -- 172.21.15.186:0/2604855279 wait complete. 2024-05-08T21:21:30.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.775+0000 7f84a4c11700 1 Processor -- start 2024-05-08T21:21:30.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.775+0000 7f84a4c11700 1 -- start start 2024-05-08T21:21:30.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f84a4c11700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b50d0 0x7f84a01f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f84a4c11700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a00b5550 0x7f84a0151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f84a4c11700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a015d8c0 0x7f84a0152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:30.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f84a4c11700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a015eb90 con 0x7f84a00b50d0 2024-05-08T21:21:30.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f84a4c11700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a015ee70 con 0x7f84a00b5550 2024-05-08T21:21:30.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f84a4c11700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a015f150 con 0x7f84a015d8c0 2024-05-08T21:21:30.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f849ed9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a015d8c0 0x7f84a0152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f849dd9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a00b5550 0x7f84a0151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f849ed9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a015d8c0 0x7f84a0152390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:34702/0 (socket says 172.21.15.186:34702) 2024-05-08T21:21:30.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f849dd9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a00b5550 0x7f84a0151e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:35618/0 (socket says 172.21.15.186:35618) 2024-05-08T21:21:30.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f849ed9d700 1 -- 172.21.15.186:0/152451841 learned_addr learned my addr 172.21.15.186:0/152451841 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-08T21:21:30.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f849e59c700 1 --2- 172.21.15.186:0/152451841 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b50d0 0x7f84a01f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:30.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f849ed9d700 1 -- 172.21.15.186:0/152451841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a00b5550 msgr2=0x7f84a0151e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f849ed9d700 1 --2- 172.21.15.186:0/152451841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a00b5550 0x7f84a0151e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f849ed9d700 1 -- 172.21.15.186:0/152451841 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b50d0 msgr2=0x7f84a01f2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f849ed9d700 1 --2- 172.21.15.186:0/152451841 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b50d0 0x7f84a01f2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f849ed9d700 1 -- 172.21.15.186:0/152451841 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8490009550 con 0x7f84a015d8c0 2024-05-08T21:21:30.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.776+0000 7f849dd9b700 1 --2- 172.21.15.186:0/152451841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a00b5550 0x7f84a0151e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:21:30.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.777+0000 7f849ed9d700 1 --2- 172.21.15.186:0/152451841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a015d8c0 0x7f84a0152390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8498007e40 tx=0x7f849800e3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:30.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.777+0000 7f848effd700 1 -- 172.21.15.186:0/152451841 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849800ed30 con 0x7f84a015d8c0 2024-05-08T21:21:30.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.777+0000 7f848effd700 1 -- 172.21.15.186:0/152451841 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8498014420 con 0x7f84a015d8c0 2024-05-08T21:21:30.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.777+0000 7f84a4c11700 1 -- 172.21.15.186:0/152451841 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a015f430 con 0x7f84a015d8c0 2024-05-08T21:21:30.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.777+0000 7f848effd700 1 -- 172.21.15.186:0/152451841 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84980147d0 con 0x7f84a015d8c0 2024-05-08T21:21:30.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.777+0000 7f84a4c11700 1 -- 172.21.15.186:0/152451841 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a01f3210 con 0x7f84a015d8c0 2024-05-08T21:21:30.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.777+0000 7f848effd700 1 -- 172.21.15.186:0/152451841 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f8498014c30 con 0x7f84a015d8c0 2024-05-08T21:21:30.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.777+0000 7f848effd700 1 -- 172.21.15.186:0/152451841 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f849801ec80 con 0x7f84a015d8c0 2024-05-08T21:21:30.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.778+0000 7f84a4c11700 1 -- 172.21.15.186:0/152451841 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a0074bc0 con 0x7f84a015d8c0 2024-05-08T21:21:30.782 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:30.780+0000 7fb71cb9a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-08T21:21:30.783 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.782+0000 7f848effd700 1 -- 172.21.15.186:0/152451841 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f849801e510 con 0x7f84a015d8c0 2024-05-08T21:21:30.868 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.867+0000 7f84a4c11700 1 -- 172.21.15.186:0/152451841 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2869e118-04fe-4577-91f4-80c935cd259e", "id": 6} v 0) v1 -- 0x7f84a006b640 con 0x7f84a015d8c0 2024-05-08T21:21:30.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.875+0000 7f848effd700 1 -- 172.21.15.186:0/152451841 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2869e118-04fe-4577-91f4-80c935cd259e", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f849801e6b0 con 0x7f84a015d8c0 2024-05-08T21:21:30.877 INFO:teuthology.orchestra.run.smithi186.stdout:6 2024-05-08T21:21:30.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.876+0000 7f84a4c11700 1 -- 172.21.15.186:0/152451841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a015d8c0 msgr2=0x7f84a0152390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:30.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.876+0000 7f84a4c11700 1 --2- 172.21.15.186:0/152451841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a015d8c0 0x7f84a0152390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8498007e40 tx=0x7f849800e3c0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.876+0000 7f84a4c11700 1 -- 172.21.15.186:0/152451841 shutdown_connections 2024-05-08T21:21:30.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.876+0000 7f84a4c11700 1 --2- 172.21.15.186:0/152451841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a015d8c0 0x7f84a0152390 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.876+0000 7f84a4c11700 1 --2- 172.21.15.186:0/152451841 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b50d0 0x7f84a01f2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.876+0000 7f84a4c11700 1 --2- 172.21.15.186:0/152451841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a00b5550 0x7f84a0151e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:30.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.876+0000 7f84a4c11700 1 -- 172.21.15.186:0/152451841 >> 172.21.15.186:0/152451841 conn(0x7f84a00b5a30 msgr2=0x7f84a006a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:30.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.876+0000 7f84a4c11700 1 -- 172.21.15.186:0/152451841 shutdown_connections 2024-05-08T21:21:30.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:30.876+0000 7f84a4c11700 1 -- 172.21.15.186:0/152451841 wait complete. 2024-05-08T21:21:30.889 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 89b4d094-f7dd-4110-8601-0263df2c6cfa 7 2024-05-08T21:21:31.014 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.013+0000 7f480081f700 1 Processor -- start 2024-05-08T21:21:31.014 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.013+0000 7f480081f700 1 -- start start 2024-05-08T21:21:31.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f480081f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47fc15b4e0 0x7f47fc0b1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:31.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f480081f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f47fc0d1910 0x7f47fc0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:31.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f480081f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47fc0cf560 con 0x7f47fc15b8c0 2024-05-08T21:21:31.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f480081f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47fc0cefe0 con 0x7f47fc0d1910 2024-05-08T21:21:31.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f480081f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc0ccff0 con 0x7f47fc15b4e0 2024-05-08T21:21:31.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f47fa59c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47fc15b4e0 0x7f47fc0b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:31.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f47fa59c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47fc15b4e0 0x7f47fc0b1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:34712/0 (socket says 172.21.15.186:34712) 2024-05-08T21:21:31.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f47fa59c700 1 -- 172.21.15.186:0/1799327036 learned_addr learned my addr 172.21.15.186:0/1799327036 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-08T21:21:31.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f47fad9d700 1 --2- 172.21.15.186:0/1799327036 >> v2:172.21.15.3:3300/0 conn(0x7f47fc0d1910 0x7f47fc0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:31.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f47fa59c700 1 -- 172.21.15.186:0/1799327036 >> v2:172.21.15.3:3300/0 conn(0x7f47fc0d1910 msgr2=0x7f47fc0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:31.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f47fa59c700 1 --2- 172.21.15.186:0/1799327036 >> v2:172.21.15.3:3300/0 conn(0x7f47fc0d1910 0x7f47fc0cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f47fa59c700 1 -- 172.21.15.186:0/1799327036 >> v1:172.21.15.3:6789/0 conn(0x7f47fc15b8c0 legacy=0x7f47fc0d1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:31.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f47fa59c700 1 -- 172.21.15.186:0/1799327036 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc0cd2d0 con 0x7f47fc15b4e0 2024-05-08T21:21:31.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f47fad9d700 1 --2- 172.21.15.186:0/1799327036 >> v2:172.21.15.3:3300/0 conn(0x7f47fc0d1910 0x7f47fc0cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:21:31.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f47fa59c700 1 --2- 172.21.15.186:0/1799327036 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47fc15b4e0 0x7f47fc0b1200 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f47ec0098f0 tx=0x7f47ec009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a6c1478d1b9eddc server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:31.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f47f959a700 1 -- 172.21.15.186:0/1799327036 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec00b8a0 con 0x7f47fc15b4e0 2024-05-08T21:21:31.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f47f959a700 1 -- 172.21.15.186:0/1799327036 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec00ba00 con 0x7f47fc15b4e0 2024-05-08T21:21:31.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.014+0000 7f47f959a700 1 -- 172.21.15.186:0/1799327036 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec00bd90 con 0x7f47fc15b4e0 2024-05-08T21:21:31.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.015+0000 7f480081f700 1 -- 172.21.15.186:0/1799327036 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47fc15b4e0 msgr2=0x7f47fc0b1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:31.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.015+0000 7f480081f700 1 --2- 172.21.15.186:0/1799327036 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47fc15b4e0 0x7f47fc0b1200 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f47ec0098f0 tx=0x7f47ec009c00 comp rx=0 tx=0).stop 2024-05-08T21:21:31.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.015+0000 7f480081f700 1 -- 172.21.15.186:0/1799327036 shutdown_connections 2024-05-08T21:21:31.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.015+0000 7f480081f700 1 --2- 172.21.15.186:0/1799327036 >> v2:172.21.15.3:3300/0 conn(0x7f47fc0d1910 0x7f47fc0cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.015+0000 7f480081f700 1 --2- 172.21.15.186:0/1799327036 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47fc15b4e0 0x7f47fc0b1200 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.015+0000 7f480081f700 1 -- 172.21.15.186:0/1799327036 >> 172.21.15.186:0/1799327036 conn(0x7f47fc08afe0 msgr2=0x7f47fc08b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:31.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.015+0000 7f480081f700 1 -- 172.21.15.186:0/1799327036 shutdown_connections 2024-05-08T21:21:31.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.015+0000 7f480081f700 1 -- 172.21.15.186:0/1799327036 wait complete. 2024-05-08T21:21:31.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.017+0000 7f480081f700 1 Processor -- start 2024-05-08T21:21:31.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.017+0000 7f480081f700 1 -- start start 2024-05-08T21:21:31.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.017+0000 7f480081f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47fc0d1910 0x7f47fc156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:31.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.017+0000 7f480081f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47fc15b4e0 0x7f47fc0b2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:31.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.017+0000 7f480081f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47fc15b8c0 0x7f47fc0b2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:31.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.017+0000 7f480081f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc0c5f50 con 0x7f47fc0d1910 2024-05-08T21:21:31.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.017+0000 7f480081f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc0c5c70 con 0x7f47fc15b4e0 2024-05-08T21:21:31.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.017+0000 7f480081f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc0c6300 con 0x7f47fc15b8c0 2024-05-08T21:21:31.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.017+0000 7f47fad9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47fc15b8c0 0x7f47fc0b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:31.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.017+0000 7f47f9d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47fc15b4e0 0x7f47fc0b2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:31.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f47fad9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47fc15b8c0 0x7f47fc0b2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:34718/0 (socket says 172.21.15.186:34718) 2024-05-08T21:21:31.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f47fad9d700 1 -- 172.21.15.186:0/557923790 learned_addr learned my addr 172.21.15.186:0/557923790 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-08T21:21:31.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f47fa59c700 1 --2- 172.21.15.186:0/557923790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47fc0d1910 0x7f47fc156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:31.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f47fad9d700 1 -- 172.21.15.186:0/557923790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47fc15b4e0 msgr2=0x7f47fc0b2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:31.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f47fad9d700 1 --2- 172.21.15.186:0/557923790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47fc15b4e0 0x7f47fc0b2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f47fad9d700 1 -- 172.21.15.186:0/557923790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47fc0d1910 msgr2=0x7f47fc156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:31.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f47fad9d700 1 --2- 172.21.15.186:0/557923790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47fc0d1910 0x7f47fc156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f47fad9d700 1 -- 172.21.15.186:0/557923790 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec009550 con 0x7f47fc15b8c0 2024-05-08T21:21:31.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f47f9d9b700 1 --2- 172.21.15.186:0/557923790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47fc15b4e0 0x7f47fc0b2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:21:31.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f47fad9d700 1 --2- 172.21.15.186:0/557923790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47fc15b8c0 0x7f47fc0b2890 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009fa0 tx=0x7f47f400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:31.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f47eaffd700 1 -- 172.21.15.186:0/557923790 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f400ddb0 con 0x7f47fc15b8c0 2024-05-08T21:21:31.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f47eaffd700 1 -- 172.21.15.186:0/557923790 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f400f040 con 0x7f47fc15b8c0 2024-05-08T21:21:31.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f480081f700 1 -- 172.21.15.186:0/557923790 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47fc0c65e0 con 0x7f47fc15b8c0 2024-05-08T21:21:31.023 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f47eaffd700 1 -- 172.21.15.186:0/557923790 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f4003c60 con 0x7f47fc15b8c0 2024-05-08T21:21:31.023 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f480081f700 1 -- 172.21.15.186:0/557923790 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47fc1572b0 con 0x7f47fc15b8c0 2024-05-08T21:21:31.023 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f47eaffd700 1 -- 172.21.15.186:0/557923790 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f47f4004330 con 0x7f47fc15b8c0 2024-05-08T21:21:31.023 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.018+0000 7f47eaffd700 1 -- 172.21.15.186:0/557923790 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f47f4015aa0 con 0x7f47fc15b8c0 2024-05-08T21:21:31.023 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.019+0000 7f480081f700 1 -- 172.21.15.186:0/557923790 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47fc0c6160 con 0x7f47fc15b8c0 2024-05-08T21:21:31.023 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.021+0000 7f47eaffd700 1 -- 172.21.15.186:0/557923790 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f47fc0c6160 con 0x7f47fc15b8c0 2024-05-08T21:21:31.062 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:31.060+0000 7f1f5b89f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-08T21:21:31.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.106+0000 7f480081f700 1 -- 172.21.15.186:0/557923790 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd new", "uuid": "89b4d094-f7dd-4110-8601-0263df2c6cfa", "id": 7} v 0) v1 -- 0x7f47fc074bc0 con 0x7f47fc15b8c0 2024-05-08T21:21:31.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.116+0000 7f47eaffd700 1 -- 172.21.15.186:0/557923790 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "89b4d094-f7dd-4110-8601-0263df2c6cfa", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f47f401a070 con 0x7f47fc15b8c0 2024-05-08T21:21:31.118 INFO:teuthology.orchestra.run.smithi186.stdout:7 2024-05-08T21:21:31.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.117+0000 7f480081f700 1 -- 172.21.15.186:0/557923790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47fc15b8c0 msgr2=0x7f47fc0b2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:31.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.117+0000 7f480081f700 1 --2- 172.21.15.186:0/557923790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47fc15b8c0 0x7f47fc0b2890 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009fa0 tx=0x7f47f400d3b0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.117+0000 7f480081f700 1 -- 172.21.15.186:0/557923790 shutdown_connections 2024-05-08T21:21:31.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.117+0000 7f480081f700 1 --2- 172.21.15.186:0/557923790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47fc15b8c0 0x7f47fc0b2890 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.117+0000 7f480081f700 1 --2- 172.21.15.186:0/557923790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47fc0d1910 0x7f47fc156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.117+0000 7f480081f700 1 --2- 172.21.15.186:0/557923790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47fc15b4e0 0x7f47fc0b2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.117+0000 7f480081f700 1 -- 172.21.15.186:0/557923790 >> 172.21.15.186:0/557923790 conn(0x7f47fc08afe0 msgr2=0x7f47fc06fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:31.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.117+0000 7f480081f700 1 -- 172.21.15.186:0/557923790 shutdown_connections 2024-05-08T21:21:31.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-08T21:21:31.117+0000 7f480081f700 1 -- 172.21.15.186:0/557923790 wait complete. 2024-05-08T21:21:31.130 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-08T21:21:31.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-05-08T21:21:31.133 INFO:tasks.ceph.osd.0:Started 2024-05-08T21:21:31.133 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-08T21:21:31.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-05-08T21:21:31.135 INFO:tasks.ceph.osd.1:Started 2024-05-08T21:21:31.135 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-08T21:21:31.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-05-08T21:21:31.137 INFO:tasks.ceph.osd.2:Started 2024-05-08T21:21:31.137 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-08T21:21:31.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-05-08T21:21:31.139 INFO:tasks.ceph.osd.3:Started 2024-05-08T21:21:31.140 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-08T21:21:31.140 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-05-08T21:21:31.141 INFO:tasks.ceph.osd.4:Started 2024-05-08T21:21:31.142 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-08T21:21:31.142 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-05-08T21:21:31.143 INFO:tasks.ceph.osd.5:Started 2024-05-08T21:21:31.144 INFO:tasks.ceph.osd.6:Restarting daemon 2024-05-08T21:21:31.144 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-05-08T21:21:31.146 INFO:tasks.ceph.osd.6:Started 2024-05-08T21:21:31.146 INFO:tasks.ceph.osd.7:Restarting daemon 2024-05-08T21:21:31.146 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-05-08T21:21:31.149 INFO:tasks.ceph.osd.7:Started 2024-05-08T21:21:31.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:21:31.162 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:31.159+0000 7fb71cb9a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-08T21:21:31.252 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:31.250+0000 7f1f5b89f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-08T21:21:31.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.329+0000 7fbb9e19b700 1 Processor -- start 2024-05-08T21:21:31.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.329+0000 7fbb9e19b700 1 -- start start 2024-05-08T21:21:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.329+0000 7fbb9e19b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbb980cd1a0 0x7fbb980b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.329+0000 7fbb9e19b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbb980af1c0 0x7fbb980b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb9e19b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb980ce8c0 con 0x7fbb98157f30 2024-05-08T21:21:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb9e19b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb9815c0d0 con 0x7fbb980af1c0 2024-05-08T21:21:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb9e19b700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb9815ae40 con 0x7fbb980cd1a0 2024-05-08T21:21:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb97fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbb980af1c0 0x7fbb980b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb977fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbb98157f30 0x7fbb9815b9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54598/0 (socket says 172.21.15.3:54598) 2024-05-08T21:21:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb977fe700 1 -- 172.21.15.3:0/544296830 learned_addr learned my addr 172.21.15.3:0/544296830 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb97fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbb980af1c0 0x7fbb980b4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35424/0 (socket says 172.21.15.3:35424) 2024-05-08T21:21:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb96ffd700 1 --2- 172.21.15.3:0/544296830 >> v2:172.21.15.186:3300/0 conn(0x7fbb980cd1a0 0x7fbb980b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:31.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb97fff700 1 -- 172.21.15.3:0/544296830 >> v2:172.21.15.186:3300/0 conn(0x7fbb980cd1a0 msgr2=0x7fbb980b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:31.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb97fff700 1 --2- 172.21.15.3:0/544296830 >> v2:172.21.15.186:3300/0 conn(0x7fbb980cd1a0 0x7fbb980b3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb97fff700 1 -- 172.21.15.3:0/544296830 >> v1:172.21.15.3:6789/0 conn(0x7fbb98157f30 legacy=0x7fbb9815b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:31.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb97fff700 1 -- 172.21.15.3:0/544296830 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9815b120 con 0x7fbb980af1c0 2024-05-08T21:21:31.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb967fc700 1 -- 172.21.15.3:0/544296830 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035128617 0 0) 0x7fbb980ce8c0 con 0x7fbb98157f30 2024-05-08T21:21:31.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb97fff700 1 --2- 172.21.15.3:0/544296830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb980af1c0 0x7fbb980b4340 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbb980727a0 tx=0x7fbb90009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b96e110c6c8d1e82 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:31.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb967fc700 1 -- 172.21.15.3:0/544296830 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb90011830 con 0x7fbb980af1c0 2024-05-08T21:21:31.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb967fc700 1 -- 172.21.15.3:0/544296830 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb90011990 con 0x7fbb980af1c0 2024-05-08T21:21:31.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.330+0000 7fbb967fc700 1 -- 172.21.15.3:0/544296830 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb90011d20 con 0x7fbb980af1c0 2024-05-08T21:21:31.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.331+0000 7fbb9e19b700 1 -- 172.21.15.3:0/544296830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb980af1c0 msgr2=0x7fbb980b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:31.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.331+0000 7fbb9e19b700 1 --2- 172.21.15.3:0/544296830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb980af1c0 0x7fbb980b4340 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbb980727a0 tx=0x7fbb90009b20 comp rx=0 tx=0).stop 2024-05-08T21:21:31.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.331+0000 7fbb9e19b700 1 -- 172.21.15.3:0/544296830 shutdown_connections 2024-05-08T21:21:31.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.331+0000 7fbb9e19b700 1 --2- 172.21.15.3:0/544296830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb980af1c0 0x7fbb980b4340 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.331+0000 7fbb9e19b700 1 --2- 172.21.15.3:0/544296830 >> v2:172.21.15.186:3300/0 conn(0x7fbb980cd1a0 0x7fbb980b3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.331+0000 7fbb9e19b700 1 -- 172.21.15.3:0/544296830 >> 172.21.15.3:0/544296830 conn(0x7fbb980722a0 msgr2=0x7fbb9806a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:31.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.331+0000 7fbb9e19b700 1 -- 172.21.15.3:0/544296830 shutdown_connections 2024-05-08T21:21:31.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.331+0000 7fbb9e19b700 1 -- 172.21.15.3:0/544296830 wait complete. 2024-05-08T21:21:31.336 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:31.332+0000 7f1f5b89f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-08T21:21:31.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.332+0000 7fbb9e19b700 1 Processor -- start 2024-05-08T21:21:31.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.333+0000 7fbb9e19b700 1 -- start start 2024-05-08T21:21:31.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.333+0000 7fbb9e19b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb980af1c0 0x7fbb98145f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:31.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.333+0000 7fbb9e19b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb980cd1a0 0x7fbb980a92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:31.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.333+0000 7fbb9e19b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb98157f30 0x7fbb981e2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:31.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.333+0000 7fbb9e19b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb98146480 con 0x7fbb980cd1a0 2024-05-08T21:21:31.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.333+0000 7fbb9e19b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb980a9830 con 0x7fbb98157f30 2024-05-08T21:21:31.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.333+0000 7fbb9e19b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb98146670 con 0x7fbb980af1c0 2024-05-08T21:21:31.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.334+0000 7fbb96ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb980cd1a0 0x7fbb980a92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:31.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.334+0000 7fbb96ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb980cd1a0 0x7fbb980a92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35436/0 (socket says 172.21.15.3:35436) 2024-05-08T21:21:31.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.334+0000 7fbb96ffd700 1 -- 172.21.15.3:0/987396982 learned_addr learned my addr 172.21.15.3:0/987396982 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:31.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.334+0000 7fbb977fe700 1 --2- 172.21.15.3:0/987396982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb980af1c0 0x7fbb98145f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:31.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.334+0000 7fbb97fff700 1 --2- 172.21.15.3:0/987396982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb98157f30 0x7fbb981e2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:31.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.334+0000 7fbb96ffd700 1 -- 172.21.15.3:0/987396982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb980af1c0 msgr2=0x7fbb98145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:31.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.334+0000 7fbb96ffd700 1 --2- 172.21.15.3:0/987396982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb980af1c0 0x7fbb98145f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.334+0000 7fbb96ffd700 1 -- 172.21.15.3:0/987396982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb98157f30 msgr2=0x7fbb981e2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:31.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.334+0000 7fbb96ffd700 1 --2- 172.21.15.3:0/987396982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb98157f30 0x7fbb981e2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.334+0000 7fbb96ffd700 1 -- 172.21.15.3:0/987396982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb900094a0 con 0x7fbb980cd1a0 2024-05-08T21:21:31.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.334+0000 7fbb96ffd700 1 --2- 172.21.15.3:0/987396982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb980cd1a0 0x7fbb980a92f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c00e980 tx=0x7fbb8c00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:31.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.334+0000 7fbb7ffff700 1 -- 172.21.15.3:0/987396982 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c009710 con 0x7fbb980cd1a0 2024-05-08T21:21:31.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.334+0000 7fbb7ffff700 1 -- 172.21.15.3:0/987396982 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8c013680 con 0x7fbb980cd1a0 2024-05-08T21:21:31.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.334+0000 7fbb7ffff700 1 -- 172.21.15.3:0/987396982 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c013a30 con 0x7fbb980cd1a0 2024-05-08T21:21:31.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.334+0000 7fbb9e19b700 1 -- 172.21.15.3:0/987396982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb981469b0 con 0x7fbb980cd1a0 2024-05-08T21:21:31.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.334+0000 7fbb9e19b700 1 -- 172.21.15.3:0/987396982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb981471f0 con 0x7fbb980cd1a0 2024-05-08T21:21:31.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.335+0000 7fbb7ffff700 1 -- 172.21.15.3:0/987396982 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fbb8c005a20 con 0x7fbb980cd1a0 2024-05-08T21:21:31.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.335+0000 7fbb7ffff700 1 -- 172.21.15.3:0/987396982 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fbb8c005cd0 con 0x7fbb980cd1a0 2024-05-08T21:21:31.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.335+0000 7fbb9e19b700 1 -- 172.21.15.3:0/987396982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb980ce4a0 con 0x7fbb980cd1a0 2024-05-08T21:21:31.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.338+0000 7fbb7ffff700 1 -- 172.21.15.3:0/987396982 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fbb8c021060 con 0x7fbb980cd1a0 2024-05-08T21:21:31.380 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:31.378+0000 7fb71cb9a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-08T21:21:31.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.414+0000 7fbb9e19b700 1 -- 172.21.15.3:0/987396982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb98068000 con 0x7fbb980cd1a0 2024-05-08T21:21:31.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.414+0000 7fbb7ffff700 1 -- 172.21.15.3:0/987396982 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fbb8c029020 con 0x7fbb980cd1a0 2024-05-08T21:21:31.418 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:21:31.418 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":10,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:31.108524+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:21:31.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.415+0000 7fbb9e19b700 1 -- 172.21.15.3:0/987396982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb980cd1a0 msgr2=0x7fbb980a92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:31.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.415+0000 7fbb9e19b700 1 --2- 172.21.15.3:0/987396982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb980cd1a0 0x7fbb980a92f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c00e980 tx=0x7fbb8c00ec90 comp rx=0 tx=0).stop 2024-05-08T21:21:31.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.415+0000 7fbb9e19b700 1 -- 172.21.15.3:0/987396982 shutdown_connections 2024-05-08T21:21:31.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.415+0000 7fbb9e19b700 1 --2- 172.21.15.3:0/987396982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb980af1c0 0x7fbb98145f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.415+0000 7fbb9e19b700 1 --2- 172.21.15.3:0/987396982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb98157f30 0x7fbb981e2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.415+0000 7fbb9e19b700 1 --2- 172.21.15.3:0/987396982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb980cd1a0 0x7fbb980a92f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:31.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.415+0000 7fbb9e19b700 1 -- 172.21.15.3:0/987396982 >> 172.21.15.3:0/987396982 conn(0x7fbb980722a0 msgr2=0x7fbb98158620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:31.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.415+0000 7fbb9e19b700 1 -- 172.21.15.3:0/987396982 shutdown_connections 2024-05-08T21:21:31.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:31.415+0000 7fbb9e19b700 1 -- 172.21.15.3:0/987396982 wait complete. 2024-05-08T21:21:31.432 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-08T21:21:31.432 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-08T21:21:31.471 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:31.469+0000 7fb71cb9a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-08T21:21:31.743 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-05-08T21:21:31.741+0000 7f6ed208c7c0 -1 Falling back to public interface 2024-05-08T21:21:31.761 INFO:tasks.ceph.osd.3.smithi003.stderr:2024-05-08T21:21:31.759+0000 7f717e40b7c0 -1 Falling back to public interface 2024-05-08T21:21:31.764 INFO:tasks.ceph.osd.2.smithi003.stderr:2024-05-08T21:21:31.761+0000 7f11e514c7c0 -1 Falling back to public interface 2024-05-08T21:21:31.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-08T21:21:31.768+0000 7fe62eecd7c0 -1 Falling back to public interface 2024-05-08T21:21:31.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-08T21:21:31.769+0000 7fb2e51de7c0 -1 Falling back to public interface 2024-05-08T21:21:31.771 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-08T21:21:31.769+0000 7fdd4640b7c0 -1 Falling back to public interface 2024-05-08T21:21:31.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-08T21:21:31.769+0000 7fb2dc59a700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6809/0: (98) Address already in use 2024-05-08T21:21:31.771 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-08T21:21:31.770+0000 7fdd3d7c7700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6817/0: (98) Address already in use 2024-05-08T21:21:31.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-08T21:21:31.771+0000 7fe626289700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6824/0: (98) Address already in use 2024-05-08T21:21:31.777 INFO:tasks.ceph.osd.1.smithi003.stderr:2024-05-08T21:21:31.774+0000 7f5fe7f8c7c0 -1 Falling back to public interface 2024-05-08T21:21:31.795 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-05-08T21:21:31.792+0000 7fdd7fc797c0 -1 Falling back to public interface 2024-05-08T21:21:32.052 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:32.050+0000 7f1f5b89f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-08T21:21:32.135 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:32.134+0000 7f1f5b89f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-08T21:21:32.238 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-08T21:21:32.268 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:32.267+0000 7f1f5b89f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-08T21:21:32.271 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:32.268+0000 7fb71cb9a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-08T21:21:32.364 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.362+0000 7fdeef0e5700 1 Processor -- start 2024-05-08T21:21:32.364 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.362+0000 7fdeef0e5700 1 -- start start 2024-05-08T21:21:32.366 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeef0e5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdee815da90 0x7fdee80b1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:32.367 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeef0e5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdee80b4e70 0x7fdee80ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:32.367 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeef0e5700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdee80ad7a0 0x7fdee80b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:32.367 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeef0e5700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee80b5630 con 0x7fdee815da90 2024-05-08T21:21:32.367 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeef0e5700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee80b1dc0 con 0x7fdee80ad7a0 2024-05-08T21:21:32.367 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeef0e5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee815de70 con 0x7fdee80b4e70 2024-05-08T21:21:32.367 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeece81700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdee815da90 0x7fdee80b1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:32.367 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeece81700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdee815da90 0x7fdee80b1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35450/0 (socket says 172.21.15.3:35450) 2024-05-08T21:21:32.367 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeece81700 1 -- 172.21.15.3:0/3621842382 learned_addr learned my addr 172.21.15.3:0/3621842382 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:32.367 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdee7fff700 1 --2- 172.21.15.3:0/3621842382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdee80b4e70 0x7fdee80ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:32.367 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeed682700 1 --2- 172.21.15.3:0/3621842382 >> v2:172.21.15.186:3300/0 conn(0x7fdee80ad7a0 0x7fdee80b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:32.367 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeece81700 1 -- 172.21.15.3:0/3621842382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdee80b4e70 msgr2=0x7fdee80ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:32.367 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeece81700 1 --2- 172.21.15.3:0/3621842382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdee80b4e70 0x7fdee80ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:32.367 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeece81700 1 -- 172.21.15.3:0/3621842382 >> v2:172.21.15.186:3300/0 conn(0x7fdee80ad7a0 msgr2=0x7fdee80b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:32.368 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeece81700 1 --2- 172.21.15.3:0/3621842382 >> v2:172.21.15.186:3300/0 conn(0x7fdee80ad7a0 0x7fdee80b08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:32.368 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeece81700 1 -- 172.21.15.3:0/3621842382 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee80b5350 con 0x7fdee815da90 2024-05-08T21:21:32.368 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdee7fff700 1 --2- 172.21.15.3:0/3621842382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdee80b4e70 0x7fdee80ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:21:32.368 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeed682700 1 --2- 172.21.15.3:0/3621842382 >> v2:172.21.15.186:3300/0 conn(0x7fdee80ad7a0 0x7fdee80b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:21:32.368 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.364+0000 7fdeece81700 1 --2- 172.21.15.3:0/3621842382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee815da90 0x7fdee80b1880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdee8072d00 tx=0x7fdedc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=101b47059d3aaa43 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:32.368 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.365+0000 7fdee77fe700 1 -- 172.21.15.3:0/3621842382 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc00b870 con 0x7fdee815da90 2024-05-08T21:21:32.368 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.365+0000 7fdee77fe700 1 -- 172.21.15.3:0/3621842382 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdedc00b9d0 con 0x7fdee815da90 2024-05-08T21:21:32.368 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.365+0000 7fdee77fe700 1 -- 172.21.15.3:0/3621842382 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc00bd60 con 0x7fdee815da90 2024-05-08T21:21:32.368 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:32.365+0000 7fb71cb9a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-08T21:21:32.371 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.365+0000 7fdeef0e5700 1 -- 172.21.15.3:0/3621842382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee815da90 msgr2=0x7fdee80b1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:32.371 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.365+0000 7fdeef0e5700 1 --2- 172.21.15.3:0/3621842382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee815da90 0x7fdee80b1880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdee8072d00 tx=0x7fdedc009bd0 comp rx=0 tx=0).stop 2024-05-08T21:21:32.371 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.365+0000 7fdeef0e5700 1 -- 172.21.15.3:0/3621842382 shutdown_connections 2024-05-08T21:21:32.371 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.365+0000 7fdeef0e5700 1 --2- 172.21.15.3:0/3621842382 >> v2:172.21.15.186:3300/0 conn(0x7fdee80ad7a0 0x7fdee80b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:32.371 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.365+0000 7fdeef0e5700 1 --2- 172.21.15.3:0/3621842382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee815da90 0x7fdee80b1880 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:32.371 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.365+0000 7fdeef0e5700 1 --2- 172.21.15.3:0/3621842382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdee80b4e70 0x7fdee80ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:32.371 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.365+0000 7fdeef0e5700 1 -- 172.21.15.3:0/3621842382 >> 172.21.15.3:0/3621842382 conn(0x7fdee80b57e0 msgr2=0x7fdee80b5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:32.372 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.365+0000 7fdeef0e5700 1 -- 172.21.15.3:0/3621842382 shutdown_connections 2024-05-08T21:21:32.372 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.365+0000 7fdeef0e5700 1 -- 172.21.15.3:0/3621842382 wait complete. 2024-05-08T21:21:32.372 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.366+0000 7fdeef0e5700 1 Processor -- start 2024-05-08T21:21:32.372 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.366+0000 7fdeef0e5700 1 -- start start 2024-05-08T21:21:32.372 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdeef0e5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdee80ad7a0 0x7fdee815a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:32.372 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdeef0e5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdee80b4e70 0x7fdee80cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:32.372 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdeef0e5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee815da90 0x7fdee80cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:32.372 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdeef0e5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee815b3f0 con 0x7fdee815da90 2024-05-08T21:21:32.372 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdeef0e5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee8148510 con 0x7fdee80ad7a0 2024-05-08T21:21:32.372 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdeef0e5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee815ae30 con 0x7fdee80b4e70 2024-05-08T21:21:32.372 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdeed682700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee815da90 0x7fdee80cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:32.372 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdeed682700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee815da90 0x7fdee80cc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35458/0 (socket says 172.21.15.3:35458) 2024-05-08T21:21:32.372 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdeed682700 1 -- 172.21.15.3:0/1733542520 learned_addr learned my addr 172.21.15.3:0/1733542520 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:32.373 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdee7fff700 1 --2- 172.21.15.3:0/1733542520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdee80b4e70 0x7fdee80cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:32.373 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdeece81700 1 --2- 172.21.15.3:0/1733542520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdee80ad7a0 0x7fdee815a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:32.373 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdeed682700 1 -- 172.21.15.3:0/1733542520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdee80b4e70 msgr2=0x7fdee80cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:32.373 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdeed682700 1 --2- 172.21.15.3:0/1733542520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdee80b4e70 0x7fdee80cbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:32.373 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdeed682700 1 -- 172.21.15.3:0/1733542520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdee80ad7a0 msgr2=0x7fdee815a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:32.373 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdeed682700 1 --2- 172.21.15.3:0/1733542520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdee80ad7a0 0x7fdee815a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:32.373 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdeed682700 1 -- 172.21.15.3:0/1733542520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdedc009550 con 0x7fdee815da90 2024-05-08T21:21:32.373 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdeed682700 1 --2- 172.21.15.3:0/1733542520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee815da90 0x7fdee80cc0d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdee0007de0 tx=0x7fdee000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:32.373 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.367+0000 7fdee57fa700 1 -- 172.21.15.3:0/1733542520 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee000dd20 con 0x7fdee815da90 2024-05-08T21:21:32.373 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.368+0000 7fdee57fa700 1 -- 172.21.15.3:0/1733542520 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee000f040 con 0x7fdee815da90 2024-05-08T21:21:32.373 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.368+0000 7fdeef0e5700 1 -- 172.21.15.3:0/1733542520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee815b110 con 0x7fdee815da90 2024-05-08T21:21:32.373 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.368+0000 7fdee57fa700 1 -- 172.21.15.3:0/1733542520 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee0003c60 con 0x7fdee815da90 2024-05-08T21:21:32.373 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.368+0000 7fdeef0e5700 1 -- 172.21.15.3:0/1733542520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee815aa00 con 0x7fdee815da90 2024-05-08T21:21:32.373 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.368+0000 7fdee57fa700 1 -- 172.21.15.3:0/1733542520 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fdee0004330 con 0x7fdee815da90 2024-05-08T21:21:32.373 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.368+0000 7fdee57fa700 1 -- 172.21.15.3:0/1733542520 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fdee0015b10 con 0x7fdee815da90 2024-05-08T21:21:32.374 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.370+0000 7fdeef0e5700 1 -- 172.21.15.3:0/1733542520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee8148680 con 0x7fdee815da90 2024-05-08T21:21:32.374 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.371+0000 7fdee57fa700 1 -- 172.21.15.3:0/1733542520 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fdee001a070 con 0x7fdee815da90 2024-05-08T21:21:32.450 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.448+0000 7fdeef0e5700 1 -- 172.21.15.3:0/1733542520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdee804ec60 con 0x7fdee815da90 2024-05-08T21:21:32.451 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.449+0000 7fdee57fa700 1 -- 172.21.15.3:0/1733542520 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fdee0004810 con 0x7fdee815da90 2024-05-08T21:21:32.451 INFO:teuthology.misc.health.smithi003.stdout: 2024-05-08T21:21:32.451 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:31.108524+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:21:32.452 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.450+0000 7fdeef0e5700 1 -- 172.21.15.3:0/1733542520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee815da90 msgr2=0x7fdee80cc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:32.453 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.450+0000 7fdeef0e5700 1 --2- 172.21.15.3:0/1733542520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee815da90 0x7fdee80cc0d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdee0007de0 tx=0x7fdee000d3b0 comp rx=0 tx=0).stop 2024-05-08T21:21:32.453 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.450+0000 7fdeef0e5700 1 -- 172.21.15.3:0/1733542520 shutdown_connections 2024-05-08T21:21:32.453 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.450+0000 7fdeef0e5700 1 --2- 172.21.15.3:0/1733542520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee815da90 0x7fdee80cc0d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:32.453 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.451+0000 7fdeef0e5700 1 --2- 172.21.15.3:0/1733542520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdee80ad7a0 0x7fdee815a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:32.453 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.451+0000 7fdeef0e5700 1 --2- 172.21.15.3:0/1733542520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdee80b4e70 0x7fdee80cbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:32.453 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.451+0000 7fdeef0e5700 1 -- 172.21.15.3:0/1733542520 >> 172.21.15.3:0/1733542520 conn(0x7fdee80b57e0 msgr2=0x7fdee815ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:32.453 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.451+0000 7fdeef0e5700 1 -- 172.21.15.3:0/1733542520 shutdown_connections 2024-05-08T21:21:32.453 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:32.451+0000 7fdeef0e5700 1 -- 172.21.15.3:0/1733542520 wait complete. 2024-05-08T21:21:32.469 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-08T21:21:32.527 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:32.525+0000 7fb71cb9a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-08T21:21:32.830 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:32.829+0000 7f1f5b89f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-08T21:21:33.011 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:33.010+0000 7f1f5b89f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-08T21:21:33.162 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:33.160+0000 7fb71cb9a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-08T21:21:33.180 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:33.179+0000 7f1f5b89f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-08T21:21:33.345 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:33.344+0000 7fb71cb9a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-08T21:21:33.530 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:33.529+0000 7fb71cb9a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-08T21:21:34.013 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:34.011+0000 7f1f5b89f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-08T21:21:34.261 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:34.259+0000 7f1f5b89f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-08T21:21:34.336 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:34.335+0000 7f1f5b89f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-08T21:21:34.467 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:34.465+0000 7fb71cb9a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-08T21:21:34.506 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:34.504+0000 7f1f5b89f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-08T21:21:34.794 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:34.793+0000 7fb71cb9a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-08T21:21:34.878 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:34.876+0000 7fb71cb9a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-08T21:21:35.074 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-05-08T21:21:35.072+0000 7fb71cb9a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-08T21:21:35.252 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-05-08T21:21:35.251+0000 7f6ed208c7c0 -1 osd.5 0 log_to_monitors true 2024-05-08T21:21:35.278 INFO:tasks.ceph.osd.3.smithi003.stderr:2024-05-08T21:21:35.277+0000 7f717e40b7c0 -1 osd.3 0 log_to_monitors true 2024-05-08T21:21:35.297 INFO:tasks.ceph.osd.1.smithi003.stderr:2024-05-08T21:21:35.295+0000 7f5fe7f8c7c0 -1 osd.1 0 log_to_monitors true 2024-05-08T21:21:35.306 INFO:tasks.ceph.osd.2.smithi003.stderr:2024-05-08T21:21:35.304+0000 7f11e514c7c0 -1 osd.2 0 log_to_monitors true 2024-05-08T21:21:35.307 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-08T21:21:35.306+0000 7fdd4640b7c0 -1 osd.7 0 log_to_monitors true 2024-05-08T21:21:35.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-08T21:21:35.315+0000 7fb2e51de7c0 -1 osd.4 0 log_to_monitors true 2024-05-08T21:21:35.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-08T21:21:35.318+0000 7fe62eecd7c0 -1 osd.6 0 log_to_monitors true 2024-05-08T21:21:35.331 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-05-08T21:21:35.329+0000 7fdd7fc797c0 -1 osd.0 0 log_to_monitors true 2024-05-08T21:21:35.533 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:35.532+0000 7f1f16b3b700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-08T21:21:35.534 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-08T21:21:35.532+0000 7f1f16b3b700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-08T21:21:38.612 INFO:tasks.ceph.osd.2.smithi003.stderr:2024-05-08T21:21:38.610+0000 7f11d8d01700 -1 osd.2 0 waiting for initial osdmap 2024-05-08T21:21:38.621 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-05-08T21:21:38.616+0000 7f6ec5c41700 -1 osd.5 0 waiting for initial osdmap 2024-05-08T21:21:38.622 INFO:tasks.ceph.osd.1.smithi003.stderr:2024-05-08T21:21:38.620+0000 7f5fdbb41700 -1 osd.1 0 waiting for initial osdmap 2024-05-08T21:21:38.630 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-08T21:21:38.627+0000 7fdd39fc0700 -1 osd.7 0 waiting for initial osdmap 2024-05-08T21:21:38.631 INFO:tasks.ceph.osd.1.smithi003.stderr:2024-05-08T21:21:38.630+0000 7f5fd3f1c700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T21:21:38.633 INFO:tasks.ceph.osd.2.smithi003.stderr:2024-05-08T21:21:38.632+0000 7f11d10dc700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T21:21:38.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-08T21:21:38.638+0000 7fb2d8d93700 -1 osd.4 0 waiting for initial osdmap 2024-05-08T21:21:38.641 INFO:tasks.ceph.osd.3.smithi003.stderr:2024-05-08T21:21:38.639+0000 7f7171fc0700 -1 osd.3 0 waiting for initial osdmap 2024-05-08T21:21:38.646 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-05-08T21:21:38.644+0000 7f6ebe01c700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T21:21:38.648 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-08T21:21:38.646+0000 7fdd3239b700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T21:21:38.653 INFO:tasks.ceph.osd.3.smithi003.stderr:2024-05-08T21:21:38.652+0000 7f716a39b700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T21:21:38.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-08T21:21:38.654+0000 7fb2d116e700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T21:21:38.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-08T21:21:38.663+0000 7fe622a82700 -1 osd.6 0 waiting for initial osdmap 2024-05-08T21:21:38.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-08T21:21:38.678+0000 7fe61ae5d700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T21:21:38.694 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-05-08T21:21:38.693+0000 7fdd7382e700 -1 osd.0 0 waiting for initial osdmap 2024-05-08T21:21:38.705 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-05-08T21:21:38.704+0000 7fdd6bc09700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T21:21:39.276 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-08T21:21:39.411 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.409+0000 7f1b141c4700 1 Processor -- start 2024-05-08T21:21:39.411 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.409+0000 7f1b141c4700 1 -- start start 2024-05-08T21:21:39.413 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.410+0000 7f1b141c4700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b0c15b870 0x7f1b0c0b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:39.414 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.410+0000 7f1b141c4700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1b0c0b1be0 0x7f1b0c0cc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:39.414 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.410+0000 7f1b141c4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1b0c0cc5b0 0x7f1b0c0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:39.414 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.410+0000 7f1b141c4700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b0c0cf670 con 0x7f1b0c0cc5b0 2024-05-08T21:21:39.414 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.410+0000 7f1b141c4700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b0c15bc50 con 0x7f1b0c0b1be0 2024-05-08T21:21:39.414 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.410+0000 7f1b141c4700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c0cf2b0 con 0x7f1b0c15b870 2024-05-08T21:21:39.414 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.410+0000 7f1b12761700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1b0c0cc5b0 0x7f1b0c0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:39.414 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.410+0000 7f1b12761700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1b0c0cc5b0 0x7f1b0c0cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35496/0 (socket says 172.21.15.3:35496) 2024-05-08T21:21:39.414 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.410+0000 7f1b12761700 1 -- 172.21.15.3:0/4122717337 learned_addr learned my addr 172.21.15.3:0/4122717337 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:39.414 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.411+0000 7f1b1175f700 1 --2- 172.21.15.3:0/4122717337 >> v2:172.21.15.186:3300/0 conn(0x7f1b0c0b1be0 0x7f1b0c0cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:39.414 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.411+0000 7f1b11f60700 1 --2- 172.21.15.3:0/4122717337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b0c15b870 0x7f1b0c0b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:39.414 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.411+0000 7f1b12761700 1 -- 172.21.15.3:0/4122717337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b0c15b870 msgr2=0x7f1b0c0b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:39.414 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.411+0000 7f1b12761700 1 --2- 172.21.15.3:0/4122717337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b0c15b870 0x7f1b0c0b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:39.414 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.411+0000 7f1b12761700 1 -- 172.21.15.3:0/4122717337 >> v2:172.21.15.186:3300/0 conn(0x7f1b0c0b1be0 msgr2=0x7f1b0c0cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:39.415 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.411+0000 7f1b12761700 1 --2- 172.21.15.3:0/4122717337 >> v2:172.21.15.186:3300/0 conn(0x7f1b0c0b1be0 0x7f1b0c0cc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:39.415 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.411+0000 7f1b12761700 1 -- 172.21.15.3:0/4122717337 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0c0b20d0 con 0x7f1b0c0cc5b0 2024-05-08T21:21:39.415 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.411+0000 7f1b12761700 1 --2- 172.21.15.3:0/4122717337 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b0c0cc5b0 0x7f1b0c0cfa90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c067560 tx=0x7f1b0400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f803d9afa9bf81 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:39.415 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.411+0000 7f1b10f5e700 1 -- 172.21.15.3:0/4122717337 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0400e070 con 0x7f1b0c0cc5b0 2024-05-08T21:21:39.415 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.411+0000 7f1b10f5e700 1 -- 172.21.15.3:0/4122717337 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b0400b9c0 con 0x7f1b0c0cc5b0 2024-05-08T21:21:39.415 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.411+0000 7f1b10f5e700 1 -- 172.21.15.3:0/4122717337 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0400bd70 con 0x7f1b0c0cc5b0 2024-05-08T21:21:39.415 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.412+0000 7f1b141c4700 1 -- 172.21.15.3:0/4122717337 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b0c0cc5b0 msgr2=0x7f1b0c0cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:39.415 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.412+0000 7f1b141c4700 1 --2- 172.21.15.3:0/4122717337 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b0c0cc5b0 0x7f1b0c0cfa90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c067560 tx=0x7f1b0400b330 comp rx=0 tx=0).stop 2024-05-08T21:21:39.415 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.412+0000 7f1b141c4700 1 -- 172.21.15.3:0/4122717337 shutdown_connections 2024-05-08T21:21:39.415 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.412+0000 7f1b141c4700 1 --2- 172.21.15.3:0/4122717337 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b0c0cc5b0 0x7f1b0c0cfa90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:39.415 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.412+0000 7f1b141c4700 1 --2- 172.21.15.3:0/4122717337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b0c15b870 0x7f1b0c0b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:39.415 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.412+0000 7f1b141c4700 1 --2- 172.21.15.3:0/4122717337 >> v2:172.21.15.186:3300/0 conn(0x7f1b0c0b1be0 0x7f1b0c0cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:39.415 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.412+0000 7f1b141c4700 1 -- 172.21.15.3:0/4122717337 >> 172.21.15.3:0/4122717337 conn(0x7f1b0c09a8a0 msgr2=0x7f1b0c08afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:39.415 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.412+0000 7f1b141c4700 1 -- 172.21.15.3:0/4122717337 shutdown_connections 2024-05-08T21:21:39.416 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.412+0000 7f1b141c4700 1 -- 172.21.15.3:0/4122717337 wait complete. 2024-05-08T21:21:39.418 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.413+0000 7f1b141c4700 1 Processor -- start 2024-05-08T21:21:39.418 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.413+0000 7f1b141c4700 1 -- start start 2024-05-08T21:21:39.418 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.414+0000 7f1b141c4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b0c0b1be0 0x7f1b0c152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:39.418 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.414+0000 7f1b141c4700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b0c0cc5b0 0x7f1b0c0afd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:39.418 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.414+0000 7f1b141c4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b0c15b870 0x7f1b0c0b0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:39.418 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.414+0000 7f1b141c4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c1530e0 con 0x7f1b0c0b1be0 2024-05-08T21:21:39.419 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.414+0000 7f1b141c4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c15bfc0 con 0x7f1b0c15b870 2024-05-08T21:21:39.419 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.414+0000 7f1b141c4700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c15c2a0 con 0x7f1b0c0cc5b0 2024-05-08T21:21:39.419 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.414+0000 7f1b11f60700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b0c0b1be0 0x7f1b0c152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:39.419 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.414+0000 7f1b11f60700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b0c0b1be0 0x7f1b0c152ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35502/0 (socket says 172.21.15.3:35502) 2024-05-08T21:21:39.419 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.414+0000 7f1b11f60700 1 -- 172.21.15.3:0/3598475031 learned_addr learned my addr 172.21.15.3:0/3598475031 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:39.419 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.414+0000 7f1b12761700 1 --2- 172.21.15.3:0/3598475031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b0c15b870 0x7f1b0c0b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:39.419 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.414+0000 7f1b1175f700 1 --2- 172.21.15.3:0/3598475031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b0c0cc5b0 0x7f1b0c0afd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:39.419 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.415+0000 7f1b11f60700 1 -- 172.21.15.3:0/3598475031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b0c0cc5b0 msgr2=0x7f1b0c0afd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:39.419 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.415+0000 7f1b11f60700 1 --2- 172.21.15.3:0/3598475031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b0c0cc5b0 0x7f1b0c0afd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:39.419 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.415+0000 7f1b11f60700 1 -- 172.21.15.3:0/3598475031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b0c15b870 msgr2=0x7f1b0c0b0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:39.419 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.415+0000 7f1b11f60700 1 --2- 172.21.15.3:0/3598475031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b0c15b870 0x7f1b0c0b0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:39.419 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.415+0000 7f1b11f60700 1 -- 172.21.15.3:0/3598475031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b04009b30 con 0x7f1b0c0b1be0 2024-05-08T21:21:39.419 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.415+0000 7f1b12761700 1 --2- 172.21.15.3:0/3598475031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b0c15b870 0x7f1b0c0b0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:21:39.420 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.415+0000 7f1b11f60700 1 --2- 172.21.15.3:0/3598475031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b0c0b1be0 0x7f1b0c152ba0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1b0800d800 tx=0x7f1b0800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:39.420 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.415+0000 7f1b027fc700 1 -- 172.21.15.3:0/3598475031 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b080125f0 con 0x7f1b0c0b1be0 2024-05-08T21:21:39.420 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.415+0000 7f1b027fc700 1 -- 172.21.15.3:0/3598475031 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b08012750 con 0x7f1b0c0b1be0 2024-05-08T21:21:39.420 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.415+0000 7f1b027fc700 1 -- 172.21.15.3:0/3598475031 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b08012b00 con 0x7f1b0c0b1be0 2024-05-08T21:21:39.420 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.415+0000 7f1b141c4700 1 -- 172.21.15.3:0/3598475031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b0c15c580 con 0x7f1b0c0b1be0 2024-05-08T21:21:39.420 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.415+0000 7f1b141c4700 1 -- 172.21.15.3:0/3598475031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b0c15cb40 con 0x7f1b0c0b1be0 2024-05-08T21:21:39.420 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.416+0000 7f1b141c4700 1 -- 172.21.15.3:0/3598475031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b0c04ec60 con 0x7f1b0c0b1be0 2024-05-08T21:21:39.421 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.417+0000 7f1b027fc700 1 -- 172.21.15.3:0/3598475031 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7f1b08012c60 con 0x7f1b0c0b1be0 2024-05-08T21:21:39.421 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.417+0000 7f1b027fc700 1 --2- 172.21.15.3:0/3598475031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1af006ac00 0x7f1af006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:39.421 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.417+0000 7f1b027fc700 1 -- 172.21.15.3:0/3598475031 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f1b08088830 con 0x7f1b0c0b1be0 2024-05-08T21:21:39.421 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.417+0000 7f1b1175f700 1 --2- 172.21.15.3:0/3598475031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1af006ac00 0x7f1af006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:39.421 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.418+0000 7f1b1175f700 1 --2- 172.21.15.3:0/3598475031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1af006ac00 0x7f1af006d0c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c0d1160 tx=0x7f1afc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:39.423 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.422+0000 7f1b027fc700 1 -- 172.21.15.3:0/3598475031 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f1b08054880 con 0x7f1b0c0b1be0 2024-05-08T21:21:39.643 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.642+0000 7f1b141c4700 1 -- 172.21.15.3:0/3598475031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b0c0b7030 con 0x7f1b0c0b1be0 2024-05-08T21:21:39.644 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.642+0000 7f1b027fc700 1 -- 172.21.15.3:0/3598475031 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f1b08057ea0 con 0x7f1b0c0b1be0 2024-05-08T21:21:39.644 INFO:teuthology.misc.health.smithi003.stdout: 2024-05-08T21:21:39.644 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":12,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:36.610478+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:21:39.647 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.645+0000 7f1b141c4700 1 -- 172.21.15.3:0/3598475031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1af006ac00 msgr2=0x7f1af006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:39.647 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.645+0000 7f1b141c4700 1 --2- 172.21.15.3:0/3598475031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1af006ac00 0x7f1af006d0c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c0d1160 tx=0x7f1afc008040 comp rx=0 tx=0).stop 2024-05-08T21:21:39.647 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.645+0000 7f1b141c4700 1 -- 172.21.15.3:0/3598475031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b0c0b1be0 msgr2=0x7f1b0c152ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:39.647 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.645+0000 7f1b141c4700 1 --2- 172.21.15.3:0/3598475031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b0c0b1be0 0x7f1b0c152ba0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1b0800d800 tx=0x7f1b0800dbc0 comp rx=0 tx=0).stop 2024-05-08T21:21:39.647 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.645+0000 7f1b141c4700 1 -- 172.21.15.3:0/3598475031 shutdown_connections 2024-05-08T21:21:39.647 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.645+0000 7f1b141c4700 1 --2- 172.21.15.3:0/3598475031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1af006ac00 0x7f1af006d0c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:39.647 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.645+0000 7f1b141c4700 1 --2- 172.21.15.3:0/3598475031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b0c15b870 0x7f1b0c0b0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:39.647 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.645+0000 7f1b141c4700 1 --2- 172.21.15.3:0/3598475031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b0c0b1be0 0x7f1b0c152ba0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:39.647 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.645+0000 7f1b141c4700 1 --2- 172.21.15.3:0/3598475031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b0c0cc5b0 0x7f1b0c0afd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:39.647 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.645+0000 7f1b141c4700 1 -- 172.21.15.3:0/3598475031 >> 172.21.15.3:0/3598475031 conn(0x7f1b0c09a8a0 msgr2=0x7f1b0c0cdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:39.648 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.646+0000 7f1b141c4700 1 -- 172.21.15.3:0/3598475031 shutdown_connections 2024-05-08T21:21:39.651 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:39.650+0000 7f1b141c4700 1 -- 172.21.15.3:0/3598475031 wait complete. 2024-05-08T21:21:39.666 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-08T21:21:41.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-08T21:21:41.732 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-08T21:21:46.473 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-08T21:21:46.607 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.605+0000 7fd65c446700 1 Processor -- start 2024-05-08T21:21:46.607 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.605+0000 7fd65c446700 1 -- start start 2024-05-08T21:21:46.611 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.606+0000 7fd65c446700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd65406c940 0x7fd6540cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:46.611 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.606+0000 7fd65c446700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6540ceb70 con 0x7fd65415d920 2024-05-08T21:21:46.611 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.606+0000 7fd65c446700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6540cdf60 con 0x7fd65406c4e0 2024-05-08T21:21:46.611 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.606+0000 7fd65c446700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6540ce240 con 0x7fd65406c940 2024-05-08T21:21:46.611 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd65a1e2700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd65415d920 0x7fd6540b09b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:41242/0 (socket says 172.21.15.3:41242) 2024-05-08T21:21:46.611 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd65a1e2700 1 -- 172.21.15.3:0/1873579395 learned_addr learned my addr 172.21.15.3:0/1873579395 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:46.611 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd65a9e3700 1 --2- 172.21.15.3:0/1873579395 >> v2:172.21.15.186:3300/0 conn(0x7fd65406c940 0x7fd6540cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:46.611 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 -- 172.21.15.3:0/1873579395 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188429552 0 0) 0x7fd6540ceb70 con 0x7fd65415d920 2024-05-08T21:21:46.611 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 -- 172.21.15.3:0/1873579395 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6400035b0 con 0x7fd65415d920 2024-05-08T21:21:46.612 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 -- 172.21.15.3:0/1873579395 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611255609 0 0) 0x7fd6540cdf60 con 0x7fd65406c4e0 2024-05-08T21:21:46.612 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 -- 172.21.15.3:0/1873579395 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6540ceb70 con 0x7fd65406c4e0 2024-05-08T21:21:46.612 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 -- 172.21.15.3:0/1873579395 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 252241076 0 0) 0x7fd6400035b0 con 0x7fd65415d920 2024-05-08T21:21:46.612 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 -- 172.21.15.3:0/1873579395 >> v2:172.21.15.186:3300/0 conn(0x7fd65406c940 msgr2=0x7fd6540cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:46.612 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 --2- 172.21.15.3:0/1873579395 >> v2:172.21.15.186:3300/0 conn(0x7fd65406c940 0x7fd6540cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:46.612 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 -- 172.21.15.3:0/1873579395 >> v1:172.21.15.186:6789/0 conn(0x7fd65406c4e0 legacy=0x7fd6540cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:46.612 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 -- 172.21.15.3:0/1873579395 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6540ce520 con 0x7fd65415d920 2024-05-08T21:21:46.612 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 -- 172.21.15.3:0/1873579395 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7fd650001670 con 0x7fd65415d920 2024-05-08T21:21:46.612 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 -- 172.21.15.3:0/1873579395 >> v1:172.21.15.3:6789/0 conn(0x7fd65415d920 legacy=0x7fd6540b09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:46.612 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 --2- 172.21.15.3:0/1873579395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd640003be0 0x7fd640003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:46.612 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 --2- 172.21.15.3:0/1873579395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd640004850 0x7fd640004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:46.612 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 --2- 172.21.15.3:0/1873579395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd640005210 0x7fd64000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:46.612 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 -- 172.21.15.3:0/1873579395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6540ce240 con 0x7fd640004850 2024-05-08T21:21:46.612 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 -- 172.21.15.3:0/1873579395 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64000c0e0 con 0x7fd640003be0 2024-05-08T21:21:46.613 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6591e0700 1 -- 172.21.15.3:0/1873579395 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64000c2a0 con 0x7fd640005210 2024-05-08T21:21:46.613 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd6599e1700 1 --2- 172.21.15.3:0/1873579395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd640004850 0x7fd640004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:46.613 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.607+0000 7fd65a1e2700 1 --2- 172.21.15.3:0/1873579395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd640003be0 0x7fd640003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:46.613 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.608+0000 7fd65a9e3700 1 --2- 172.21.15.3:0/1873579395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd640005210 0x7fd64000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:46.613 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.608+0000 7fd6599e1700 1 -- 172.21.15.3:0/1873579395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd640005210 msgr2=0x7fd64000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:46.613 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.608+0000 7fd6599e1700 1 --2- 172.21.15.3:0/1873579395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd640005210 0x7fd64000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:46.613 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.608+0000 7fd6599e1700 1 -- 172.21.15.3:0/1873579395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd640003be0 msgr2=0x7fd640003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:46.613 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.608+0000 7fd6599e1700 1 --2- 172.21.15.3:0/1873579395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd640003be0 0x7fd640003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:46.613 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.608+0000 7fd6599e1700 1 -- 172.21.15.3:0/1873579395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6400035b0 con 0x7fd640004850 2024-05-08T21:21:46.613 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.608+0000 7fd65a9e3700 1 -- 172.21.15.3:0/1873579395 reap_dead start 2024-05-08T21:21:46.613 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.608+0000 7fd6599e1700 1 --2- 172.21.15.3:0/1873579395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd640004850 0x7fd640004cd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd644001940 tx=0x7fd640005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec057d5908be8af6 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:46.613 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.608+0000 7fd6591e0700 1 -- 172.21.15.3:0/1873579395 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd644001430 con 0x7fd640004850 2024-05-08T21:21:46.613 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.608+0000 7fd6591e0700 1 -- 172.21.15.3:0/1873579395 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6440052d0 con 0x7fd640004850 2024-05-08T21:21:46.613 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.608+0000 7fd6591e0700 1 -- 172.21.15.3:0/1873579395 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd644001d10 con 0x7fd640004850 2024-05-08T21:21:46.614 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.609+0000 7fd65c446700 1 -- 172.21.15.3:0/1873579395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd640004850 msgr2=0x7fd640004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:46.614 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.609+0000 7fd65c446700 1 --2- 172.21.15.3:0/1873579395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd640004850 0x7fd640004cd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd644001940 tx=0x7fd640005880 comp rx=0 tx=0).stop 2024-05-08T21:21:46.614 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.609+0000 7fd65c446700 1 -- 172.21.15.3:0/1873579395 shutdown_connections 2024-05-08T21:21:46.614 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.609+0000 7fd65c446700 1 --2- 172.21.15.3:0/1873579395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd640004850 0x7fd640004cd0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:46.614 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.609+0000 7fd65c446700 1 -- 172.21.15.3:0/1873579395 >> 172.21.15.3:0/1873579395 conn(0x7fd65409a8a0 msgr2=0x7fd65408afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:46.614 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.609+0000 7fd65c446700 1 -- 172.21.15.3:0/1873579395 shutdown_connections 2024-05-08T21:21:46.614 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.609+0000 7fd65c446700 1 -- 172.21.15.3:0/1873579395 wait complete. 2024-05-08T21:21:46.617 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.610+0000 7fd65c446700 1 Processor -- start 2024-05-08T21:21:46.617 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.610+0000 7fd65c446700 1 -- start start 2024-05-08T21:21:46.617 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.611+0000 7fd65c446700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd640004850 0x7fd654156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:46.617 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.611+0000 7fd65c446700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6540afd60 0x7fd6541f3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:46.617 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.611+0000 7fd65c446700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6541575e0 0x7fd6541f4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:46.617 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.611+0000 7fd65c446700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd654156cc0 con 0x7fd6540afd60 2024-05-08T21:21:46.617 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.611+0000 7fd65c446700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd654157150 con 0x7fd6541575e0 2024-05-08T21:21:46.617 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.611+0000 7fd65c446700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd654156e20 con 0x7fd640004850 2024-05-08T21:21:46.617 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.611+0000 7fd6599e1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6540afd60 0x7fd6541f3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:46.618 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.611+0000 7fd6599e1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6540afd60 0x7fd6541f3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38322/0 (socket says 172.21.15.3:38322) 2024-05-08T21:21:46.618 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.611+0000 7fd6599e1700 1 -- 172.21.15.3:0/1896912031 learned_addr learned my addr 172.21.15.3:0/1896912031 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:46.618 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.611+0000 7fd65a1e2700 1 --2- 172.21.15.3:0/1896912031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd640004850 0x7fd654156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:46.618 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.611+0000 7fd65a9e3700 1 --2- 172.21.15.3:0/1896912031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6541575e0 0x7fd6541f4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:46.618 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.612+0000 7fd6599e1700 1 -- 172.21.15.3:0/1896912031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd640004850 msgr2=0x7fd654156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:46.618 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.612+0000 7fd6599e1700 1 --2- 172.21.15.3:0/1896912031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd640004850 0x7fd654156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:46.618 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.612+0000 7fd6599e1700 1 -- 172.21.15.3:0/1896912031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6541575e0 msgr2=0x7fd6541f4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:46.618 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.612+0000 7fd6599e1700 1 --2- 172.21.15.3:0/1896912031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6541575e0 0x7fd6541f4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:46.618 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.612+0000 7fd6599e1700 1 -- 172.21.15.3:0/1896912031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6541f3f30 con 0x7fd6540afd60 2024-05-08T21:21:46.618 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.612+0000 7fd6599e1700 1 --2- 172.21.15.3:0/1896912031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6540afd60 0x7fd6541f3670 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd64400c3c0 tx=0x7fd6440012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:46.618 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.612+0000 7fd64affd700 1 -- 172.21.15.3:0/1896912031 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd644001d10 con 0x7fd6540afd60 2024-05-08T21:21:46.618 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.612+0000 7fd64affd700 1 -- 172.21.15.3:0/1896912031 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6440050d0 con 0x7fd6540afd60 2024-05-08T21:21:46.618 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.612+0000 7fd65c446700 1 -- 172.21.15.3:0/1896912031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6541f4210 con 0x7fd6540afd60 2024-05-08T21:21:46.618 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.612+0000 7fd64affd700 1 -- 172.21.15.3:0/1896912031 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd644007280 con 0x7fd6540afd60 2024-05-08T21:21:46.619 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.612+0000 7fd65c446700 1 -- 172.21.15.3:0/1896912031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6541f3bb0 con 0x7fd6540afd60 2024-05-08T21:21:46.619 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.613+0000 7fd65c446700 1 -- 172.21.15.3:0/1896912031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd65404ec60 con 0x7fd6540afd60 2024-05-08T21:21:46.619 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.613+0000 7fd64affd700 1 -- 172.21.15.3:0/1896912031 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd6440073e0 con 0x7fd6540afd60 2024-05-08T21:21:46.619 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.614+0000 7fd64affd700 1 --2- 172.21.15.3:0/1896912031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd63806ac90 0x7fd63806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:46.619 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.614+0000 7fd64affd700 1 -- 172.21.15.3:0/1896912031 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd644089550 con 0x7fd6540afd60 2024-05-08T21:21:46.619 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.614+0000 7fd65a1e2700 1 --2- 172.21.15.3:0/1896912031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd63806ac90 0x7fd63806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:46.619 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.615+0000 7fd65a1e2700 1 --2- 172.21.15.3:0/1896912031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd63806ac90 0x7fd63806d150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd650001050 tx=0x7fd650001b00 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:46.620 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.619+0000 7fd64affd700 1 -- 172.21.15.3:0/1896912031 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd644054ce0 con 0x7fd6540afd60 2024-05-08T21:21:46.840 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.839+0000 7fd65c446700 1 -- 172.21.15.3:0/1896912031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd654157290 con 0x7fd6540afd60 2024-05-08T21:21:46.842 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.840+0000 7fd64affd700 1 -- 172.21.15.3:0/1896912031 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12334 (secure 0 0 0) 0x7fd644021090 con 0x7fd6540afd60 2024-05-08T21:21:46.842 INFO:teuthology.misc.health.smithi003.stdout: 2024-05-08T21:21:46.842 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":16,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:42.880367+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:21:46.844 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.842+0000 7fd65c446700 1 -- 172.21.15.3:0/1896912031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd63806ac90 msgr2=0x7fd63806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:46.844 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.842+0000 7fd65c446700 1 --2- 172.21.15.3:0/1896912031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd63806ac90 0x7fd63806d150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd650001050 tx=0x7fd650001b00 comp rx=0 tx=0).stop 2024-05-08T21:21:46.844 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.842+0000 7fd65c446700 1 -- 172.21.15.3:0/1896912031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6540afd60 msgr2=0x7fd6541f3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:46.845 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.842+0000 7fd65c446700 1 --2- 172.21.15.3:0/1896912031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6540afd60 0x7fd6541f3670 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd64400c3c0 tx=0x7fd6440012d0 comp rx=0 tx=0).stop 2024-05-08T21:21:46.845 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.842+0000 7fd65c446700 1 -- 172.21.15.3:0/1896912031 shutdown_connections 2024-05-08T21:21:46.845 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.842+0000 7fd65c446700 1 --2- 172.21.15.3:0/1896912031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd640004850 0x7fd654156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:46.845 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.842+0000 7fd65c446700 1 --2- 172.21.15.3:0/1896912031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd63806ac90 0x7fd63806d150 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:46.845 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.842+0000 7fd65c446700 1 --2- 172.21.15.3:0/1896912031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6541575e0 0x7fd6541f4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:46.845 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.842+0000 7fd65c446700 1 --2- 172.21.15.3:0/1896912031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6540afd60 0x7fd6541f3670 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:46.845 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.842+0000 7fd65c446700 1 -- 172.21.15.3:0/1896912031 >> 172.21.15.3:0/1896912031 conn(0x7fd65409a8a0 msgr2=0x7fd6540d0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:46.845 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.842+0000 7fd65c446700 1 -- 172.21.15.3:0/1896912031 shutdown_connections 2024-05-08T21:21:46.845 INFO:teuthology.misc.health.smithi003.stderr:2024-05-08T21:21:46.842+0000 7fd65c446700 1 -- 172.21.15.3:0/1896912031 wait complete. 2024-05-08T21:21:46.857 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-05-08T21:21:46.857 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-08T21:21:46.857 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-08T21:21:46.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-08T21:21:46.860 INFO:tasks.ceph.mds.a:Started 2024-05-08T21:21:46.860 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-08T21:21:46.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-08T21:21:46.862 INFO:tasks.ceph.mds.c:Started 2024-05-08T21:21:46.862 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-08T21:21:46.862 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-08T21:21:46.865 INFO:tasks.ceph.mds.b:Started 2024-05-08T21:21:46.865 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-08T21:21:46.865 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-08T21:21:46.867 INFO:tasks.ceph.mds.d:Started 2024-05-08T21:21:46.867 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-05-08T21:21:46.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:21:46.950 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2024-05-08T21:21:46.951 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2024-05-08T21:21:46.955 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2024-05-08T21:21:46.955 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2024-05-08T21:21:47.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.010+0000 7fa172a6e700 1 Processor -- start 2024-05-08T21:21:47.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.010+0000 7fa172a6e700 1 -- start start 2024-05-08T21:21:47.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.011+0000 7fa172a6e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa16c06ef30 0x7fa16c06fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:47.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.011+0000 7fa172a6e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa16c15e310 0x7fa16c15e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:47.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.011+0000 7fa172a6e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16c0cef60 con 0x7fa16c15dcc0 2024-05-08T21:21:47.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.011+0000 7fa172a6e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa16c0b2de0 con 0x7fa16c06ef30 2024-05-08T21:21:47.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.011+0000 7fa172a6e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa16c15ecc0 con 0x7fa16c15e310 2024-05-08T21:21:47.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.011+0000 7fa16bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa16c06ef30 0x7fa16c06fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:47.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.011+0000 7fa16bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa16c06ef30 0x7fa16c06fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38334/0 (socket says 172.21.15.3:38334) 2024-05-08T21:21:47.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.011+0000 7fa16bfff700 1 -- 172.21.15.3:0/3097657791 learned_addr learned my addr 172.21.15.3:0/3097657791 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:47.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.011+0000 7fa17100b700 1 --2- 172.21.15.3:0/3097657791 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa16c15e310 0x7fa16c15e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:47.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.011+0000 7fa16bfff700 1 -- 172.21.15.3:0/3097657791 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa16c15e310 msgr2=0x7fa16c15e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:47.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.011+0000 7fa16bfff700 1 --2- 172.21.15.3:0/3097657791 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa16c15e310 0x7fa16c15e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.011+0000 7fa16bfff700 1 -- 172.21.15.3:0/3097657791 >> v1:172.21.15.186:6789/0 conn(0x7fa16c15dcc0 legacy=0x7fa16c0b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.011+0000 7fa16b7fe700 1 -- 172.21.15.3:0/3097657791 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938432657 0 0) 0x7fa16c0cef60 con 0x7fa16c15dcc0 2024-05-08T21:21:47.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.011+0000 7fa16bfff700 1 -- 172.21.15.3:0/3097657791 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16c15eee0 con 0x7fa16c06ef30 2024-05-08T21:21:47.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.012+0000 7fa16bfff700 1 --2- 172.21.15.3:0/3097657791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa16c06ef30 0x7fa16c06fa10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa158009920 tx=0x7fa158009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e12c4d85c27d7232 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:47.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.012+0000 7fa16b7fe700 1 -- 172.21.15.3:0/3097657791 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa158011830 con 0x7fa16c06ef30 2024-05-08T21:21:47.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.012+0000 7fa16b7fe700 1 -- 172.21.15.3:0/3097657791 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa158011990 con 0x7fa16c06ef30 2024-05-08T21:21:47.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.012+0000 7fa16b7fe700 1 -- 172.21.15.3:0/3097657791 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa158011d20 con 0x7fa16c06ef30 2024-05-08T21:21:47.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.012+0000 7fa172a6e700 1 -- 172.21.15.3:0/3097657791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa16c06ef30 msgr2=0x7fa16c06fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.012+0000 7fa172a6e700 1 --2- 172.21.15.3:0/3097657791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa16c06ef30 0x7fa16c06fa10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa158009920 tx=0x7fa158009c30 comp rx=0 tx=0).stop 2024-05-08T21:21:47.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.012+0000 7fa172a6e700 1 -- 172.21.15.3:0/3097657791 shutdown_connections 2024-05-08T21:21:47.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.012+0000 7fa172a6e700 1 --2- 172.21.15.3:0/3097657791 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa16c15e310 0x7fa16c15e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.012+0000 7fa172a6e700 1 --2- 172.21.15.3:0/3097657791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa16c06ef30 0x7fa16c06fa10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.012+0000 7fa172a6e700 1 -- 172.21.15.3:0/3097657791 >> 172.21.15.3:0/3097657791 conn(0x7fa16c099a50 msgr2=0x7fa16c0b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:47.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.012+0000 7fa172a6e700 1 -- 172.21.15.3:0/3097657791 shutdown_connections 2024-05-08T21:21:47.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.012+0000 7fa172a6e700 1 -- 172.21.15.3:0/3097657791 wait complete. 2024-05-08T21:21:47.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.014+0000 7fa172a6e700 1 Processor -- start 2024-05-08T21:21:47.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.014+0000 7fa172a6e700 1 -- start start 2024-05-08T21:21:47.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa172a6e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa16c06ef30 0x7fa16c1f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:47.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa172a6e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa16c15dcc0 0x7fa16c0c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:47.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa172a6e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa16c15e310 0x7fa16c0c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:47.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa172a6e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa16c1f7470 con 0x7fa16c06ef30 2024-05-08T21:21:47.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa172a6e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa16c0b0130 con 0x7fa16c15e310 2024-05-08T21:21:47.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa172a6e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa16c0c6c80 con 0x7fa16c15dcc0 2024-05-08T21:21:47.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa17080a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa16c06ef30 0x7fa16c1f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:47.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa17080a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa16c06ef30 0x7fa16c1f65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38348/0 (socket says 172.21.15.3:38348) 2024-05-08T21:21:47.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa17080a700 1 -- 172.21.15.3:0/1995597806 learned_addr learned my addr 172.21.15.3:0/1995597806 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:47.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa16bfff700 1 --2- 172.21.15.3:0/1995597806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa16c15dcc0 0x7fa16c0c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:47.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa17100b700 1 --2- 172.21.15.3:0/1995597806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa16c15e310 0x7fa16c0c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:47.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa17080a700 1 -- 172.21.15.3:0/1995597806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa16c15dcc0 msgr2=0x7fa16c0c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa17080a700 1 --2- 172.21.15.3:0/1995597806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa16c15dcc0 0x7fa16c0c9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa17080a700 1 -- 172.21.15.3:0/1995597806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa16c15e310 msgr2=0x7fa16c0c6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa17080a700 1 --2- 172.21.15.3:0/1995597806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa16c15e310 0x7fa16c0c6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa17080a700 1 -- 172.21.15.3:0/1995597806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa158009550 con 0x7fa16c06ef30 2024-05-08T21:21:47.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.015+0000 7fa17080a700 1 --2- 172.21.15.3:0/1995597806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa16c06ef30 0x7fa16c1f65d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa16000c580 tx=0x7fa16000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:47.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.016+0000 7fa1697fa700 1 -- 172.21.15.3:0/1995597806 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1600054e0 con 0x7fa16c06ef30 2024-05-08T21:21:47.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.016+0000 7fa1697fa700 1 -- 172.21.15.3:0/1995597806 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa160003680 con 0x7fa16c06ef30 2024-05-08T21:21:47.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.016+0000 7fa172a6e700 1 -- 172.21.15.3:0/1995597806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa16c1f6bd0 con 0x7fa16c06ef30 2024-05-08T21:21:47.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.016+0000 7fa1697fa700 1 -- 172.21.15.3:0/1995597806 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa160003a30 con 0x7fa16c06ef30 2024-05-08T21:21:47.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.016+0000 7fa172a6e700 1 -- 172.21.15.3:0/1995597806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa16c1f7190 con 0x7fa16c06ef30 2024-05-08T21:21:47.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.017+0000 7fa172a6e700 1 -- 172.21.15.3:0/1995597806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa16c068000 con 0x7fa16c06ef30 2024-05-08T21:21:47.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.018+0000 7fa1697fa700 1 -- 172.21.15.3:0/1995597806 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa160003b90 con 0x7fa16c06ef30 2024-05-08T21:21:47.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.018+0000 7fa1697fa700 1 --2- 172.21.15.3:0/1995597806 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa15006ac90 0x7fa15006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:47.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.018+0000 7fa1697fa700 1 -- 172.21.15.3:0/1995597806 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa160088600 con 0x7fa16c06ef30 2024-05-08T21:21:47.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.018+0000 7fa16bfff700 1 --2- 172.21.15.3:0/1995597806 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa15006ac90 0x7fa15006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:47.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.019+0000 7fa16bfff700 1 --2- 172.21.15.3:0/1995597806 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa15006ac90 0x7fa15006d150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa158000c00 tx=0x7fa158008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:47.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.022+0000 7fa1697fa700 1 -- 172.21.15.3:0/1995597806 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fa160053ca0 con 0x7fa16c06ef30 2024-05-08T21:21:47.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.243+0000 7fa172a6e700 1 -- 172.21.15.3:0/1995597806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa16c0cd920 con 0x7fa16c06ef30 2024-05-08T21:21:47.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.244+0000 7fa1697fa700 1 -- 172.21.15.3:0/1995597806 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12334 (secure 0 0 0) 0x7fa160017070 con 0x7fa16c06ef30 2024-05-08T21:21:47.245 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:21:47.246 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":16,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:42.880367+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:21:47.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.246+0000 7fa172a6e700 1 -- 172.21.15.3:0/1995597806 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa15006ac90 msgr2=0x7fa15006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.246+0000 7fa172a6e700 1 --2- 172.21.15.3:0/1995597806 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa15006ac90 0x7fa15006d150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa158000c00 tx=0x7fa158008030 comp rx=0 tx=0).stop 2024-05-08T21:21:47.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.246+0000 7fa172a6e700 1 -- 172.21.15.3:0/1995597806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa16c06ef30 msgr2=0x7fa16c1f65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.246+0000 7fa172a6e700 1 --2- 172.21.15.3:0/1995597806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa16c06ef30 0x7fa16c1f65d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa16000c580 tx=0x7fa16000c890 comp rx=0 tx=0).stop 2024-05-08T21:21:47.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.246+0000 7fa172a6e700 1 -- 172.21.15.3:0/1995597806 shutdown_connections 2024-05-08T21:21:47.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.246+0000 7fa172a6e700 1 --2- 172.21.15.3:0/1995597806 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa15006ac90 0x7fa15006d150 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.246+0000 7fa172a6e700 1 --2- 172.21.15.3:0/1995597806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa16c15e310 0x7fa16c0c6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.246+0000 7fa172a6e700 1 --2- 172.21.15.3:0/1995597806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa16c06ef30 0x7fa16c1f65d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.246+0000 7fa172a6e700 1 --2- 172.21.15.3:0/1995597806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa16c15dcc0 0x7fa16c0c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.246+0000 7fa172a6e700 1 -- 172.21.15.3:0/1995597806 >> 172.21.15.3:0/1995597806 conn(0x7fa16c099a50 msgr2=0x7fa16c0b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:47.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.246+0000 7fa172a6e700 1 -- 172.21.15.3:0/1995597806 shutdown_connections 2024-05-08T21:21:47.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.246+0000 7fa172a6e700 1 -- 172.21.15.3:0/1995597806 wait complete. 2024-05-08T21:21:47.261 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:21:47.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:21:47.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.399+0000 7f66e3ff7700 1 Processor -- start 2024-05-08T21:21:47.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.399+0000 7f66e3ff7700 1 -- start start 2024-05-08T21:21:47.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.400+0000 7f66e3ff7700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f66dc0cff30 0x7f66dc0b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:47.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.400+0000 7f66e3ff7700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66dc0d04b0 con 0x7f66dc09a160 2024-05-08T21:21:47.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.400+0000 7f66e3ff7700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66dc0b0910 con 0x7f66dc15b710 2024-05-08T21:21:47.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.400+0000 7f66e3ff7700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66dc0b0bf0 con 0x7f66dc0cff30 2024-05-08T21:21:47.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.401+0000 7f66e1592700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f66dc09a160 0x7f66dc0b16d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:41268/0 (socket says 172.21.15.3:41268) 2024-05-08T21:21:47.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.401+0000 7f66e1592700 1 -- 172.21.15.3:0/2686072192 learned_addr learned my addr 172.21.15.3:0/2686072192 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:47.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.401+0000 7f66e2594700 1 --2- 172.21.15.3:0/2686072192 >> v2:172.21.15.186:3300/0 conn(0x7f66dc0cff30 0x7f66dc0b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:47.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.401+0000 7f66e0d91700 1 -- 172.21.15.3:0/2686072192 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928893800 0 0) 0x7f66dc0d04b0 con 0x7f66dc09a160 2024-05-08T21:21:47.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.401+0000 7f66e0d91700 1 -- 172.21.15.3:0/2686072192 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66c80035b0 con 0x7f66dc09a160 2024-05-08T21:21:47.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.401+0000 7f66e0d91700 1 -- 172.21.15.3:0/2686072192 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653161324 0 0) 0x7f66dc0b0910 con 0x7f66dc15b710 2024-05-08T21:21:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.401+0000 7f66e0d91700 1 -- 172.21.15.3:0/2686072192 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66dc0d04b0 con 0x7f66dc15b710 2024-05-08T21:21:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.401+0000 7f66e2594700 1 -- 172.21.15.3:0/2686072192 >> v1:172.21.15.186:6789/0 conn(0x7f66dc15b710 legacy=0x7f66dc099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.401+0000 7f66e2594700 1 -- 172.21.15.3:0/2686072192 >> v1:172.21.15.3:6789/0 conn(0x7f66dc09a160 legacy=0x7f66dc0b16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.401+0000 7f66e0d91700 1 -- 172.21.15.3:0/2686072192 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2392024972 0 0) 0x7f66c80035b0 con 0x7f66dc09a160 2024-05-08T21:21:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.401+0000 7f66e2594700 1 -- 172.21.15.3:0/2686072192 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc0b0ed0 con 0x7f66dc0cff30 2024-05-08T21:21:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.401+0000 7f66e2594700 1 --2- 172.21.15.3:0/2686072192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66dc0cff30 0x7f66dc0b1de0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f66d4009840 tx=0x7f66d4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf0f4e048f5bed9e server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.401+0000 7f66e0d91700 1 -- 172.21.15.3:0/2686072192 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d4011830 con 0x7f66dc0cff30 2024-05-08T21:21:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.401+0000 7f66e0d91700 1 -- 172.21.15.3:0/2686072192 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d4011990 con 0x7f66dc0cff30 2024-05-08T21:21:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.401+0000 7f66e0d91700 1 -- 172.21.15.3:0/2686072192 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d4011d20 con 0x7f66dc0cff30 2024-05-08T21:21:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.402+0000 7f66e3ff7700 1 -- 172.21.15.3:0/2686072192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66dc0cff30 msgr2=0x7f66dc0b1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.402+0000 7f66e3ff7700 1 --2- 172.21.15.3:0/2686072192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66dc0cff30 0x7f66dc0b1de0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f66d4009840 tx=0x7f66d4009b50 comp rx=0 tx=0).stop 2024-05-08T21:21:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.402+0000 7f66e3ff7700 1 -- 172.21.15.3:0/2686072192 shutdown_connections 2024-05-08T21:21:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.402+0000 7f66e3ff7700 1 --2- 172.21.15.3:0/2686072192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66dc0cff30 0x7f66dc0b1de0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.402+0000 7f66e3ff7700 1 -- 172.21.15.3:0/2686072192 >> 172.21.15.3:0/2686072192 conn(0x7f66dc0b2f60 msgr2=0x7f66dc0b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.402+0000 7f66e3ff7700 1 -- 172.21.15.3:0/2686072192 shutdown_connections 2024-05-08T21:21:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.402+0000 7f66e3ff7700 1 -- 172.21.15.3:0/2686072192 wait complete. 2024-05-08T21:21:47.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.403+0000 7f66e3ff7700 1 Processor -- start 2024-05-08T21:21:47.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.403+0000 7f66e3ff7700 1 -- start start 2024-05-08T21:21:47.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.404+0000 7f66e3ff7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66dc09a160 0x7f66dc158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:47.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.404+0000 7f66e3ff7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66dc0cff30 0x7f66dc0c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:47.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.404+0000 7f66e3ff7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66dc15b710 0x7f66dc157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.404+0000 7f66e3ff7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc06d570 con 0x7f66dc09a160 2024-05-08T21:21:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.404+0000 7f66e3ff7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc06c8f0 con 0x7f66dc0cff30 2024-05-08T21:21:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.404+0000 7f66e3ff7700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc06cbd0 con 0x7f66dc15b710 2024-05-08T21:21:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.404+0000 7f66e1d93700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66dc09a160 0x7f66dc158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.404+0000 7f66e1d93700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66dc09a160 0x7f66dc158df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38360/0 (socket says 172.21.15.3:38360) 2024-05-08T21:21:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.404+0000 7f66e1d93700 1 -- 172.21.15.3:0/4066941933 learned_addr learned my addr 172.21.15.3:0/4066941933 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.405+0000 7f66e1592700 1 --2- 172.21.15.3:0/4066941933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66dc0cff30 0x7f66dc0c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.405+0000 7f66e2594700 1 --2- 172.21.15.3:0/4066941933 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66dc15b710 0x7f66dc157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.405+0000 7f66e1d93700 1 -- 172.21.15.3:0/4066941933 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66dc15b710 msgr2=0x7f66dc157460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.405+0000 7f66e1d93700 1 --2- 172.21.15.3:0/4066941933 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66dc15b710 0x7f66dc157460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.405+0000 7f66e1d93700 1 -- 172.21.15.3:0/4066941933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66dc0cff30 msgr2=0x7f66dc0c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.405+0000 7f66e1d93700 1 --2- 172.21.15.3:0/4066941933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66dc0cff30 0x7f66dc0c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.405+0000 7f66e1d93700 1 -- 172.21.15.3:0/4066941933 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d40094a0 con 0x7f66dc09a160 2024-05-08T21:21:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.405+0000 7f66e1d93700 1 --2- 172.21.15.3:0/4066941933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66dc09a160 0x7f66dc158df0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f66d800cc50 tx=0x7f66d800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.405+0000 7f66d27fc700 1 -- 172.21.15.3:0/4066941933 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d800ade0 con 0x7f66dc09a160 2024-05-08T21:21:47.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.405+0000 7f66d27fc700 1 -- 172.21.15.3:0/4066941933 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d800ce50 con 0x7f66dc09a160 2024-05-08T21:21:47.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.405+0000 7f66d27fc700 1 -- 172.21.15.3:0/4066941933 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8003c40 con 0x7f66dc09a160 2024-05-08T21:21:47.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.405+0000 7f66e3ff7700 1 -- 172.21.15.3:0/4066941933 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66dc06ceb0 con 0x7f66dc09a160 2024-05-08T21:21:47.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.405+0000 7f66e3ff7700 1 -- 172.21.15.3:0/4066941933 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66dc06d990 con 0x7f66dc09a160 2024-05-08T21:21:47.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.406+0000 7f66e3ff7700 1 -- 172.21.15.3:0/4066941933 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66dc06d7e0 con 0x7f66dc09a160 2024-05-08T21:21:47.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.407+0000 7f66d27fc700 1 -- 172.21.15.3:0/4066941933 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f66d80050d0 con 0x7f66dc09a160 2024-05-08T21:21:47.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.407+0000 7f66d27fc700 1 --2- 172.21.15.3:0/4066941933 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f66c006ac90 0x7f66c006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:47.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.407+0000 7f66d27fc700 1 -- 172.21.15.3:0/4066941933 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f66d8089270 con 0x7f66dc09a160 2024-05-08T21:21:47.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.408+0000 7f66e1592700 1 --2- 172.21.15.3:0/4066941933 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f66c006ac90 0x7f66c006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:47.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.408+0000 7f66e1592700 1 --2- 172.21.15.3:0/4066941933 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f66c006ac90 0x7f66c006d150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f66dc159740 tx=0x7f66cc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:47.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.412+0000 7f66d27fc700 1 -- 172.21.15.3:0/4066941933 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f66d8054680 con 0x7f66dc09a160 2024-05-08T21:21:47.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.630+0000 7f66e3ff7700 1 -- 172.21.15.3:0/4066941933 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f66dc074bc0 con 0x7f66dc09a160 2024-05-08T21:21:47.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.630+0000 7f66d27fc700 1 -- 172.21.15.3:0/4066941933 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f66d8057ca0 con 0x7f66dc09a160 2024-05-08T21:21:47.632 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:21:47.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.633+0000 7f66e3ff7700 1 -- 172.21.15.3:0/4066941933 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f66c006ac90 msgr2=0x7f66c006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.633+0000 7f66e3ff7700 1 --2- 172.21.15.3:0/4066941933 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f66c006ac90 0x7f66c006d150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f66dc159740 tx=0x7f66cc009040 comp rx=0 tx=0).stop 2024-05-08T21:21:47.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.633+0000 7f66e3ff7700 1 -- 172.21.15.3:0/4066941933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66dc09a160 msgr2=0x7f66dc158df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.633+0000 7f66e3ff7700 1 --2- 172.21.15.3:0/4066941933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66dc09a160 0x7f66dc158df0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f66d800cc50 tx=0x7f66d800a340 comp rx=0 tx=0).stop 2024-05-08T21:21:47.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.633+0000 7f66e3ff7700 1 -- 172.21.15.3:0/4066941933 shutdown_connections 2024-05-08T21:21:47.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.633+0000 7f66e3ff7700 1 --2- 172.21.15.3:0/4066941933 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66dc15b710 0x7f66dc157460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.633+0000 7f66e3ff7700 1 --2- 172.21.15.3:0/4066941933 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f66c006ac90 0x7f66c006d150 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.633+0000 7f66e3ff7700 1 --2- 172.21.15.3:0/4066941933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66dc09a160 0x7f66dc158df0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.633+0000 7f66e3ff7700 1 --2- 172.21.15.3:0/4066941933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66dc0cff30 0x7f66dc0c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.633+0000 7f66e3ff7700 1 -- 172.21.15.3:0/4066941933 >> 172.21.15.3:0/4066941933 conn(0x7f66dc0b2f60 msgr2=0x7f66dc06f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:47.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.633+0000 7f66e3ff7700 1 -- 172.21.15.3:0/4066941933 shutdown_connections 2024-05-08T21:21:47.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.633+0000 7f66e3ff7700 1 -- 172.21.15.3:0/4066941933 wait complete. 2024-05-08T21:21:47.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:21:47.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.782+0000 7f86d2693700 1 Processor -- start 2024-05-08T21:21:47.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.782+0000 7f86d2693700 1 -- start start 2024-05-08T21:21:47.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.783+0000 7f86d2693700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f86cc06a960 0x7f86cc15d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:47.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.783+0000 7f86d2693700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86cc0cd980 con 0x7f86cc0d0eb0 2024-05-08T21:21:47.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.783+0000 7f86d2693700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86cc0ccdb0 con 0x7f86cc0cf840 2024-05-08T21:21:47.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.783+0000 7f86d2693700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86cc0cd050 con 0x7f86cc06a960 2024-05-08T21:21:47.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.783+0000 7f86d0c30700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f86cc0d0eb0 0x7f86cc161620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:41270/0 (socket says 172.21.15.3:41270) 2024-05-08T21:21:47.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.783+0000 7f86d0c30700 1 -- 172.21.15.3:0/1593840753 learned_addr learned my addr 172.21.15.3:0/1593840753 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:47.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.783+0000 7f86cb7fe700 1 --2- 172.21.15.3:0/1593840753 >> v2:172.21.15.186:3300/0 conn(0x7f86cc06a960 0x7f86cc15d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:47.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.783+0000 7f86caffd700 1 -- 172.21.15.3:0/1593840753 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995931747 0 0) 0x7f86cc0cd980 con 0x7f86cc0d0eb0 2024-05-08T21:21:47.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.784+0000 7f86caffd700 1 -- 172.21.15.3:0/1593840753 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b40035b0 con 0x7f86cc0d0eb0 2024-05-08T21:21:47.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.784+0000 7f86caffd700 1 -- 172.21.15.3:0/1593840753 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2286801235 0 0) 0x7f86b40035b0 con 0x7f86cc0d0eb0 2024-05-08T21:21:47.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.784+0000 7f86cb7fe700 1 -- 172.21.15.3:0/1593840753 >> v1:172.21.15.186:6789/0 conn(0x7f86cc0cf840 legacy=0x7f86cc0af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.784+0000 7f86cb7fe700 1 -- 172.21.15.3:0/1593840753 >> v1:172.21.15.3:6789/0 conn(0x7f86cc0d0eb0 legacy=0x7f86cc161620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.784+0000 7f86cb7fe700 1 -- 172.21.15.3:0/1593840753 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86cc0cd330 con 0x7f86cc06a960 2024-05-08T21:21:47.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.784+0000 7f86cb7fe700 1 --2- 172.21.15.3:0/1593840753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86cc06a960 0x7f86cc15d4f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009840 tx=0x7f86c0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd9e5ccb5945c504 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:47.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.784+0000 7f86caffd700 1 -- 172.21.15.3:0/1593840753 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c40019f0 con 0x7f86cc06a960 2024-05-08T21:21:47.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.784+0000 7f86caffd700 1 -- 172.21.15.3:0/1593840753 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c0004030 con 0x7f86cc06a960 2024-05-08T21:21:47.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.784+0000 7f86caffd700 1 -- 172.21.15.3:0/1593840753 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c00043c0 con 0x7f86cc06a960 2024-05-08T21:21:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.785+0000 7f86d2693700 1 -- 172.21.15.3:0/1593840753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86cc06a960 msgr2=0x7f86cc15d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.785+0000 7f86d2693700 1 --2- 172.21.15.3:0/1593840753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86cc06a960 0x7f86cc15d4f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009840 tx=0x7f86c0009b50 comp rx=0 tx=0).stop 2024-05-08T21:21:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.785+0000 7f86d2693700 1 -- 172.21.15.3:0/1593840753 shutdown_connections 2024-05-08T21:21:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.785+0000 7f86d2693700 1 --2- 172.21.15.3:0/1593840753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86cc06a960 0x7f86cc15d4f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.785+0000 7f86d2693700 1 -- 172.21.15.3:0/1593840753 >> 172.21.15.3:0/1593840753 conn(0x7f86cc0d0320 msgr2=0x7f86cc0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.785+0000 7f86d2693700 1 -- 172.21.15.3:0/1593840753 shutdown_connections 2024-05-08T21:21:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.785+0000 7f86d2693700 1 -- 172.21.15.3:0/1593840753 wait complete. 2024-05-08T21:21:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.786+0000 7f86d2693700 1 Processor -- start 2024-05-08T21:21:47.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.786+0000 7f86d2693700 1 -- start start 2024-05-08T21:21:47.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.787+0000 7f86d2693700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86cc06a960 0x7f86cc1525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:47.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.787+0000 7f86d2693700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86cc0cf840 0x7f86cc151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:47.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.787+0000 7f86d2693700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86cc0d0eb0 0x7f86cc1562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:47.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.787+0000 7f86d2693700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc1523f0 con 0x7f86cc0d0eb0 2024-05-08T21:21:47.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.787+0000 7f86d2693700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc0b4340 con 0x7f86cc06a960 2024-05-08T21:21:47.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.787+0000 7f86d2693700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc0b4620 con 0x7f86cc0cf840 2024-05-08T21:21:47.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.787+0000 7f86d0c30700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86cc0d0eb0 0x7f86cc1562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:47.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.787+0000 7f86d0c30700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86cc0d0eb0 0x7f86cc1562a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38374/0 (socket says 172.21.15.3:38374) 2024-05-08T21:21:47.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.787+0000 7f86d0c30700 1 -- 172.21.15.3:0/256978334 learned_addr learned my addr 172.21.15.3:0/256978334 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:47.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.788+0000 7f86cbfff700 1 --2- 172.21.15.3:0/256978334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86cc06a960 0x7f86cc1525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:47.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.788+0000 7f86cb7fe700 1 --2- 172.21.15.3:0/256978334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86cc0cf840 0x7f86cc151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:47.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.788+0000 7f86d0c30700 1 -- 172.21.15.3:0/256978334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86cc0cf840 msgr2=0x7f86cc151eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.788+0000 7f86d0c30700 1 --2- 172.21.15.3:0/256978334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86cc0cf840 0x7f86cc151eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.788+0000 7f86d0c30700 1 -- 172.21.15.3:0/256978334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86cc06a960 msgr2=0x7f86cc1525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:47.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.788+0000 7f86d0c30700 1 --2- 172.21.15.3:0/256978334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86cc06a960 0x7f86cc1525c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:47.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.788+0000 7f86d0c30700 1 -- 172.21.15.3:0/256978334 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c00094a0 con 0x7f86cc0d0eb0 2024-05-08T21:21:47.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.788+0000 7f86d0c30700 1 --2- 172.21.15.3:0/256978334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86cc0d0eb0 0x7f86cc1562a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f86c400cbe0 tx=0x7f86c400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:47.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.788+0000 7f86c8ff9700 1 -- 172.21.15.3:0/256978334 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c400ab80 con 0x7f86cc0d0eb0 2024-05-08T21:21:47.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.788+0000 7f86c8ff9700 1 -- 172.21.15.3:0/256978334 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c400ad00 con 0x7f86cc0d0eb0 2024-05-08T21:21:47.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.788+0000 7f86c8ff9700 1 -- 172.21.15.3:0/256978334 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c4007310 con 0x7f86cc0d0eb0 2024-05-08T21:21:47.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.788+0000 7f86d2693700 1 -- 172.21.15.3:0/256978334 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86cc0b4900 con 0x7f86cc0d0eb0 2024-05-08T21:21:47.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.788+0000 7f86d2693700 1 -- 172.21.15.3:0/256978334 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86cc0b4ec0 con 0x7f86cc0d0eb0 2024-05-08T21:21:47.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.789+0000 7f86d2693700 1 -- 172.21.15.3:0/256978334 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86cc0b50b0 con 0x7f86cc0d0eb0 2024-05-08T21:21:47.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.790+0000 7f86c8ff9700 1 -- 172.21.15.3:0/256978334 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f86c4007470 con 0x7f86cc0d0eb0 2024-05-08T21:21:47.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.790+0000 7f86c8ff9700 1 --2- 172.21.15.3:0/256978334 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f86b806ac90 0x7f86b806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:47.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.790+0000 7f86c8ff9700 1 -- 172.21.15.3:0/256978334 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f86c4088bb0 con 0x7f86cc0d0eb0 2024-05-08T21:21:47.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.790+0000 7f86cbfff700 1 --2- 172.21.15.3:0/256978334 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f86b806ac90 0x7f86b806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:47.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.791+0000 7f86cbfff700 1 --2- 172.21.15.3:0/256978334 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f86b806ac90 0x7f86b806d150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0b41b0 tx=0x7f86bc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:47.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:47.795+0000 7f86c8ff9700 1 -- 172.21.15.3:0/256978334 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f86c4054060 con 0x7f86cc0d0eb0 2024-05-08T21:21:48.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.064+0000 7f86d2693700 1 -- 172.21.15.3:0/256978334 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86cc068000 con 0x7f86cc0d0eb0 2024-05-08T21:21:48.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.065+0000 7f86c8ff9700 1 -- 172.21.15.3:0/256978334 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3524 (secure 0 0 0) 0x7f86c4018090 con 0x7f86cc0d0eb0 2024-05-08T21:21:48.067 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:21:48.067 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/2839854293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2839854293},{"type":"v1","addr":"172.21.15.3:6834","nonce":2839854293}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1878326567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":1878326567},{"type":"v1","addr":"172.21.15.3:6835","nonce":1878326567}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/3624849830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3624849830},{"type":"v1","addr":"172.21.15.186:6836","nonce":3624849830}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1462886680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1462886680},{"type":"v1","addr":"172.21.15.186:6837","nonce":1462886680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-05-08T21:21:48.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.067+0000 7f86d2693700 1 -- 172.21.15.3:0/256978334 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f86b806ac90 msgr2=0x7f86b806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.067+0000 7f86d2693700 1 --2- 172.21.15.3:0/256978334 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f86b806ac90 0x7f86b806d150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0b41b0 tx=0x7f86bc009040 comp rx=0 tx=0).stop 2024-05-08T21:21:48.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.067+0000 7f86d2693700 1 -- 172.21.15.3:0/256978334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86cc0d0eb0 msgr2=0x7f86cc1562a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.067+0000 7f86d2693700 1 --2- 172.21.15.3:0/256978334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86cc0d0eb0 0x7f86cc1562a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f86c400cbe0 tx=0x7f86c400a060 comp rx=0 tx=0).stop 2024-05-08T21:21:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.067+0000 7f86d2693700 1 -- 172.21.15.3:0/256978334 shutdown_connections 2024-05-08T21:21:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.067+0000 7f86d2693700 1 --2- 172.21.15.3:0/256978334 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f86b806ac90 0x7f86b806d150 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.067+0000 7f86d2693700 1 --2- 172.21.15.3:0/256978334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86cc0d0eb0 0x7f86cc1562a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.067+0000 7f86d2693700 1 --2- 172.21.15.3:0/256978334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86cc06a960 0x7f86cc1525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.067+0000 7f86d2693700 1 --2- 172.21.15.3:0/256978334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86cc0cf840 0x7f86cc151eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.067+0000 7f86d2693700 1 -- 172.21.15.3:0/256978334 >> 172.21.15.3:0/256978334 conn(0x7f86cc0d0320 msgr2=0x7f86cc0b12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.067+0000 7f86d2693700 1 -- 172.21.15.3:0/256978334 shutdown_connections 2024-05-08T21:21:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.067+0000 7f86d2693700 1 -- 172.21.15.3:0/256978334 wait complete. 2024-05-08T21:21:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 3 2024-05-08T21:21:48.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:21:48.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.217+0000 7fcf9d997700 1 Processor -- start 2024-05-08T21:21:48.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.217+0000 7fcf9d997700 1 -- start start 2024-05-08T21:21:48.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.218+0000 7fcf9d997700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf9806b810 0x7fcf980cf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.218+0000 7fcf9d997700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf980d0880 con 0x7fcf9815d9c0 2024-05-08T21:21:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.218+0000 7fcf9d997700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf980d0b60 con 0x7fcf9806bc70 2024-05-08T21:21:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.218+0000 7fcf9d997700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980ce5b0 con 0x7fcf9806b810 2024-05-08T21:21:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.218+0000 7fcf96ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fcf9815d9c0 0x7fcf980cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:41276/0 (socket says 172.21.15.3:41276) 2024-05-08T21:21:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.218+0000 7fcf96ffd700 1 -- 172.21.15.3:0/2454845639 learned_addr learned my addr 172.21.15.3:0/2454845639 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.218+0000 7fcf967fc700 1 --2- 172.21.15.3:0/2454845639 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf9806b810 0x7fcf980cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.218+0000 7fcf95ffb700 1 -- 172.21.15.3:0/2454845639 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610624867 0 0) 0x7fcf980d0880 con 0x7fcf9815d9c0 2024-05-08T21:21:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.219+0000 7fcf95ffb700 1 -- 172.21.15.3:0/2454845639 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf800035b0 con 0x7fcf9815d9c0 2024-05-08T21:21:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.219+0000 7fcf95ffb700 1 -- 172.21.15.3:0/2454845639 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436179246 0 0) 0x7fcf980d0b60 con 0x7fcf9806bc70 2024-05-08T21:21:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.219+0000 7fcf95ffb700 1 -- 172.21.15.3:0/2454845639 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf980d0880 con 0x7fcf9806bc70 2024-05-08T21:21:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.219+0000 7fcf95ffb700 1 -- 172.21.15.3:0/2454845639 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3297517796 0 0) 0x7fcf800035b0 con 0x7fcf9815d9c0 2024-05-08T21:21:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.219+0000 7fcf967fc700 1 -- 172.21.15.3:0/2454845639 >> v1:172.21.15.186:6789/0 conn(0x7fcf9806bc70 legacy=0x7fcf980d16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.219+0000 7fcf967fc700 1 -- 172.21.15.3:0/2454845639 >> v1:172.21.15.3:6789/0 conn(0x7fcf9815d9c0 legacy=0x7fcf980cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.219+0000 7fcf967fc700 1 -- 172.21.15.3:0/2454845639 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf980ce890 con 0x7fcf9806b810 2024-05-08T21:21:48.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.219+0000 7fcf967fc700 1 --2- 172.21.15.3:0/2454845639 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf9806b810 0x7fcf980cf090 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009850 tx=0x7fcf8c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9a2db803f69a460 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:48.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.219+0000 7fcf95ffb700 1 -- 172.21.15.3:0/2454845639 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf88001670 con 0x7fcf9806b810 2024-05-08T21:21:48.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.219+0000 7fcf95ffb700 1 -- 172.21.15.3:0/2454845639 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c005c30 con 0x7fcf9806b810 2024-05-08T21:21:48.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.219+0000 7fcf95ffb700 1 -- 172.21.15.3:0/2454845639 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c004030 con 0x7fcf9806b810 2024-05-08T21:21:48.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.220+0000 7fcf9d997700 1 -- 172.21.15.3:0/2454845639 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf9806b810 msgr2=0x7fcf980cf090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.220+0000 7fcf9d997700 1 --2- 172.21.15.3:0/2454845639 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf9806b810 0x7fcf980cf090 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009850 tx=0x7fcf8c009b60 comp rx=0 tx=0).stop 2024-05-08T21:21:48.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.220+0000 7fcf9d997700 1 -- 172.21.15.3:0/2454845639 shutdown_connections 2024-05-08T21:21:48.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.220+0000 7fcf9d997700 1 --2- 172.21.15.3:0/2454845639 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf9806b810 0x7fcf980cf090 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.220+0000 7fcf9d997700 1 -- 172.21.15.3:0/2454845639 >> 172.21.15.3:0/2454845639 conn(0x7fcf9808afe0 msgr2=0x7fcf980d3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:48.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.220+0000 7fcf9d997700 1 -- 172.21.15.3:0/2454845639 shutdown_connections 2024-05-08T21:21:48.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.220+0000 7fcf9d997700 1 -- 172.21.15.3:0/2454845639 wait complete. 2024-05-08T21:21:48.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.221+0000 7fcf9d997700 1 Processor -- start 2024-05-08T21:21:48.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.221+0000 7fcf9d997700 1 -- start start 2024-05-08T21:21:48.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.222+0000 7fcf9d997700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf9806b810 0x7fcf981594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:48.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.222+0000 7fcf9d997700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9806bc70 0x7fcf9806c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:48.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.222+0000 7fcf9d997700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf9815d9c0 0x7fcf9806c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:48.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.222+0000 7fcf9d997700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9815e620 con 0x7fcf9806bc70 2024-05-08T21:21:48.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.222+0000 7fcf9d997700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98159a00 con 0x7fcf9815d9c0 2024-05-08T21:21:48.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.222+0000 7fcf9d997700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98160d10 con 0x7fcf9806b810 2024-05-08T21:21:48.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.222+0000 7fcf967fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9806bc70 0x7fcf9806c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:48.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.222+0000 7fcf967fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9806bc70 0x7fcf9806c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38380/0 (socket says 172.21.15.3:38380) 2024-05-08T21:21:48.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.222+0000 7fcf967fc700 1 -- 172.21.15.3:0/1942613569 learned_addr learned my addr 172.21.15.3:0/1942613569 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:48.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.222+0000 7fcf96ffd700 1 --2- 172.21.15.3:0/1942613569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf9806b810 0x7fcf981594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:48.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.222+0000 7fcf977fe700 1 --2- 172.21.15.3:0/1942613569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf9815d9c0 0x7fcf9806c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:48.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.223+0000 7fcf967fc700 1 -- 172.21.15.3:0/1942613569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf9806b810 msgr2=0x7fcf981594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.223+0000 7fcf967fc700 1 --2- 172.21.15.3:0/1942613569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf9806b810 0x7fcf981594c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.223+0000 7fcf967fc700 1 -- 172.21.15.3:0/1942613569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf9815d9c0 msgr2=0x7fcf9806c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.223+0000 7fcf967fc700 1 --2- 172.21.15.3:0/1942613569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf9815d9c0 0x7fcf9806c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.223+0000 7fcf967fc700 1 -- 172.21.15.3:0/1942613569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8c009430 con 0x7fcf9806bc70 2024-05-08T21:21:48.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.223+0000 7fcf967fc700 1 --2- 172.21.15.3:0/1942613569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9806bc70 0x7fcf9806c140 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009eb0 tx=0x7fcf8c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:48.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.223+0000 7fcf7ffff700 1 -- 172.21.15.3:0/1942613569 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c003b30 con 0x7fcf9806bc70 2024-05-08T21:21:48.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.223+0000 7fcf7ffff700 1 -- 172.21.15.3:0/1942613569 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c003c90 con 0x7fcf9806bc70 2024-05-08T21:21:48.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.223+0000 7fcf9d997700 1 -- 172.21.15.3:0/1942613569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf981f7a30 con 0x7fcf9806bc70 2024-05-08T21:21:48.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.223+0000 7fcf7ffff700 1 -- 172.21.15.3:0/1942613569 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c01d430 con 0x7fcf9806bc70 2024-05-08T21:21:48.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.223+0000 7fcf9d997700 1 -- 172.21.15.3:0/1942613569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9815e240 con 0x7fcf9806bc70 2024-05-08T21:21:48.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.224+0000 7fcf9d997700 1 -- 172.21.15.3:0/1942613569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf9815e760 con 0x7fcf9806bc70 2024-05-08T21:21:48.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.224+0000 7fcf7ffff700 1 -- 172.21.15.3:0/1942613569 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcf8c01d590 con 0x7fcf9806bc70 2024-05-08T21:21:48.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.225+0000 7fcf7ffff700 1 --2- 172.21.15.3:0/1942613569 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcf8406ac90 0x7fcf8406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:48.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.225+0000 7fcf7ffff700 1 -- 172.21.15.3:0/1942613569 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcf8c08a2e0 con 0x7fcf9806bc70 2024-05-08T21:21:48.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.225+0000 7fcf96ffd700 1 --2- 172.21.15.3:0/1942613569 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcf8406ac90 0x7fcf8406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:48.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.226+0000 7fcf96ffd700 1 --2- 172.21.15.3:0/1942613569 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcf8406ac90 0x7fcf8406d150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcf88001380 tx=0x7fcf88001590 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:48.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.230+0000 7fcf7ffff700 1 -- 172.21.15.3:0/1942613569 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fcf9815e760 con 0x7fcf9806bc70 2024-05-08T21:21:48.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.450+0000 7fcf9d997700 1 -- 172.21.15.3:0/1942613569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf9815e760 con 0x7fcf9806bc70 2024-05-08T21:21:48.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.451+0000 7fcf7ffff700 1 -- 172.21.15.3:0/1942613569 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12334 (secure 0 0 0) 0x7fcf9815e760 con 0x7fcf9806bc70 2024-05-08T21:21:48.453 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:21:48.453 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":16,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:42.880367+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:21:48.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.453+0000 7fcf9d997700 1 -- 172.21.15.3:0/1942613569 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcf8406ac90 msgr2=0x7fcf8406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.453+0000 7fcf9d997700 1 --2- 172.21.15.3:0/1942613569 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcf8406ac90 0x7fcf8406d150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcf88001380 tx=0x7fcf88001590 comp rx=0 tx=0).stop 2024-05-08T21:21:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.453+0000 7fcf9d997700 1 -- 172.21.15.3:0/1942613569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9806bc70 msgr2=0x7fcf9806c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.453+0000 7fcf9d997700 1 --2- 172.21.15.3:0/1942613569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9806bc70 0x7fcf9806c140 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009eb0 tx=0x7fcf8c004750 comp rx=0 tx=0).stop 2024-05-08T21:21:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.453+0000 7fcf9d997700 1 -- 172.21.15.3:0/1942613569 shutdown_connections 2024-05-08T21:21:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.453+0000 7fcf9d997700 1 --2- 172.21.15.3:0/1942613569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf9806b810 0x7fcf981594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.453+0000 7fcf9d997700 1 --2- 172.21.15.3:0/1942613569 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcf8406ac90 0x7fcf8406d150 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.453+0000 7fcf9d997700 1 --2- 172.21.15.3:0/1942613569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf9815d9c0 0x7fcf9806c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.453+0000 7fcf9d997700 1 --2- 172.21.15.3:0/1942613569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9806bc70 0x7fcf9806c140 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.453+0000 7fcf9d997700 1 -- 172.21.15.3:0/1942613569 >> 172.21.15.3:0/1942613569 conn(0x7fcf9808afe0 msgr2=0x7fcf980b2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.453+0000 7fcf9d997700 1 -- 172.21.15.3:0/1942613569 shutdown_connections 2024-05-08T21:21:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.453+0000 7fcf9d997700 1 -- 172.21.15.3:0/1942613569 wait complete. 2024-05-08T21:21:48.469 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:21:48.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:21:48.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.606+0000 7fc750538700 1 Processor -- start 2024-05-08T21:21:48.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.607+0000 7fc750538700 1 -- start start 2024-05-08T21:21:48.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc750538700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc7480b5a80 0x7fc7480b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:48.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc750538700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc7480cf440 0x7fc74815b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:48.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc750538700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7480cf8a0 con 0x7fc74815b7d0 2024-05-08T21:21:48.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc750538700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7480b3a70 con 0x7fc7480cf440 2024-05-08T21:21:48.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc750538700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc74815ab60 con 0x7fc7480b5a80 2024-05-08T21:21:48.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc74dad3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc7480cf440 0x7fc74815b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:48.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc74dad3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc7480cf440 0x7fc74815b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38390/0 (socket says 172.21.15.3:38390) 2024-05-08T21:21:48.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc74dad3700 1 -- 172.21.15.3:0/2342655858 learned_addr learned my addr 172.21.15.3:0/2342655858 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:48.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc74e2d4700 1 --2- 172.21.15.3:0/2342655858 >> v2:172.21.15.186:3300/0 conn(0x7fc7480b5a80 0x7fc7480b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:48.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc74dad3700 1 -- 172.21.15.3:0/2342655858 >> v2:172.21.15.186:3300/0 conn(0x7fc7480b5a80 msgr2=0x7fc7480b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:48.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc74dad3700 1 --2- 172.21.15.3:0/2342655858 >> v2:172.21.15.186:3300/0 conn(0x7fc7480b5a80 0x7fc7480b2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc74dad3700 1 -- 172.21.15.3:0/2342655858 >> v1:172.21.15.186:6789/0 conn(0x7fc74815b7d0 legacy=0x7fc74815bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:48.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc74dad3700 1 -- 172.21.15.3:0/2342655858 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc748155050 con 0x7fc7480cf440 2024-05-08T21:21:48.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc74e2d4700 1 --2- 172.21.15.3:0/2342655858 >> v2:172.21.15.186:3300/0 conn(0x7fc7480b5a80 0x7fc7480b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:21:48.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc74dad3700 1 --2- 172.21.15.3:0/2342655858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480cf440 0x7fc74815b290 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc748066910 tx=0x7fc738009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e68f4296748001a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:48.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc74d2d2700 1 -- 172.21.15.3:0/2342655858 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc738004230 con 0x7fc7480cf440 2024-05-08T21:21:48.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc74d2d2700 1 -- 172.21.15.3:0/2342655858 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc738004390 con 0x7fc7480cf440 2024-05-08T21:21:48.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.608+0000 7fc74d2d2700 1 -- 172.21.15.3:0/2342655858 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7380036a0 con 0x7fc7480cf440 2024-05-08T21:21:48.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.609+0000 7fc750538700 1 -- 172.21.15.3:0/2342655858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480cf440 msgr2=0x7fc74815b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.609+0000 7fc750538700 1 --2- 172.21.15.3:0/2342655858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480cf440 0x7fc74815b290 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc748066910 tx=0x7fc738009bd0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.609+0000 7fc750538700 1 -- 172.21.15.3:0/2342655858 shutdown_connections 2024-05-08T21:21:48.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.609+0000 7fc750538700 1 --2- 172.21.15.3:0/2342655858 >> v2:172.21.15.186:3300/0 conn(0x7fc7480b5a80 0x7fc7480b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.609+0000 7fc750538700 1 --2- 172.21.15.3:0/2342655858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480cf440 0x7fc74815b290 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.609+0000 7fc750538700 1 -- 172.21.15.3:0/2342655858 >> 172.21.15.3:0/2342655858 conn(0x7fc7480713c0 msgr2=0x7fc7480717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:48.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.609+0000 7fc750538700 1 -- 172.21.15.3:0/2342655858 shutdown_connections 2024-05-08T21:21:48.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.609+0000 7fc750538700 1 -- 172.21.15.3:0/2342655858 wait complete. 2024-05-08T21:21:48.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.611+0000 7fc750538700 1 Processor -- start 2024-05-08T21:21:48.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.611+0000 7fc750538700 1 -- start start 2024-05-08T21:21:48.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.611+0000 7fc750538700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480b5a80 0x7fc74814ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:48.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.611+0000 7fc750538700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7480cf440 0x7fc7481469e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:48.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.611+0000 7fc750538700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc74815b7d0 0x7fc7480b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:48.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.611+0000 7fc750538700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74814f380 con 0x7fc7480b5a80 2024-05-08T21:21:48.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.611+0000 7fc750538700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc748159200 con 0x7fc7480cf440 2024-05-08T21:21:48.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.611+0000 7fc750538700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7481585f0 con 0x7fc74815b7d0 2024-05-08T21:21:48.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.611+0000 7fc74e2d4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480b5a80 0x7fc74814ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:48.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.612+0000 7fc74e2d4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480b5a80 0x7fc74814ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38398/0 (socket says 172.21.15.3:38398) 2024-05-08T21:21:48.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.612+0000 7fc74e2d4700 1 -- 172.21.15.3:0/90392272 learned_addr learned my addr 172.21.15.3:0/90392272 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:48.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.612+0000 7fc74dad3700 1 --2- 172.21.15.3:0/90392272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7480cf440 0x7fc7481469e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:48.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.612+0000 7fc74ead5700 1 --2- 172.21.15.3:0/90392272 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc74815b7d0 0x7fc7480b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:48.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.612+0000 7fc74e2d4700 1 -- 172.21.15.3:0/90392272 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc74815b7d0 msgr2=0x7fc7480b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.612+0000 7fc74e2d4700 1 --2- 172.21.15.3:0/90392272 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc74815b7d0 0x7fc7480b02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.612+0000 7fc74e2d4700 1 -- 172.21.15.3:0/90392272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7480cf440 msgr2=0x7fc7481469e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.612+0000 7fc74e2d4700 1 --2- 172.21.15.3:0/90392272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7480cf440 0x7fc7481469e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.612+0000 7fc74e2d4700 1 -- 172.21.15.3:0/90392272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc738009550 con 0x7fc7480b5a80 2024-05-08T21:21:48.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.612+0000 7fc74dad3700 1 --2- 172.21.15.3:0/90392272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7480cf440 0x7fc7481469e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:21:48.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.612+0000 7fc74e2d4700 1 --2- 172.21.15.3:0/90392272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480b5a80 0x7fc74814ee40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc74400c7f0 tx=0x7fc74400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:48.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.612+0000 7fc73effd700 1 -- 172.21.15.3:0/90392272 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc744011800 con 0x7fc7480b5a80 2024-05-08T21:21:48.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.612+0000 7fc73effd700 1 -- 172.21.15.3:0/90392272 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc744011960 con 0x7fc7480b5a80 2024-05-08T21:21:48.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.612+0000 7fc750538700 1 -- 172.21.15.3:0/90392272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7481588d0 con 0x7fc7480b5a80 2024-05-08T21:21:48.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.612+0000 7fc73effd700 1 -- 172.21.15.3:0/90392272 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc744011d10 con 0x7fc7480b5a80 2024-05-08T21:21:48.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.612+0000 7fc750538700 1 -- 172.21.15.3:0/90392272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc748158e90 con 0x7fc7480b5a80 2024-05-08T21:21:48.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.613+0000 7fc750538700 1 -- 172.21.15.3:0/90392272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7480b3750 con 0x7fc7480b5a80 2024-05-08T21:21:48.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.614+0000 7fc73effd700 1 -- 172.21.15.3:0/90392272 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc744018360 con 0x7fc7480b5a80 2024-05-08T21:21:48.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.614+0000 7fc73effd700 1 --2- 172.21.15.3:0/90392272 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc72c06ac90 0x7fc72c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:48.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.614+0000 7fc73effd700 1 -- 172.21.15.3:0/90392272 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc744089080 con 0x7fc7480b5a80 2024-05-08T21:21:48.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.615+0000 7fc74dad3700 1 --2- 172.21.15.3:0/90392272 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc72c06ac90 0x7fc72c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:48.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.615+0000 7fc74dad3700 1 --2- 172.21.15.3:0/90392272 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc72c06ac90 0x7fc72c06d150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc738009130 tx=0x7fc738008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:48.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.619+0000 7fc73effd700 1 -- 172.21.15.3:0/90392272 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc744054520 con 0x7fc7480b5a80 2024-05-08T21:21:48.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.835+0000 7fc750538700 1 -- 172.21.15.3:0/90392272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc748074bc0 con 0x7fc7480b5a80 2024-05-08T21:21:48.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.835+0000 7fc73effd700 1 -- 172.21.15.3:0/90392272 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc74401f0a0 con 0x7fc7480b5a80 2024-05-08T21:21:48.837 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:21:48.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.837+0000 7fc750538700 1 -- 172.21.15.3:0/90392272 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc72c06ac90 msgr2=0x7fc72c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.837+0000 7fc750538700 1 --2- 172.21.15.3:0/90392272 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc72c06ac90 0x7fc72c06d150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc738009130 tx=0x7fc738008030 comp rx=0 tx=0).stop 2024-05-08T21:21:48.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.837+0000 7fc750538700 1 -- 172.21.15.3:0/90392272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480b5a80 msgr2=0x7fc74814ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.837+0000 7fc750538700 1 --2- 172.21.15.3:0/90392272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480b5a80 0x7fc74814ee40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc74400c7f0 tx=0x7fc74400cbb0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.837+0000 7fc750538700 1 -- 172.21.15.3:0/90392272 shutdown_connections 2024-05-08T21:21:48.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.837+0000 7fc750538700 1 --2- 172.21.15.3:0/90392272 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc74815b7d0 0x7fc7480b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.837+0000 7fc750538700 1 --2- 172.21.15.3:0/90392272 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc72c06ac90 0x7fc72c06d150 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.837+0000 7fc750538700 1 --2- 172.21.15.3:0/90392272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480b5a80 0x7fc74814ee40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.837+0000 7fc750538700 1 --2- 172.21.15.3:0/90392272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7480cf440 0x7fc7481469e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.837+0000 7fc750538700 1 -- 172.21.15.3:0/90392272 >> 172.21.15.3:0/90392272 conn(0x7fc7480713c0 msgr2=0x7fc7480717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:48.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.838+0000 7fc750538700 1 -- 172.21.15.3:0/90392272 shutdown_connections 2024-05-08T21:21:48.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.838+0000 7fc750538700 1 -- 172.21.15.3:0/90392272 wait complete. 2024-05-08T21:21:48.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-08T21:21:48.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.989+0000 7fad752a7700 1 Processor -- start 2024-05-08T21:21:48.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.989+0000 7fad752a7700 1 -- start start 2024-05-08T21:21:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.989+0000 7fad752a7700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fad7015d780 0x7fad700b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.989+0000 7fad752a7700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad700b4430 con 0x7fad700d22f0 2024-05-08T21:21:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.989+0000 7fad752a7700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad700b3820 con 0x7fad700d13e0 2024-05-08T21:21:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.989+0000 7fad752a7700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad700d45c0 con 0x7fad7015d780 2024-05-08T21:21:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.990+0000 7fad6effd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fad7015d780 0x7fad700b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.990+0000 7fad6f7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fad700d22f0 0x7fad700ce3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:41286/0 (socket says 172.21.15.3:41286) 2024-05-08T21:21:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.990+0000 7fad6f7fe700 1 -- 172.21.15.3:0/2632736879 learned_addr learned my addr 172.21.15.3:0/2632736879 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.990+0000 7fad6dffb700 1 -- 172.21.15.3:0/2632736879 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853987908 0 0) 0x7fad700b4430 con 0x7fad700d22f0 2024-05-08T21:21:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.990+0000 7fad6dffb700 1 -- 172.21.15.3:0/2632736879 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad580035b0 con 0x7fad700d22f0 2024-05-08T21:21:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.990+0000 7fad6effd700 1 -- 172.21.15.3:0/2632736879 >> v1:172.21.15.186:6789/0 conn(0x7fad700d13e0 legacy=0x7fad700d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.990+0000 7fad6effd700 1 -- 172.21.15.3:0/2632736879 >> v1:172.21.15.3:6789/0 conn(0x7fad700d22f0 legacy=0x7fad700ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.990+0000 7fad6effd700 1 -- 172.21.15.3:0/2632736879 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad700b3b00 con 0x7fad7015d780 2024-05-08T21:21:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.990+0000 7fad6effd700 1 --2- 172.21.15.3:0/2632736879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad7015d780 0x7fad700b48c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fad60009820 tx=0x7fad60009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8d2719d7cfc30fb server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.990+0000 7fad6dffb700 1 -- 172.21.15.3:0/2632736879 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad60005e30 con 0x7fad7015d780 2024-05-08T21:21:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.990+0000 7fad6dffb700 1 -- 172.21.15.3:0/2632736879 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad60006030 con 0x7fad7015d780 2024-05-08T21:21:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.990+0000 7fad6dffb700 1 -- 172.21.15.3:0/2632736879 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad600063c0 con 0x7fad7015d780 2024-05-08T21:21:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.991+0000 7fad752a7700 1 -- 172.21.15.3:0/2632736879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad7015d780 msgr2=0x7fad700b48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.991+0000 7fad752a7700 1 --2- 172.21.15.3:0/2632736879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad7015d780 0x7fad700b48c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fad60009820 tx=0x7fad60009b30 comp rx=0 tx=0).stop 2024-05-08T21:21:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.991+0000 7fad752a7700 1 -- 172.21.15.3:0/2632736879 shutdown_connections 2024-05-08T21:21:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.991+0000 7fad752a7700 1 --2- 172.21.15.3:0/2632736879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad7015d780 0x7fad700b48c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.991+0000 7fad752a7700 1 -- 172.21.15.3:0/2632736879 >> 172.21.15.3:0/2632736879 conn(0x7fad7008afe0 msgr2=0x7fad700d3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.991+0000 7fad752a7700 1 -- 172.21.15.3:0/2632736879 shutdown_connections 2024-05-08T21:21:48.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.991+0000 7fad752a7700 1 -- 172.21.15.3:0/2632736879 wait complete. 2024-05-08T21:21:48.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.992+0000 7fad752a7700 1 Processor -- start 2024-05-08T21:21:48.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.992+0000 7fad752a7700 1 -- start start 2024-05-08T21:21:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.993+0000 7fad752a7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad700d13e0 0x7fad700c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.993+0000 7fad752a7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad700d22f0 0x7fad700c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.993+0000 7fad752a7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad7015d780 0x7fad700c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.993+0000 7fad752a7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad70156770 con 0x7fad700d13e0 2024-05-08T21:21:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.993+0000 7fad752a7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad700c75d0 con 0x7fad7015d780 2024-05-08T21:21:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.993+0000 7fad752a7700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad700c78b0 con 0x7fad700d22f0 2024-05-08T21:21:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.993+0000 7fad6effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad700d13e0 0x7fad700c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.993+0000 7fad6effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad700d13e0 0x7fad700c6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38422/0 (socket says 172.21.15.3:38422) 2024-05-08T21:21:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.993+0000 7fad6effd700 1 -- 172.21.15.3:0/1431724409 learned_addr learned my addr 172.21.15.3:0/1431724409 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.994+0000 7fad6f7fe700 1 --2- 172.21.15.3:0/1431724409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad7015d780 0x7fad700c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.994+0000 7fad6e7fc700 1 --2- 172.21.15.3:0/1431724409 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad700d22f0 0x7fad700c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.994+0000 7fad6effd700 1 -- 172.21.15.3:0/1431724409 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad700d22f0 msgr2=0x7fad700c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.994+0000 7fad6effd700 1 --2- 172.21.15.3:0/1431724409 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad700d22f0 0x7fad700c8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.994+0000 7fad6effd700 1 -- 172.21.15.3:0/1431724409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad7015d780 msgr2=0x7fad700c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:48.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.994+0000 7fad6effd700 1 --2- 172.21.15.3:0/1431724409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad7015d780 0x7fad700c96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:48.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.994+0000 7fad6effd700 1 -- 172.21.15.3:0/1431724409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad60009430 con 0x7fad700d13e0 2024-05-08T21:21:48.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.994+0000 7fad6effd700 1 --2- 172.21.15.3:0/1431724409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad700d13e0 0x7fad700c6ff0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fad60000c00 tx=0x7fad60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:48.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.994+0000 7fad577fe700 1 -- 172.21.15.3:0/1431724409 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad600065b0 con 0x7fad700d13e0 2024-05-08T21:21:48.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.994+0000 7fad577fe700 1 -- 172.21.15.3:0/1431724409 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad60006710 con 0x7fad700d13e0 2024-05-08T21:21:48.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.994+0000 7fad752a7700 1 -- 172.21.15.3:0/1431724409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad700c7b90 con 0x7fad700d13e0 2024-05-08T21:21:48.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.994+0000 7fad577fe700 1 -- 172.21.15.3:0/1431724409 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad600036a0 con 0x7fad700d13e0 2024-05-08T21:21:48.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.994+0000 7fad752a7700 1 -- 172.21.15.3:0/1431724409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad700c8150 con 0x7fad700d13e0 2024-05-08T21:21:48.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.995+0000 7fad752a7700 1 -- 172.21.15.3:0/1431724409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad700d1a50 con 0x7fad700d13e0 2024-05-08T21:21:48.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.996+0000 7fad577fe700 1 -- 172.21.15.3:0/1431724409 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fad60022070 con 0x7fad700d13e0 2024-05-08T21:21:48.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.996+0000 7fad577fe700 1 --2- 172.21.15.3:0/1431724409 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad5c06ac90 0x7fad5c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:48.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.996+0000 7fad577fe700 1 -- 172.21.15.3:0/1431724409 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fad6008a860 con 0x7fad700d13e0 2024-05-08T21:21:48.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.996+0000 7fad6e7fc700 1 --2- 172.21.15.3:0/1431724409 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad5c06ac90 0x7fad5c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:48.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:48.997+0000 7fad6e7fc700 1 --2- 172.21.15.3:0/1431724409 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad5c06ac90 0x7fad5c06d150 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fad700b9440 tx=0x7fad64009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:49.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.001+0000 7fad577fe700 1 -- 172.21.15.3:0/1431724409 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fad60055d90 con 0x7fad700d13e0 2024-05-08T21:21:49.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.220+0000 7fad752a7700 1 -- 172.21.15.3:0/1431724409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fad700d3090 con 0x7fad700d13e0 2024-05-08T21:21:49.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.220+0000 7fad577fe700 1 -- 172.21.15.3:0/1431724409 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7fad6002a090 con 0x7fad700d13e0 2024-05-08T21:21:49.223 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:21:49.223 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2024-05-08T21:21:49.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:21:49.223 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2024-05-08T21:21:49.223 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": ".mgr" 2024-05-08T21:21:49.223 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:21:49.223 INFO:teuthology.orchestra.run.smithi003.stdout:] 2024-05-08T21:21:49.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.223+0000 7fad752a7700 1 -- 172.21.15.3:0/1431724409 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad5c06ac90 msgr2=0x7fad5c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:49.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.223+0000 7fad752a7700 1 --2- 172.21.15.3:0/1431724409 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad5c06ac90 0x7fad5c06d150 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fad700b9440 tx=0x7fad64009040 comp rx=0 tx=0).stop 2024-05-08T21:21:49.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.223+0000 7fad752a7700 1 -- 172.21.15.3:0/1431724409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad700d13e0 msgr2=0x7fad700c6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:49.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.223+0000 7fad752a7700 1 --2- 172.21.15.3:0/1431724409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad700d13e0 0x7fad700c6ff0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fad60000c00 tx=0x7fad60004750 comp rx=0 tx=0).stop 2024-05-08T21:21:49.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.223+0000 7fad752a7700 1 -- 172.21.15.3:0/1431724409 shutdown_connections 2024-05-08T21:21:49.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.223+0000 7fad752a7700 1 --2- 172.21.15.3:0/1431724409 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad5c06ac90 0x7fad5c06d150 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:49.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.223+0000 7fad752a7700 1 --2- 172.21.15.3:0/1431724409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad7015d780 0x7fad700c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:49.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.223+0000 7fad752a7700 1 --2- 172.21.15.3:0/1431724409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad700d13e0 0x7fad700c6ff0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:49.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.223+0000 7fad752a7700 1 --2- 172.21.15.3:0/1431724409 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad700d22f0 0x7fad700c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:49.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.223+0000 7fad752a7700 1 -- 172.21.15.3:0/1431724409 >> 172.21.15.3:0/1431724409 conn(0x7fad7008afe0 msgr2=0x7fad700cec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:49.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.223+0000 7fad752a7700 1 -- 172.21.15.3:0/1431724409 shutdown_connections 2024-05-08T21:21:49.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.223+0000 7fad752a7700 1 -- 172.21.15.3:0/1431724409 wait complete. 2024-05-08T21:21:49.238 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-08T21:21:49.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-08T21:21:49.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.377+0000 7fddd7f7c700 1 Processor -- start 2024-05-08T21:21:49.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.377+0000 7fddd7f7c700 1 -- start start 2024-05-08T21:21:49.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.378+0000 7fddd7f7c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd00af860 0x7fddd00d0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:49.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.378+0000 7fddd7f7c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fddd00c5720 0x7fddd00c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:49.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.378+0000 7fddd7f7c700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd00cfbd0 con 0x7fddd015a750 2024-05-08T21:21:49.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.378+0000 7fddd7f7c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd00c60f0 con 0x7fddd00c5720 2024-05-08T21:21:49.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.378+0000 7fddd7f7c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd00cf440 con 0x7fddd00af860 2024-05-08T21:21:49.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.378+0000 7fddd6519700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fddd00c5720 0x7fddd00c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:49.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.378+0000 7fddd5d18700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fddd015a750 0x7fddd00ad260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:41294/0 (socket says 172.21.15.3:41294) 2024-05-08T21:21:49.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.378+0000 7fddd5d18700 1 -- 172.21.15.3:0/2013103876 learned_addr learned my addr 172.21.15.3:0/2013103876 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:49.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.378+0000 7fddd6519700 1 --2- 172.21.15.3:0/2013103876 >> v2:172.21.15.3:3300/0 conn(0x7fddd00c5720 0x7fddd00c5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38436/0 (socket says 172.21.15.3:38436) 2024-05-08T21:21:49.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.378+0000 7fddd5517700 1 --2- 172.21.15.3:0/2013103876 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd00af860 0x7fddd00d0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:49.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.379+0000 7fddd6519700 1 -- 172.21.15.3:0/2013103876 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd00af860 msgr2=0x7fddd00d0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:49.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.379+0000 7fddd6519700 1 --2- 172.21.15.3:0/2013103876 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd00af860 0x7fddd00d0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:49.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.379+0000 7fddd6519700 1 -- 172.21.15.3:0/2013103876 >> v1:172.21.15.3:6789/0 conn(0x7fddd015a750 legacy=0x7fddd00ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.379+0000 7fddd6519700 1 -- 172.21.15.3:0/2013103876 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd00cb780 con 0x7fddd00c5720 2024-05-08T21:21:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.379+0000 7fddd4d16700 1 -- 172.21.15.3:0/2013103876 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066531422 0 0) 0x7fddd00cfbd0 con 0x7fddd015a750 2024-05-08T21:21:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.379+0000 7fddd6519700 1 --2- 172.21.15.3:0/2013103876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddd00c5720 0x7fddd00c5bb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fddc800bfa0 tx=0x7fddc800d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=175c398cf21c235f server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.379+0000 7fddd4d16700 1 -- 172.21.15.3:0/2013103876 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc8010070 con 0x7fddd00c5720 2024-05-08T21:21:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.379+0000 7fddd4d16700 1 -- 172.21.15.3:0/2013103876 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc800b2e0 con 0x7fddd00c5720 2024-05-08T21:21:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.379+0000 7fddd4d16700 1 -- 172.21.15.3:0/2013103876 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc800dc40 con 0x7fddd00c5720 2024-05-08T21:21:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.380+0000 7fddd7f7c700 1 -- 172.21.15.3:0/2013103876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddd00c5720 msgr2=0x7fddd00c5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.380+0000 7fddd7f7c700 1 --2- 172.21.15.3:0/2013103876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddd00c5720 0x7fddd00c5bb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fddc800bfa0 tx=0x7fddc800d330 comp rx=0 tx=0).stop 2024-05-08T21:21:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.380+0000 7fddd7f7c700 1 -- 172.21.15.3:0/2013103876 shutdown_connections 2024-05-08T21:21:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.380+0000 7fddd7f7c700 1 --2- 172.21.15.3:0/2013103876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddd00c5720 0x7fddd00c5bb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.380+0000 7fddd7f7c700 1 --2- 172.21.15.3:0/2013103876 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd00af860 0x7fddd00d0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.380+0000 7fddd7f7c700 1 -- 172.21.15.3:0/2013103876 >> 172.21.15.3:0/2013103876 conn(0x7fddd00d03a0 msgr2=0x7fddd00d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.380+0000 7fddd7f7c700 1 -- 172.21.15.3:0/2013103876 shutdown_connections 2024-05-08T21:21:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.380+0000 7fddd7f7c700 1 -- 172.21.15.3:0/2013103876 wait complete. 2024-05-08T21:21:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.381+0000 7fddd7f7c700 1 Processor -- start 2024-05-08T21:21:49.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.381+0000 7fddd7f7c700 1 -- start start 2024-05-08T21:21:49.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.382+0000 7fddd7f7c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddd00af860 0x7fddd01462e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:49.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.382+0000 7fddd7f7c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd00c5720 0x7fddd01572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:49.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.382+0000 7fddd7f7c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddd015a750 0x7fddd0157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:49.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.382+0000 7fddd7f7c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd00c8150 con 0x7fddd015a750 2024-05-08T21:21:49.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.382+0000 7fddd7f7c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd00c7580 con 0x7fddd00af860 2024-05-08T21:21:49.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.382+0000 7fddd7f7c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd0146820 con 0x7fddd00c5720 2024-05-08T21:21:49.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.382+0000 7fddd6519700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddd015a750 0x7fddd0157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:49.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.382+0000 7fddd6519700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddd015a750 0x7fddd0157810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38446/0 (socket says 172.21.15.3:38446) 2024-05-08T21:21:49.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.382+0000 7fddd6519700 1 -- 172.21.15.3:0/30453534 learned_addr learned my addr 172.21.15.3:0/30453534 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:49.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.382+0000 7fddd5d18700 1 --2- 172.21.15.3:0/30453534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddd00af860 0x7fddd01462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:49.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.382+0000 7fddd5517700 1 --2- 172.21.15.3:0/30453534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd00c5720 0x7fddd01572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:49.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.383+0000 7fddd6519700 1 -- 172.21.15.3:0/30453534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd00c5720 msgr2=0x7fddd01572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:49.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.383+0000 7fddd6519700 1 --2- 172.21.15.3:0/30453534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd00c5720 0x7fddd01572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:49.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.383+0000 7fddd6519700 1 -- 172.21.15.3:0/30453534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddd00af860 msgr2=0x7fddd01462e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:49.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.383+0000 7fddd6519700 1 --2- 172.21.15.3:0/30453534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddd00af860 0x7fddd01462e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:49.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.383+0000 7fddd6519700 1 -- 172.21.15.3:0/30453534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc800bc80 con 0x7fddd015a750 2024-05-08T21:21:49.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.383+0000 7fddd6519700 1 --2- 172.21.15.3:0/30453534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddd015a750 0x7fddd0157810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fddd00717e0 tx=0x7fddc8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:49.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.383+0000 7fddc27fc700 1 -- 172.21.15.3:0/30453534 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc8004450 con 0x7fddd015a750 2024-05-08T21:21:49.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.383+0000 7fddc27fc700 1 -- 172.21.15.3:0/30453534 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc80045b0 con 0x7fddd015a750 2024-05-08T21:21:49.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.383+0000 7fddd7f7c700 1 -- 172.21.15.3:0/30453534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd00c7820 con 0x7fddd015a750 2024-05-08T21:21:49.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.383+0000 7fddc27fc700 1 -- 172.21.15.3:0/30453534 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc8004960 con 0x7fddd015a750 2024-05-08T21:21:49.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.383+0000 7fddd7f7c700 1 -- 172.21.15.3:0/30453534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd00c7de0 con 0x7fddd015a750 2024-05-08T21:21:49.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.384+0000 7fddd7f7c700 1 -- 172.21.15.3:0/30453534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd00c7fa0 con 0x7fddd015a750 2024-05-08T21:21:49.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.385+0000 7fddc27fc700 1 -- 172.21.15.3:0/30453534 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fddc8026450 con 0x7fddd015a750 2024-05-08T21:21:49.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.385+0000 7fddc27fc700 1 --2- 172.21.15.3:0/30453534 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fddb406ac90 0x7fddb406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:49.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.385+0000 7fddc27fc700 1 -- 172.21.15.3:0/30453534 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fddc808ab50 con 0x7fddd015a750 2024-05-08T21:21:49.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.385+0000 7fddd5d18700 1 --2- 172.21.15.3:0/30453534 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fddb406ac90 0x7fddb406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:49.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.386+0000 7fddd5d18700 1 --2- 172.21.15.3:0/30453534 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fddb406ac90 0x7fddb406d150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fddcc007fd0 tx=0x7fddcc007de0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:49.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.390+0000 7fddc27fc700 1 -- 172.21.15.3:0/30453534 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fddc8056110 con 0x7fddd015a750 2024-05-08T21:21:49.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:49.606+0000 7fddd7f7c700 1 -- 172.21.15.3:0/30453534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fddd0074bc0 con 0x7fddd015a750 2024-05-08T21:21:50.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.592+0000 7fddc27fc700 1 -- 172.21.15.3:0/30453534 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fddc8059730 con 0x7fddd015a750 2024-05-08T21:21:50.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.597+0000 7fddd7f7c700 1 -- 172.21.15.3:0/30453534 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fddb406ac90 msgr2=0x7fddb406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:50.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.597+0000 7fddd7f7c700 1 --2- 172.21.15.3:0/30453534 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fddb406ac90 0x7fddb406d150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fddcc007fd0 tx=0x7fddcc007de0 comp rx=0 tx=0).stop 2024-05-08T21:21:50.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.597+0000 7fddd7f7c700 1 -- 172.21.15.3:0/30453534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddd015a750 msgr2=0x7fddd0157810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:50.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.597+0000 7fddd7f7c700 1 --2- 172.21.15.3:0/30453534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddd015a750 0x7fddd0157810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fddd00717e0 tx=0x7fddc8004ad0 comp rx=0 tx=0).stop 2024-05-08T21:21:50.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.597+0000 7fddd7f7c700 1 -- 172.21.15.3:0/30453534 shutdown_connections 2024-05-08T21:21:50.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.597+0000 7fddd7f7c700 1 --2- 172.21.15.3:0/30453534 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fddb406ac90 0x7fddb406d150 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:50.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.597+0000 7fddd7f7c700 1 --2- 172.21.15.3:0/30453534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddd015a750 0x7fddd0157810 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:50.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.597+0000 7fddd7f7c700 1 --2- 172.21.15.3:0/30453534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddd00af860 0x7fddd01462e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:50.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.597+0000 7fddd7f7c700 1 --2- 172.21.15.3:0/30453534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd00c5720 0x7fddd01572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:50.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.597+0000 7fddd7f7c700 1 -- 172.21.15.3:0/30453534 >> 172.21.15.3:0/30453534 conn(0x7fddd00d03a0 msgr2=0x7fddd00ae060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:50.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.597+0000 7fddd7f7c700 1 -- 172.21.15.3:0/30453534 shutdown_connections 2024-05-08T21:21:50.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.597+0000 7fddd7f7c700 1 -- 172.21.15.3:0/30453534 wait complete. 2024-05-08T21:21:50.600 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2024-05-08T21:21:50.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-08T21:21:50.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.761+0000 7f947a9a0700 1 Processor -- start 2024-05-08T21:21:50.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.761+0000 7f947a9a0700 1 -- start start 2024-05-08T21:21:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.762+0000 7f947a9a0700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f947415ba10 0x7f94740cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.762+0000 7f947a9a0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947406b070 0x7f94740ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.762+0000 7f947a9a0700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947415f5f0 con 0x7f94740cfde0 2024-05-08T21:21:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.762+0000 7f947a9a0700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947415f3e0 con 0x7f947415ba10 2024-05-08T21:21:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.762+0000 7f947a9a0700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94741603b0 con 0x7f947406b070 2024-05-08T21:21:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.763+0000 7f9473fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f947415ba10 0x7f94740cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.763+0000 7f9478f3d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947406b070 0x7f94740ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.763+0000 7f946bfff700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f94740cfde0 0x7f947406a960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:40564/0 (socket says 172.21.15.3:40564) 2024-05-08T21:21:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.763+0000 7f946bfff700 1 -- 172.21.15.3:0/1677849666 learned_addr learned my addr 172.21.15.3:0/1677849666 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.763+0000 7f9478f3d700 1 -- 172.21.15.3:0/1677849666 >> v2:172.21.15.186:3300/0 conn(0x7f947415ba10 msgr2=0x7f94740cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.763+0000 7f9478f3d700 1 --2- 172.21.15.3:0/1677849666 >> v2:172.21.15.186:3300/0 conn(0x7f947415ba10 0x7f94740cf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.763+0000 7f9478f3d700 1 -- 172.21.15.3:0/1677849666 >> v1:172.21.15.186:6789/0 conn(0x7f94740cfde0 legacy=0x7f947406a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.763+0000 7f9478f3d700 1 -- 172.21.15.3:0/1677849666 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947415f8d0 con 0x7f947406b070 2024-05-08T21:21:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.763+0000 7f9473fff700 1 --2- 172.21.15.3:0/1677849666 >> v2:172.21.15.186:3300/0 conn(0x7f947415ba10 0x7f94740cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:21:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.763+0000 7f9478f3d700 1 --2- 172.21.15.3:0/1677849666 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947406b070 0x7f94740ce230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f946c009d90 tx=0x7f946c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ada0e6e086aa5daf server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.763+0000 7f94737fe700 1 -- 172.21.15.3:0/1677849666 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c00e070 con 0x7f947406b070 2024-05-08T21:21:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.764+0000 7f94737fe700 1 -- 172.21.15.3:0/1677849666 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946c00bb20 con 0x7f947406b070 2024-05-08T21:21:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.764+0000 7f94737fe700 1 -- 172.21.15.3:0/1677849666 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c012430 con 0x7f947406b070 2024-05-08T21:21:50.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.764+0000 7f947a9a0700 1 -- 172.21.15.3:0/1677849666 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947406b070 msgr2=0x7f94740ce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:50.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.764+0000 7f947a9a0700 1 --2- 172.21.15.3:0/1677849666 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947406b070 0x7f94740ce230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f946c009d90 tx=0x7f946c00b330 comp rx=0 tx=0).stop 2024-05-08T21:21:50.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.764+0000 7f947a9a0700 1 -- 172.21.15.3:0/1677849666 shutdown_connections 2024-05-08T21:21:50.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.764+0000 7f947a9a0700 1 --2- 172.21.15.3:0/1677849666 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947406b070 0x7f94740ce230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:50.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.764+0000 7f947a9a0700 1 --2- 172.21.15.3:0/1677849666 >> v2:172.21.15.186:3300/0 conn(0x7f947415ba10 0x7f94740cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:50.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.764+0000 7f947a9a0700 1 -- 172.21.15.3:0/1677849666 >> 172.21.15.3:0/1677849666 conn(0x7f9474099a50 msgr2=0x7f94740d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:50.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.764+0000 7f947a9a0700 1 -- 172.21.15.3:0/1677849666 shutdown_connections 2024-05-08T21:21:50.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.764+0000 7f947a9a0700 1 -- 172.21.15.3:0/1677849666 wait complete. 2024-05-08T21:21:50.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.766+0000 7f947a9a0700 1 Processor -- start 2024-05-08T21:21:50.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.766+0000 7f947a9a0700 1 -- start start 2024-05-08T21:21:50.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.767+0000 7f947a9a0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f947406b070 0x7f9474148590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:50.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.767+0000 7f947a9a0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94740cfde0 0x7f9474157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:50.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.767+0000 7f947a9a0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947415ba10 0x7f9474158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:50.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.767+0000 7f947a9a0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94740c6d20 con 0x7f94740cfde0 2024-05-08T21:21:50.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.767+0000 7f947a9a0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94740c6480 con 0x7f947406b070 2024-05-08T21:21:50.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.767+0000 7f947a9a0700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94740c6760 con 0x7f947415ba10 2024-05-08T21:21:50.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.767+0000 7f946bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94740cfde0 0x7f9474157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.767+0000 7f946bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94740cfde0 0x7f9474157b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38458/0 (socket says 172.21.15.3:38458) 2024-05-08T21:21:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.767+0000 7f946bfff700 1 -- 172.21.15.3:0/2445294658 learned_addr learned my addr 172.21.15.3:0/2445294658 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.767+0000 7f9473fff700 1 --2- 172.21.15.3:0/2445294658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f947406b070 0x7f9474148590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.767+0000 7f9478f3d700 1 --2- 172.21.15.3:0/2445294658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947415ba10 0x7f9474158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.768+0000 7f9473fff700 1 -- 172.21.15.3:0/2445294658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947415ba10 msgr2=0x7f9474158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.768+0000 7f9473fff700 1 --2- 172.21.15.3:0/2445294658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947415ba10 0x7f9474158050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.768+0000 7f9473fff700 1 -- 172.21.15.3:0/2445294658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94740cfde0 msgr2=0x7f9474157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.768+0000 7f9473fff700 1 --2- 172.21.15.3:0/2445294658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94740cfde0 0x7f9474157b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.768+0000 7f9473fff700 1 -- 172.21.15.3:0/2445294658 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94640094a0 con 0x7f947406b070 2024-05-08T21:21:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.768+0000 7f946bfff700 1 --2- 172.21.15.3:0/2445294658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94740cfde0 0x7f9474157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:21:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.768+0000 7f9473fff700 1 --2- 172.21.15.3:0/2445294658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f947406b070 0x7f9474148590 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f946400e900 tx=0x7f946400ec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.768+0000 7f94717fa700 1 -- 172.21.15.3:0/2445294658 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464013680 con 0x7f947406b070 2024-05-08T21:21:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.768+0000 7f94717fa700 1 -- 172.21.15.3:0/2445294658 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94640137e0 con 0x7f947406b070 2024-05-08T21:21:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.768+0000 7f94717fa700 1 -- 172.21.15.3:0/2445294658 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464013b70 con 0x7f947406b070 2024-05-08T21:21:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.768+0000 7f947a9a0700 1 -- 172.21.15.3:0/2445294658 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946c009a70 con 0x7f947406b070 2024-05-08T21:21:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.768+0000 7f947a9a0700 1 -- 172.21.15.3:0/2445294658 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94740c75a0 con 0x7f947406b070 2024-05-08T21:21:50.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.769+0000 7f947a9a0700 1 -- 172.21.15.3:0/2445294658 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94740c73e0 con 0x7f947406b070 2024-05-08T21:21:50.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.770+0000 7f94717fa700 1 -- 172.21.15.3:0/2445294658 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9464013cd0 con 0x7f947406b070 2024-05-08T21:21:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.770+0000 7f94717fa700 1 --2- 172.21.15.3:0/2445294658 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f945406acc0 0x7f945406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.770+0000 7f94717fa700 1 -- 172.21.15.3:0/2445294658 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f9464089180 con 0x7f947406b070 2024-05-08T21:21:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.770+0000 7f946bfff700 1 --2- 172.21.15.3:0/2445294658 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f945406acc0 0x7f945406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.771+0000 7f946bfff700 1 --2- 172.21.15.3:0/2445294658 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f945406acc0 0x7f945406d180 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f94740c3830 tx=0x7f945c001120 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:50.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.775+0000 7f94717fa700 1 -- 172.21.15.3:0/2445294658 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f94640544b0 con 0x7f947406b070 2024-05-08T21:21:50.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:50.992+0000 7f947a9a0700 1 -- 172.21.15.3:0/2445294658 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f94740c77c0 con 0x7f947406b070 2024-05-08T21:21:51.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.602+0000 7f94717fa700 1 -- 172.21.15.3:0/2445294658 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7f94640210d0 con 0x7f947406b070 2024-05-08T21:21:51.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.612+0000 7f947a9a0700 1 -- 172.21.15.3:0/2445294658 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f945406acc0 msgr2=0x7f945406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:51.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.612+0000 7f947a9a0700 1 --2- 172.21.15.3:0/2445294658 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f945406acc0 0x7f945406d180 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f94740c3830 tx=0x7f945c001120 comp rx=0 tx=0).stop 2024-05-08T21:21:51.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.612+0000 7f947a9a0700 1 -- 172.21.15.3:0/2445294658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f947406b070 msgr2=0x7f9474148590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:51.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.612+0000 7f947a9a0700 1 --2- 172.21.15.3:0/2445294658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f947406b070 0x7f9474148590 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f946400e900 tx=0x7f946400ec10 comp rx=0 tx=0).stop 2024-05-08T21:21:51.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.615+0000 7f947a9a0700 1 -- 172.21.15.3:0/2445294658 shutdown_connections 2024-05-08T21:21:51.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.615+0000 7f947a9a0700 1 --2- 172.21.15.3:0/2445294658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947415ba10 0x7f9474158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:51.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.615+0000 7f947a9a0700 1 --2- 172.21.15.3:0/2445294658 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f945406acc0 0x7f945406d180 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:51.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.615+0000 7f947a9a0700 1 --2- 172.21.15.3:0/2445294658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f947406b070 0x7f9474148590 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:51.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.615+0000 7f947a9a0700 1 --2- 172.21.15.3:0/2445294658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94740cfde0 0x7f9474157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:51.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.615+0000 7f947a9a0700 1 -- 172.21.15.3:0/2445294658 >> 172.21.15.3:0/2445294658 conn(0x7f9474099a50 msgr2=0x7f94740b1370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:51.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.616+0000 7f947a9a0700 1 -- 172.21.15.3:0/2445294658 shutdown_connections 2024-05-08T21:21:51.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.616+0000 7f947a9a0700 1 -- 172.21.15.3:0/2445294658 wait complete. 2024-05-08T21:21:51.619 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2024-05-08T21:21:51.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-08T21:21:51.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.776+0000 7f083c2b4700 1 Processor -- start 2024-05-08T21:21:51.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.776+0000 7f083c2b4700 1 -- start start 2024-05-08T21:21:51.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.777+0000 7f083c2b4700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08340cba90 0x7f083406fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:51.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.777+0000 7f083c2b4700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083406f930 con 0x7f08340d0690 2024-05-08T21:21:51.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.777+0000 7f083c2b4700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08340cfdf0 con 0x7f0834099a50 2024-05-08T21:21:51.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.777+0000 7f083c2b4700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08340d00d0 con 0x7f08340cba90 2024-05-08T21:21:51.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.777+0000 7f083984f700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f08340d0690 0x7f083409a790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:32906/0 (socket says 172.21.15.3:32906) 2024-05-08T21:21:51.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.777+0000 7f083984f700 1 -- 172.21.15.3:0/2477320977 learned_addr learned my addr 172.21.15.3:0/2477320977 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:51.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083a851700 1 --2- 172.21.15.3:0/2477320977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08340cba90 0x7f083406fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:51.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 -- 172.21.15.3:0/2477320977 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827727249 0 0) 0x7f083406f930 con 0x7f08340d0690 2024-05-08T21:21:51.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 -- 172.21.15.3:0/2477320977 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08200035b0 con 0x7f08340d0690 2024-05-08T21:21:51.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 -- 172.21.15.3:0/2477320977 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121085282 0 0) 0x7f08340cfdf0 con 0x7f0834099a50 2024-05-08T21:21:51.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 -- 172.21.15.3:0/2477320977 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f083406f930 con 0x7f0834099a50 2024-05-08T21:21:51.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 -- 172.21.15.3:0/2477320977 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 433328660 0 0) 0x7f08200035b0 con 0x7f08340d0690 2024-05-08T21:21:51.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 -- 172.21.15.3:0/2477320977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08340cba90 msgr2=0x7f083406fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:51.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 --2- 172.21.15.3:0/2477320977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08340cba90 0x7f083406fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:51.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 -- 172.21.15.3:0/2477320977 >> v1:172.21.15.186:6789/0 conn(0x7f0834099a50 legacy=0x7f0834099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:51.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 -- 172.21.15.3:0/2477320977 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08340d03b0 con 0x7f08340d0690 2024-05-08T21:21:51.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 -- 172.21.15.3:0/2477320977 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f0828001670 con 0x7f08340d0690 2024-05-08T21:21:51.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 -- 172.21.15.3:0/2477320977 >> v1:172.21.15.3:6789/0 conn(0x7f08340d0690 legacy=0x7f083409a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:51.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 --2- 172.21.15.3:0/2477320977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0820003bc0 0x7f0820003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:51.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 --2- 172.21.15.3:0/2477320977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0820004850 0x7f0820004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:51.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 --2- 172.21.15.3:0/2477320977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08340cba90 0x7f0820005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:51.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 -- 172.21.15.3:0/2477320977 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08340d00d0 con 0x7f0820003bc0 2024-05-08T21:21:51.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 -- 172.21.15.3:0/2477320977 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082000bb50 con 0x7f0820004850 2024-05-08T21:21:51.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083904e700 1 -- 172.21.15.3:0/2477320977 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f082000bd10 con 0x7f08340cba90 2024-05-08T21:21:51.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083a050700 1 --2- 172.21.15.3:0/2477320977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0820003bc0 0x7f0820003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:51.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083984f700 1 --2- 172.21.15.3:0/2477320977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0820004850 0x7f0820004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:51.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.778+0000 7f083a851700 1 --2- 172.21.15.3:0/2477320977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08340cba90 0x7f0820005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:51.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.779+0000 7f083a050700 1 -- 172.21.15.3:0/2477320977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08340cba90 msgr2=0x7f0820005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:51.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.779+0000 7f083a050700 1 --2- 172.21.15.3:0/2477320977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08340cba90 0x7f0820005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:51.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.779+0000 7f083a050700 1 -- 172.21.15.3:0/2477320977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0820004850 msgr2=0x7f0820004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:51.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.779+0000 7f083a050700 1 --2- 172.21.15.3:0/2477320977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0820004850 0x7f0820004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:51.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.779+0000 7f083a050700 1 -- 172.21.15.3:0/2477320977 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08200035b0 con 0x7f0820003bc0 2024-05-08T21:21:51.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.779+0000 7f083a050700 1 --2- 172.21.15.3:0/2477320977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0820003bc0 0x7f0820003fc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0830001980 tx=0x7f0820005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aded3a2d2b5a979a server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:51.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.779+0000 7f083904e700 1 -- 172.21.15.3:0/2477320977 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830001470 con 0x7f0820003bc0 2024-05-08T21:21:51.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.779+0000 7f083904e700 1 -- 172.21.15.3:0/2477320977 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08300052d0 con 0x7f0820003bc0 2024-05-08T21:21:51.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.779+0000 7f083904e700 1 -- 172.21.15.3:0/2477320977 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830003680 con 0x7f0820003bc0 2024-05-08T21:21:51.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.779+0000 7f083c2b4700 1 -- 172.21.15.3:0/2477320977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0820003bc0 msgr2=0x7f0820003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:51.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.779+0000 7f083c2b4700 1 --2- 172.21.15.3:0/2477320977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0820003bc0 0x7f0820003fc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0830001980 tx=0x7f0820005880 comp rx=0 tx=0).stop 2024-05-08T21:21:51.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.779+0000 7f083a851700 1 -- 172.21.15.3:0/2477320977 reap_dead start 2024-05-08T21:21:51.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.780+0000 7f083c2b4700 1 -- 172.21.15.3:0/2477320977 shutdown_connections 2024-05-08T21:21:51.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.780+0000 7f083c2b4700 1 -- 172.21.15.3:0/2477320977 >> 172.21.15.3:0/2477320977 conn(0x7f08340b2f30 msgr2=0x7f08340b3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:51.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.780+0000 7f083c2b4700 1 -- 172.21.15.3:0/2477320977 shutdown_connections 2024-05-08T21:21:51.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.780+0000 7f083c2b4700 1 -- 172.21.15.3:0/2477320977 wait complete. 2024-05-08T21:21:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.781+0000 7f083c2b4700 1 Processor -- start 2024-05-08T21:21:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.781+0000 7f083c2b4700 1 -- start start 2024-05-08T21:21:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083c2b4700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0820003bc0 0x7f08341f6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083c2b4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08340cafb0 0x7f083406cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083c2b4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08340c8d40 0x7f08340c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083c2b4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083406bc10 con 0x7f08340cafb0 2024-05-08T21:21:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083c2b4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08340c9700 con 0x7f08340c8d40 2024-05-08T21:21:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083c2b4700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08341f7970 con 0x7f0820003bc0 2024-05-08T21:21:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083984f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08340cafb0 0x7f083406cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083984f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08340cafb0 0x7f083406cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52144/0 (socket says 172.21.15.3:52144) 2024-05-08T21:21:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083984f700 1 -- 172.21.15.3:0/1540523698 learned_addr learned my addr 172.21.15.3:0/1540523698 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083a851700 1 --2- 172.21.15.3:0/1540523698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08340c8d40 0x7f08340c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083a050700 1 --2- 172.21.15.3:0/1540523698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0820003bc0 0x7f08341f6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083984f700 1 -- 172.21.15.3:0/1540523698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0820003bc0 msgr2=0x7f08341f6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083984f700 1 --2- 172.21.15.3:0/1540523698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0820003bc0 0x7f08341f6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083984f700 1 -- 172.21.15.3:0/1540523698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08340c8d40 msgr2=0x7f08340c91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083984f700 1 --2- 172.21.15.3:0/1540523698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08340c8d40 0x7f08340c91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083984f700 1 -- 172.21.15.3:0/1540523698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08340cb430 con 0x7f08340cafb0 2024-05-08T21:21:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083a851700 1 --2- 172.21.15.3:0/1540523698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08340c8d40 0x7f08340c91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:21:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f083984f700 1 --2- 172.21.15.3:0/1540523698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08340cafb0 0x7f083406cc30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0828007340 tx=0x7f0828007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f0826ffd700 1 -- 172.21.15.3:0/1540523698 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082800f660 con 0x7f08340cafb0 2024-05-08T21:21:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.782+0000 7f0826ffd700 1 -- 172.21.15.3:0/1540523698 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082800f7e0 con 0x7f08340cafb0 2024-05-08T21:21:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.783+0000 7f0826ffd700 1 -- 172.21.15.3:0/1540523698 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082800fb90 con 0x7f08340cafb0 2024-05-08T21:21:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.783+0000 7f083c2b4700 1 -- 172.21.15.3:0/1540523698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08341f70d0 con 0x7f08340cafb0 2024-05-08T21:21:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.783+0000 7f083c2b4700 1 -- 172.21.15.3:0/1540523698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08341f7690 con 0x7f08340cafb0 2024-05-08T21:21:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.784+0000 7f083c2b4700 1 -- 172.21.15.3:0/1540523698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08340d1240 con 0x7f08340cafb0 2024-05-08T21:21:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.784+0000 7f0826ffd700 1 -- 172.21.15.3:0/1540523698 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f082800fcf0 con 0x7f08340cafb0 2024-05-08T21:21:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.785+0000 7f0826ffd700 1 --2- 172.21.15.3:0/1540523698 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f081806ac90 0x7f081806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.785+0000 7f0826ffd700 1 -- 172.21.15.3:0/1540523698 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f0828089240 con 0x7f08340cafb0 2024-05-08T21:21:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.785+0000 7f083a050700 1 --2- 172.21.15.3:0/1540523698 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f081806ac90 0x7f081806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.785+0000 7f083a050700 1 --2- 172.21.15.3:0/1540523698 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f081806ac90 0x7f081806d150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f083415cef0 tx=0x7f083000a000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:51.790+0000 7f0826ffd700 1 -- 172.21.15.3:0/1540523698 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0828054490 con 0x7f08340cafb0 2024-05-08T21:21:52.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.060+0000 7f083c2b4700 1 -- 172.21.15.3:0/1540523698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0834074bc0 con 0x7f08340cafb0 2024-05-08T21:21:52.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.065+0000 7f0826ffd700 1 -- 172.21.15.3:0/1540523698 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0828057ab0 con 0x7f08340cafb0 2024-05-08T21:21:52.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.071+0000 7f083c2b4700 1 -- 172.21.15.3:0/1540523698 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f081806ac90 msgr2=0x7f081806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:52.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.071+0000 7f083c2b4700 1 --2- 172.21.15.3:0/1540523698 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f081806ac90 0x7f081806d150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f083415cef0 tx=0x7f083000a000 comp rx=0 tx=0).stop 2024-05-08T21:21:52.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.072+0000 7f083c2b4700 1 -- 172.21.15.3:0/1540523698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08340cafb0 msgr2=0x7f083406cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:52.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.072+0000 7f083c2b4700 1 --2- 172.21.15.3:0/1540523698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08340cafb0 0x7f083406cc30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0828007340 tx=0x7f0828007650 comp rx=0 tx=0).stop 2024-05-08T21:21:52.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.073+0000 7f083c2b4700 1 -- 172.21.15.3:0/1540523698 shutdown_connections 2024-05-08T21:21:52.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.073+0000 7f083c2b4700 1 --2- 172.21.15.3:0/1540523698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0820003bc0 0x7f08341f6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.073+0000 7f083c2b4700 1 --2- 172.21.15.3:0/1540523698 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f081806ac90 0x7f081806d150 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.073+0000 7f083c2b4700 1 --2- 172.21.15.3:0/1540523698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08340c8d40 0x7f08340c91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.073+0000 7f083c2b4700 1 --2- 172.21.15.3:0/1540523698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08340cafb0 0x7f083406cc30 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0828007340 tx=0x7f0828007650 comp rx=0 tx=0).stop 2024-05-08T21:21:52.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.073+0000 7f083c2b4700 1 -- 172.21.15.3:0/1540523698 >> 172.21.15.3:0/1540523698 conn(0x7f08340b2f30 msgr2=0x7f083415faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:52.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.073+0000 7f083c2b4700 1 -- 172.21.15.3:0/1540523698 shutdown_connections 2024-05-08T21:21:52.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.074+0000 7f083c2b4700 1 -- 172.21.15.3:0/1540523698 wait complete. 2024-05-08T21:21:52.078 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 2 and data pool 3 2024-05-08T21:21:52.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:21:52.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.233+0000 7fa77b424700 1 Processor -- start 2024-05-08T21:21:52.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.233+0000 7fa77b424700 1 -- start start 2024-05-08T21:21:52.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.234+0000 7fa77b424700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa7740ccc40 0x7fa7741665c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:52.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.234+0000 7fa77b424700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7740cd3b0 con 0x7fa77415f230 2024-05-08T21:21:52.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.234+0000 7fa77b424700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7740af300 con 0x7fa7740aef20 2024-05-08T21:21:52.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.234+0000 7fa77b424700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa774141df0 con 0x7fa7740ccc40 2024-05-08T21:21:52.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.234+0000 7fa7789bf700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa77415f230 0x7fa7740cc530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:32918/0 (socket says 172.21.15.3:32918) 2024-05-08T21:21:52.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.234+0000 7fa7789bf700 1 -- 172.21.15.3:0/388550557 learned_addr learned my addr 172.21.15.3:0/388550557 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:52.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.234+0000 7fa7799c1700 1 --2- 172.21.15.3:0/388550557 >> v2:172.21.15.186:3300/0 conn(0x7fa7740ccc40 0x7fa7741665c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:52.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.234+0000 7fa76bfff700 1 -- 172.21.15.3:0/388550557 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845183183 0 0) 0x7fa7740cd3b0 con 0x7fa77415f230 2024-05-08T21:21:52.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.234+0000 7fa76bfff700 1 -- 172.21.15.3:0/388550557 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7600035b0 con 0x7fa77415f230 2024-05-08T21:21:52.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa76bfff700 1 -- 172.21.15.3:0/388550557 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3240596638 0 0) 0x7fa7600035b0 con 0x7fa77415f230 2024-05-08T21:21:52.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa76bfff700 1 -- 172.21.15.3:0/388550557 >> v2:172.21.15.186:3300/0 conn(0x7fa7740ccc40 msgr2=0x7fa7741665c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:52.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa76bfff700 1 --2- 172.21.15.3:0/388550557 >> v2:172.21.15.186:3300/0 conn(0x7fa7740ccc40 0x7fa7741665c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa76bfff700 1 -- 172.21.15.3:0/388550557 >> v1:172.21.15.186:6789/0 conn(0x7fa7740aef20 legacy=0x7fa77415c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:52.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa76bfff700 1 -- 172.21.15.3:0/388550557 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77415f6f0 con 0x7fa77415f230 2024-05-08T21:21:52.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa76bfff700 1 -- 172.21.15.3:0/388550557 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7fa764001670 con 0x7fa77415f230 2024-05-08T21:21:52.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa76bfff700 1 -- 172.21.15.3:0/388550557 >> v1:172.21.15.3:6789/0 conn(0x7fa77415f230 legacy=0x7fa7740cc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:52.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa76bfff700 1 --2- 172.21.15.3:0/388550557 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa760003a00 0x7fa760003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:52.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa76bfff700 1 --2- 172.21.15.3:0/388550557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa760004850 0x7fa760004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:52.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa76bfff700 1 --2- 172.21.15.3:0/388550557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa760005210 0x7fa76000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:52.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa76bfff700 1 -- 172.21.15.3:0/388550557 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa774141df0 con 0x7fa760003a00 2024-05-08T21:21:52.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa76bfff700 1 -- 172.21.15.3:0/388550557 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76000c1b0 con 0x7fa760005210 2024-05-08T21:21:52.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa76bfff700 1 -- 172.21.15.3:0/388550557 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76000c370 con 0x7fa760004850 2024-05-08T21:21:52.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa7791c0700 1 --2- 172.21.15.3:0/388550557 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa760003a00 0x7fa760003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:52.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa7789bf700 1 --2- 172.21.15.3:0/388550557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa760004850 0x7fa760004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa7799c1700 1 --2- 172.21.15.3:0/388550557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa760005210 0x7fa76000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa7791c0700 1 -- 172.21.15.3:0/388550557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa760004850 msgr2=0x7fa760004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa7791c0700 1 --2- 172.21.15.3:0/388550557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa760004850 0x7fa760004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa7791c0700 1 -- 172.21.15.3:0/388550557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa760005210 msgr2=0x7fa76000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa7791c0700 1 --2- 172.21.15.3:0/388550557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa760005210 0x7fa76000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa7791c0700 1 -- 172.21.15.3:0/388550557 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7600035b0 con 0x7fa760003a00 2024-05-08T21:21:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.235+0000 7fa7799c1700 1 -- 172.21.15.3:0/388550557 reap_dead start 2024-05-08T21:21:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.236+0000 7fa7791c0700 1 --2- 172.21.15.3:0/388550557 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa760003a00 0x7fa760003e20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa7700017c0 tx=0x7fa760005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810a8c2132c3b472 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.236+0000 7fa76bfff700 1 -- 172.21.15.3:0/388550557 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7700052d0 con 0x7fa760003a00 2024-05-08T21:21:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.236+0000 7fa76bfff700 1 -- 172.21.15.3:0/388550557 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa770005450 con 0x7fa760003a00 2024-05-08T21:21:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.236+0000 7fa76bfff700 1 -- 172.21.15.3:0/388550557 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7700038c0 con 0x7fa760003a00 2024-05-08T21:21:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.236+0000 7fa77b424700 1 -- 172.21.15.3:0/388550557 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa760003a00 msgr2=0x7fa760003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.236+0000 7fa77b424700 1 --2- 172.21.15.3:0/388550557 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa760003a00 0x7fa760003e20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa7700017c0 tx=0x7fa760005880 comp rx=0 tx=0).stop 2024-05-08T21:21:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.236+0000 7fa77b424700 1 -- 172.21.15.3:0/388550557 shutdown_connections 2024-05-08T21:21:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.236+0000 7fa77b424700 1 --2- 172.21.15.3:0/388550557 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa760003a00 0x7fa760003e20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.236+0000 7fa77b424700 1 -- 172.21.15.3:0/388550557 >> 172.21.15.3:0/388550557 conn(0x7fa7740cfce0 msgr2=0x7fa7740d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:52.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.236+0000 7fa77b424700 1 -- 172.21.15.3:0/388550557 shutdown_connections 2024-05-08T21:21:52.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.236+0000 7fa77b424700 1 -- 172.21.15.3:0/388550557 wait complete. 2024-05-08T21:21:52.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.238+0000 7fa77b424700 1 Processor -- start 2024-05-08T21:21:52.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.238+0000 7fa77b424700 1 -- start start 2024-05-08T21:21:52.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.238+0000 7fa77b424700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa760003a00 0x7fa77415af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:52.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.238+0000 7fa77b424700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa774157bd0 0x7fa7741f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:52.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa77b424700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa774158400 0x7fa774158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:52.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa77b424700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa774158dc0 con 0x7fa774158400 2024-05-08T21:21:52.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa77b424700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7741f48e0 con 0x7fa774157bd0 2024-05-08T21:21:52.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa77b424700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7741f4b10 con 0x7fa760003a00 2024-05-08T21:21:52.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa7799c1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa774158400 0x7fa774158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:52.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa7791c0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa760003a00 0x7fa77415af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:52.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa7799c1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa774158400 0x7fa774158880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52148/0 (socket says 172.21.15.3:52148) 2024-05-08T21:21:52.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa7799c1700 1 -- 172.21.15.3:0/3029215257 learned_addr learned my addr 172.21.15.3:0/3029215257 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:52.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa7789bf700 1 --2- 172.21.15.3:0/3029215257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa774157bd0 0x7fa7741f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:52.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa7799c1700 1 -- 172.21.15.3:0/3029215257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa760003a00 msgr2=0x7fa77415af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:52.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa7799c1700 1 --2- 172.21.15.3:0/3029215257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa760003a00 0x7fa77415af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa7799c1700 1 -- 172.21.15.3:0/3029215257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa774157bd0 msgr2=0x7fa7741f43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:52.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa7799c1700 1 --2- 172.21.15.3:0/3029215257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa774157bd0 0x7fa7741f43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa7799c1700 1 -- 172.21.15.3:0/3029215257 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7741f4df0 con 0x7fa774158400 2024-05-08T21:21:52.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa7799c1700 1 --2- 172.21.15.3:0/3029215257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa774158400 0x7fa774158880 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa7740cc480 tx=0x7fa76c011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:52.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa769ffb700 1 -- 172.21.15.3:0/3029215257 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c00e040 con 0x7fa774158400 2024-05-08T21:21:52.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa769ffb700 1 -- 172.21.15.3:0/3029215257 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76c00f040 con 0x7fa774158400 2024-05-08T21:21:52.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa77b424700 1 -- 172.21.15.3:0/3029215257 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7741f50d0 con 0x7fa774158400 2024-05-08T21:21:52.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.239+0000 7fa769ffb700 1 -- 172.21.15.3:0/3029215257 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c00fb90 con 0x7fa774158400 2024-05-08T21:21:52.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.240+0000 7fa77b424700 1 -- 172.21.15.3:0/3029215257 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa774158050 con 0x7fa774158400 2024-05-08T21:21:52.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.241+0000 7fa77b424700 1 -- 172.21.15.3:0/3029215257 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77413d520 con 0x7fa774158400 2024-05-08T21:21:52.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.241+0000 7fa769ffb700 1 -- 172.21.15.3:0/3029215257 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa76c00fcf0 con 0x7fa774158400 2024-05-08T21:21:52.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.242+0000 7fa769ffb700 1 --2- 172.21.15.3:0/3029215257 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa75806ac90 0x7fa75806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:52.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.242+0000 7fa769ffb700 1 -- 172.21.15.3:0/3029215257 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa76c089740 con 0x7fa774158400 2024-05-08T21:21:52.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.242+0000 7fa7791c0700 1 --2- 172.21.15.3:0/3029215257 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa75806ac90 0x7fa75806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:52.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.242+0000 7fa7791c0700 1 --2- 172.21.15.3:0/3029215257 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa75806ac90 0x7fa75806d150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa770000c00 tx=0x7fa770009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:52.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.247+0000 7fa769ffb700 1 -- 172.21.15.3:0/3029215257 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fa77413d520 con 0x7fa774158400 2024-05-08T21:21:52.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.466+0000 7fa77b424700 1 -- 172.21.15.3:0/3029215257 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa77413d520 con 0x7fa774158400 2024-05-08T21:21:52.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.467+0000 7fa769ffb700 1 -- 172.21.15.3:0/3029215257 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16002 (secure 0 0 0) 0x7fa77413d520 con 0x7fa774158400 2024-05-08T21:21:52.469 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:21:52.469 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":19,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:52.062213+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:21:52.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.469+0000 7fa77b424700 1 -- 172.21.15.3:0/3029215257 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa75806ac90 msgr2=0x7fa75806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:52.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.469+0000 7fa77b424700 1 --2- 172.21.15.3:0/3029215257 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa75806ac90 0x7fa75806d150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa770000c00 tx=0x7fa770009030 comp rx=0 tx=0).stop 2024-05-08T21:21:52.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.469+0000 7fa77b424700 1 -- 172.21.15.3:0/3029215257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa774158400 msgr2=0x7fa774158880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:52.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.469+0000 7fa77b424700 1 --2- 172.21.15.3:0/3029215257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa774158400 0x7fa774158880 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa7740cc480 tx=0x7fa76c011ed0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.469+0000 7fa77b424700 1 -- 172.21.15.3:0/3029215257 shutdown_connections 2024-05-08T21:21:52.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.469+0000 7fa77b424700 1 --2- 172.21.15.3:0/3029215257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa760003a00 0x7fa77415af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.469+0000 7fa77b424700 1 --2- 172.21.15.3:0/3029215257 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa75806ac90 0x7fa75806d150 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.469+0000 7fa77b424700 1 --2- 172.21.15.3:0/3029215257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa774158400 0x7fa774158880 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.469+0000 7fa77b424700 1 --2- 172.21.15.3:0/3029215257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa774157bd0 0x7fa7741f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.469+0000 7fa77b424700 1 -- 172.21.15.3:0/3029215257 >> 172.21.15.3:0/3029215257 conn(0x7fa7740cfce0 msgr2=0x7fa7740b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:52.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.469+0000 7fa77b424700 1 -- 172.21.15.3:0/3029215257 shutdown_connections 2024-05-08T21:21:52.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.469+0000 7fa77b424700 1 -- 172.21.15.3:0/3029215257 wait complete. 2024-05-08T21:21:52.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:21:52.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.621+0000 7f74d5dcc700 1 Processor -- start 2024-05-08T21:21:52.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.621+0000 7f74d5dcc700 1 -- start start 2024-05-08T21:21:52.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.622+0000 7f74d5dcc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f74d006a960 0x7f74d006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:52.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.622+0000 7f74d5dcc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74d015cfa0 0x7f74d015d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:52.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.622+0000 7f74d5dcc700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d00b3170 con 0x7f74d00b5ba0 2024-05-08T21:21:52.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.622+0000 7f74d5dcc700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74d006b350 con 0x7f74d006a960 2024-05-08T21:21:52.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.622+0000 7f74d5dcc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d00cc5e0 con 0x7f74d015cfa0 2024-05-08T21:21:52.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.622+0000 7f74ceffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f74d006a960 0x7f74d006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:52.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.622+0000 7f74cf7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f74d00b5ba0 0x7f74d00cf8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:32922/0 (socket says 172.21.15.3:32922) 2024-05-08T21:21:52.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.622+0000 7f74ceffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f74d006a960 0x7f74d006ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52160/0 (socket says 172.21.15.3:52160) 2024-05-08T21:21:52.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.622+0000 7f74cf7fe700 1 -- 172.21.15.3:0/112926235 learned_addr learned my addr 172.21.15.3:0/112926235 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:52.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.623+0000 7f74cffff700 1 --2- 172.21.15.3:0/112926235 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74d015cfa0 0x7f74d015d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:52.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.623+0000 7f74ce7fc700 1 -- 172.21.15.3:0/112926235 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469438402 0 0) 0x7f74d00b3170 con 0x7f74d00b5ba0 2024-05-08T21:21:52.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.623+0000 7f74ce7fc700 1 -- 172.21.15.3:0/112926235 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b80035b0 con 0x7f74d00b5ba0 2024-05-08T21:21:52.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.623+0000 7f74ceffd700 1 -- 172.21.15.3:0/112926235 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74d015cfa0 msgr2=0x7f74d015d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:52.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.623+0000 7f74ceffd700 1 --2- 172.21.15.3:0/112926235 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74d015cfa0 0x7f74d015d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.623+0000 7f74ceffd700 1 -- 172.21.15.3:0/112926235 >> v1:172.21.15.3:6789/0 conn(0x7f74d00b5ba0 legacy=0x7f74d00cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:52.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.623+0000 7f74ceffd700 1 -- 172.21.15.3:0/112926235 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d00cc000 con 0x7f74d006a960 2024-05-08T21:21:52.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.623+0000 7f74ceffd700 1 --2- 172.21.15.3:0/112926235 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74d006a960 0x7f74d006ae10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009840 tx=0x7f74c4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3e2b9e984b0064c server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:52.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.623+0000 7f74ce7fc700 1 -- 172.21.15.3:0/112926235 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4011830 con 0x7f74d006a960 2024-05-08T21:21:52.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.623+0000 7f74ce7fc700 1 -- 172.21.15.3:0/112926235 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c4011990 con 0x7f74d006a960 2024-05-08T21:21:52.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.623+0000 7f74ce7fc700 1 -- 172.21.15.3:0/112926235 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4011d20 con 0x7f74d006a960 2024-05-08T21:21:52.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.624+0000 7f74d5dcc700 1 -- 172.21.15.3:0/112926235 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74d006a960 msgr2=0x7f74d006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:52.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.624+0000 7f74d5dcc700 1 --2- 172.21.15.3:0/112926235 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74d006a960 0x7f74d006ae10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009840 tx=0x7f74c4009b50 comp rx=0 tx=0).stop 2024-05-08T21:21:52.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.624+0000 7f74d5dcc700 1 -- 172.21.15.3:0/112926235 shutdown_connections 2024-05-08T21:21:52.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.624+0000 7f74d5dcc700 1 --2- 172.21.15.3:0/112926235 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74d015cfa0 0x7f74d015d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.624+0000 7f74d5dcc700 1 --2- 172.21.15.3:0/112926235 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74d006a960 0x7f74d006ae10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.624+0000 7f74d5dcc700 1 -- 172.21.15.3:0/112926235 >> 172.21.15.3:0/112926235 conn(0x7f74d0099a50 msgr2=0x7f74d00d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:52.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.624+0000 7f74d5dcc700 1 -- 172.21.15.3:0/112926235 shutdown_connections 2024-05-08T21:21:52.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.624+0000 7f74d5dcc700 1 -- 172.21.15.3:0/112926235 wait complete. 2024-05-08T21:21:52.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.625+0000 7f74d5dcc700 1 Processor -- start 2024-05-08T21:21:52.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.626+0000 7f74d5dcc700 1 -- start start 2024-05-08T21:21:52.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74d5dcc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74d006a960 0x7f74d00b4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:52.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74d5dcc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74d00b5ba0 0x7f74d00b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:52.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74d5dcc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74d015cfa0 0x7f74d015f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:52.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74d5dcc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d01591b0 con 0x7f74d015cfa0 2024-05-08T21:21:52.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74d5dcc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d0159490 con 0x7f74d006a960 2024-05-08T21:21:52.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74d5dcc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d0159770 con 0x7f74d00b5ba0 2024-05-08T21:21:52.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74cffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74d015cfa0 0x7f74d015f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:52.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74cffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74d015cfa0 0x7f74d015f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52176/0 (socket says 172.21.15.3:52176) 2024-05-08T21:21:52.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74cffff700 1 -- 172.21.15.3:0/4113882268 learned_addr learned my addr 172.21.15.3:0/4113882268 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:52.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74cf7fe700 1 --2- 172.21.15.3:0/4113882268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74d006a960 0x7f74d00b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:52.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74ceffd700 1 --2- 172.21.15.3:0/4113882268 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74d00b5ba0 0x7f74d00b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:52.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74cffff700 1 -- 172.21.15.3:0/4113882268 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74d00b5ba0 msgr2=0x7f74d00b49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:52.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74cffff700 1 --2- 172.21.15.3:0/4113882268 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74d00b5ba0 0x7f74d00b49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74cffff700 1 -- 172.21.15.3:0/4113882268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74d006a960 msgr2=0x7f74d00b4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:52.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74cffff700 1 --2- 172.21.15.3:0/4113882268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74d006a960 0x7f74d00b4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74cffff700 1 -- 172.21.15.3:0/4113882268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c40094a0 con 0x7f74d015cfa0 2024-05-08T21:21:52.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.627+0000 7f74cf7fe700 1 --2- 172.21.15.3:0/4113882268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74d006a960 0x7f74d00b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:21:52.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.628+0000 7f74cffff700 1 --2- 172.21.15.3:0/4113882268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74d015cfa0 0x7f74d015f5c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f74c8007e40 tx=0x7f74c800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:52.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.628+0000 7f74b7fff700 1 -- 172.21.15.3:0/4113882268 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c800ed30 con 0x7f74d015cfa0 2024-05-08T21:21:52.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.628+0000 7f74b7fff700 1 -- 172.21.15.3:0/4113882268 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c8014420 con 0x7f74d015cfa0 2024-05-08T21:21:52.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.628+0000 7f74d5dcc700 1 -- 172.21.15.3:0/4113882268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d0159a50 con 0x7f74d015cfa0 2024-05-08T21:21:52.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.628+0000 7f74b7fff700 1 -- 172.21.15.3:0/4113882268 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c80147d0 con 0x7f74d015cfa0 2024-05-08T21:21:52.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.628+0000 7f74d5dcc700 1 -- 172.21.15.3:0/4113882268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d015a250 con 0x7f74d015cfa0 2024-05-08T21:21:52.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.629+0000 7f74d5dcc700 1 -- 172.21.15.3:0/4113882268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d00cbce0 con 0x7f74d015cfa0 2024-05-08T21:21:52.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.629+0000 7f74b7fff700 1 -- 172.21.15.3:0/4113882268 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f74c8014930 con 0x7f74d015cfa0 2024-05-08T21:21:52.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.630+0000 7f74b7fff700 1 --2- 172.21.15.3:0/4113882268 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f74bc06ac90 0x7f74bc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:52.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.630+0000 7f74b7fff700 1 -- 172.21.15.3:0/4113882268 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74c8089b10 con 0x7f74d015cfa0 2024-05-08T21:21:52.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.630+0000 7f74cf7fe700 1 --2- 172.21.15.3:0/4113882268 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f74bc06ac90 0x7f74bc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:52.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.630+0000 7f74cf7fe700 1 --2- 172.21.15.3:0/4113882268 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f74bc06ac90 0x7f74bc06d150 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f74c0007fd0 tx=0x7f74c0007dc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:52.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.635+0000 7f74b7fff700 1 -- 172.21.15.3:0/4113882268 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f74c8054bc0 con 0x7f74d015cfa0 2024-05-08T21:21:52.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.851+0000 7f74d5dcc700 1 -- 172.21.15.3:0/4113882268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74d00ce880 con 0x7f74d015cfa0 2024-05-08T21:21:52.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.853+0000 7f74b7fff700 1 -- 172.21.15.3:0/4113882268 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16002 (secure 0 0 0) 0x7f74c80581e0 con 0x7f74d015cfa0 2024-05-08T21:21:52.854 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:21:52.855 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":19,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:52.062213+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:21:52.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.855+0000 7f74d5dcc700 1 -- 172.21.15.3:0/4113882268 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f74bc06ac90 msgr2=0x7f74bc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:52.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.855+0000 7f74d5dcc700 1 --2- 172.21.15.3:0/4113882268 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f74bc06ac90 0x7f74bc06d150 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f74c0007fd0 tx=0x7f74c0007dc0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.855+0000 7f74d5dcc700 1 -- 172.21.15.3:0/4113882268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74d015cfa0 msgr2=0x7f74d015f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:52.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.855+0000 7f74d5dcc700 1 --2- 172.21.15.3:0/4113882268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74d015cfa0 0x7f74d015f5c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f74c8007e40 tx=0x7f74c800e3c0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.855+0000 7f74d5dcc700 1 -- 172.21.15.3:0/4113882268 shutdown_connections 2024-05-08T21:21:52.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.855+0000 7f74d5dcc700 1 --2- 172.21.15.3:0/4113882268 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f74bc06ac90 0x7f74bc06d150 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.855+0000 7f74d5dcc700 1 --2- 172.21.15.3:0/4113882268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74d015cfa0 0x7f74d015f5c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.855+0000 7f74d5dcc700 1 --2- 172.21.15.3:0/4113882268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74d006a960 0x7f74d00b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.855+0000 7f74d5dcc700 1 --2- 172.21.15.3:0/4113882268 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74d00b5ba0 0x7f74d00b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:52.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.855+0000 7f74d5dcc700 1 -- 172.21.15.3:0/4113882268 >> 172.21.15.3:0/4113882268 conn(0x7f74d0099a50 msgr2=0x7f74d00cf8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:52.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.855+0000 7f74d5dcc700 1 -- 172.21.15.3:0/4113882268 shutdown_connections 2024-05-08T21:21:52.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:52.855+0000 7f74d5dcc700 1 -- 172.21.15.3:0/4113882268 wait complete. 2024-05-08T21:21:52.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-08T21:21:53.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.008+0000 7fabe1fcc700 1 Processor -- start 2024-05-08T21:21:53.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.009+0000 7fabe1fcc700 1 -- start start 2024-05-08T21:21:53.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.009+0000 7fabe1fcc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabdc15dc60 0x7fabdc0ad9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:53.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.009+0000 7fabe1fcc700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabdc0aed90 con 0x7fabdc0ae620 2024-05-08T21:21:53.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.009+0000 7fabe1fcc700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabdc06dcd0 con 0x7fabdc072170 2024-05-08T21:21:53.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.009+0000 7fabe1fcc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc06e290 con 0x7fabdc15dc60 2024-05-08T21:21:53.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.010+0000 7fabdbfff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fabdc0ae620 0x7fabdc06e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:32938/0 (socket says 172.21.15.3:32938) 2024-05-08T21:21:53.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.010+0000 7fabdbfff700 1 -- 172.21.15.3:0/3390244692 learned_addr learned my addr 172.21.15.3:0/3390244692 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:53.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.010+0000 7fabdb7fe700 1 --2- 172.21.15.3:0/3390244692 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabdc15dc60 0x7fabdc0ad9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:53.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.010+0000 7fabda7fc700 1 -- 172.21.15.3:0/3390244692 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153653216 0 0) 0x7fabd4001490 con 0x7fabdc0ae620 2024-05-08T21:21:53.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.010+0000 7fabda7fc700 1 -- 172.21.15.3:0/3390244692 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc40035b0 con 0x7fabdc0ae620 2024-05-08T21:21:53.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.010+0000 7fabda7fc700 1 -- 172.21.15.3:0/3390244692 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807599903 0 0) 0x7fabdc06dcd0 con 0x7fabdc072170 2024-05-08T21:21:53.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.010+0000 7fabda7fc700 1 -- 172.21.15.3:0/3390244692 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabd4001490 con 0x7fabdc072170 2024-05-08T21:21:53.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.010+0000 7fabdb7fe700 1 -- 172.21.15.3:0/3390244692 >> v1:172.21.15.186:6789/0 conn(0x7fabdc072170 legacy=0x7fabdc0adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:53.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.010+0000 7fabdb7fe700 1 -- 172.21.15.3:0/3390244692 >> v1:172.21.15.3:6789/0 conn(0x7fabdc0ae620 legacy=0x7fabdc06e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:53.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.010+0000 7fabdb7fe700 1 -- 172.21.15.3:0/3390244692 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabdc0b2aa0 con 0x7fabdc15dc60 2024-05-08T21:21:53.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.010+0000 7fabdb7fe700 1 --2- 172.21.15.3:0/3390244692 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabdc15dc60 0x7fabdc0ad9d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fabcc009870 tx=0x7fabcc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=138b4023e8134c1b server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:53.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.011+0000 7fabda7fc700 1 -- 172.21.15.3:0/3390244692 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc011830 con 0x7fabdc15dc60 2024-05-08T21:21:53.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.011+0000 7fabda7fc700 1 -- 172.21.15.3:0/3390244692 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabcc011990 con 0x7fabdc15dc60 2024-05-08T21:21:53.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.011+0000 7fabda7fc700 1 -- 172.21.15.3:0/3390244692 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc011d20 con 0x7fabdc15dc60 2024-05-08T21:21:53.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.011+0000 7fabe1fcc700 1 -- 172.21.15.3:0/3390244692 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabdc15dc60 msgr2=0x7fabdc0ad9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:53.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.011+0000 7fabe1fcc700 1 --2- 172.21.15.3:0/3390244692 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabdc15dc60 0x7fabdc0ad9d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fabcc009870 tx=0x7fabcc009b80 comp rx=0 tx=0).stop 2024-05-08T21:21:53.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.011+0000 7fabe1fcc700 1 -- 172.21.15.3:0/3390244692 shutdown_connections 2024-05-08T21:21:53.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.011+0000 7fabe1fcc700 1 --2- 172.21.15.3:0/3390244692 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabdc15dc60 0x7fabdc0ad9d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:53.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.011+0000 7fabe1fcc700 1 -- 172.21.15.3:0/3390244692 >> 172.21.15.3:0/3390244692 conn(0x7fabdc0b5a30 msgr2=0x7fabdc0b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:53.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.011+0000 7fabe1fcc700 1 -- 172.21.15.3:0/3390244692 shutdown_connections 2024-05-08T21:21:53.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.011+0000 7fabe1fcc700 1 -- 172.21.15.3:0/3390244692 wait complete. 2024-05-08T21:21:53.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.012+0000 7fabe1fcc700 1 Processor -- start 2024-05-08T21:21:53.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.013+0000 7fabe1fcc700 1 -- start start 2024-05-08T21:21:53.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.013+0000 7fabe1fcc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabdc072170 0x7fabdc1f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:53.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.013+0000 7fabe1fcc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabdc0ae620 0x7fabdc153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:53.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.013+0000 7fabe1fcc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabdc15dc60 0x7fabdc0b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:53.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabe1fcc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc14f790 con 0x7fabdc072170 2024-05-08T21:21:53.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabe1fcc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc1f3160 con 0x7fabdc0ae620 2024-05-08T21:21:53.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabe1fcc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc1f24e0 con 0x7fabdc15dc60 2024-05-08T21:21:53.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabdb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabdc072170 0x7fabdc1f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:53.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabdb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabdc072170 0x7fabdc1f1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52178/0 (socket says 172.21.15.3:52178) 2024-05-08T21:21:53.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabdb7fe700 1 -- 172.21.15.3:0/1136929242 learned_addr learned my addr 172.21.15.3:0/1136929242 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:53.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabdbfff700 1 --2- 172.21.15.3:0/1136929242 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabdc15dc60 0x7fabdc0b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:53.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabdaffd700 1 --2- 172.21.15.3:0/1136929242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabdc0ae620 0x7fabdc153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:53.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabdb7fe700 1 -- 172.21.15.3:0/1136929242 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabdc15dc60 msgr2=0x7fabdc0b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:53.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabdb7fe700 1 --2- 172.21.15.3:0/1136929242 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabdc15dc60 0x7fabdc0b0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:53.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabdb7fe700 1 -- 172.21.15.3:0/1136929242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabdc0ae620 msgr2=0x7fabdc153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:53.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabdb7fe700 1 --2- 172.21.15.3:0/1136929242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabdc0ae620 0x7fabdc153f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:53.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabdb7fe700 1 -- 172.21.15.3:0/1136929242 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc0094a0 con 0x7fabdc072170 2024-05-08T21:21:53.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabdaffd700 1 --2- 172.21.15.3:0/1136929242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabdc0ae620 0x7fabdc153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:21:53.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabdb7fe700 1 --2- 172.21.15.3:0/1136929242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabdc072170 0x7fabdc1f1d90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fabcc0090b0 tx=0x7fabcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:53.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabc3fff700 1 -- 172.21.15.3:0/1136929242 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc01aa70 con 0x7fabdc072170 2024-05-08T21:21:53.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabc3fff700 1 -- 172.21.15.3:0/1136929242 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabcc01abd0 con 0x7fabdc072170 2024-05-08T21:21:53.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabc3fff700 1 -- 172.21.15.3:0/1136929242 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc0243f0 con 0x7fabdc072170 2024-05-08T21:21:53.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.014+0000 7fabe1fcc700 1 -- 172.21.15.3:0/1136929242 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabdc1f27c0 con 0x7fabdc072170 2024-05-08T21:21:53.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.015+0000 7fabe1fcc700 1 -- 172.21.15.3:0/1136929242 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabdc1f2d80 con 0x7fabdc072170 2024-05-08T21:21:53.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.016+0000 7fabe1fcc700 1 -- 172.21.15.3:0/1136929242 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabdc0b4be0 con 0x7fabdc072170 2024-05-08T21:21:53.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.016+0000 7fabc3fff700 1 -- 172.21.15.3:0/1136929242 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fabcc02d450 con 0x7fabdc072170 2024-05-08T21:21:53.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.016+0000 7fabc3fff700 1 --2- 172.21.15.3:0/1136929242 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fabc806ac90 0x7fabc806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:53.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.017+0000 7fabc3fff700 1 -- 172.21.15.3:0/1136929242 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fabcc08a6c0 con 0x7fabdc072170 2024-05-08T21:21:53.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.017+0000 7fabdaffd700 1 --2- 172.21.15.3:0/1136929242 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fabc806ac90 0x7fabc806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:53.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.017+0000 7fabdaffd700 1 --2- 172.21.15.3:0/1136929242 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fabc806ac90 0x7fabc806d150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fabdc09dc20 tx=0x7fabd0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:53.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.021+0000 7fabc3fff700 1 -- 172.21.15.3:0/1136929242 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fabcc055a50 con 0x7fabdc072170 2024-05-08T21:21:53.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:53.291+0000 7fabe1fcc700 1 -- 172.21.15.3:0/1136929242 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fabdc1f32a0 con 0x7fabdc072170 2024-05-08T21:21:54.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.080+0000 7fabc3fff700 1 -- 172.21.15.3:0/1136929242 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7fabcc059070 con 0x7fabdc072170 2024-05-08T21:21:54.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.083+0000 7fabe1fcc700 1 -- 172.21.15.3:0/1136929242 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fabc806ac90 msgr2=0x7fabc806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:54.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.083+0000 7fabe1fcc700 1 --2- 172.21.15.3:0/1136929242 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fabc806ac90 0x7fabc806d150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fabdc09dc20 tx=0x7fabd0009040 comp rx=0 tx=0).stop 2024-05-08T21:21:54.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.083+0000 7fabe1fcc700 1 -- 172.21.15.3:0/1136929242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabdc072170 msgr2=0x7fabdc1f1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:54.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.083+0000 7fabe1fcc700 1 --2- 172.21.15.3:0/1136929242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabdc072170 0x7fabdc1f1d90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fabcc0090b0 tx=0x7fabcc004750 comp rx=0 tx=0).stop 2024-05-08T21:21:54.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.083+0000 7fabe1fcc700 1 -- 172.21.15.3:0/1136929242 shutdown_connections 2024-05-08T21:21:54.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.083+0000 7fabe1fcc700 1 --2- 172.21.15.3:0/1136929242 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabdc15dc60 0x7fabdc0b0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:54.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.083+0000 7fabe1fcc700 1 --2- 172.21.15.3:0/1136929242 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fabc806ac90 0x7fabc806d150 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:54.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.083+0000 7fabe1fcc700 1 --2- 172.21.15.3:0/1136929242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabdc072170 0x7fabdc1f1d90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:54.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.083+0000 7fabe1fcc700 1 --2- 172.21.15.3:0/1136929242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabdc0ae620 0x7fabdc153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:54.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.083+0000 7fabe1fcc700 1 -- 172.21.15.3:0/1136929242 >> 172.21.15.3:0/1136929242 conn(0x7fabdc0b5a30 msgr2=0x7fabdc159520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:54.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.083+0000 7fabe1fcc700 1 -- 172.21.15.3:0/1136929242 shutdown_connections 2024-05-08T21:21:54.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.083+0000 7fabe1fcc700 1 -- 172.21.15.3:0/1136929242 wait complete. 2024-05-08T21:21:54.100 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-08T21:21:54.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-08T21:21:54.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.242+0000 7fb9a46fb700 1 Processor -- start 2024-05-08T21:21:54.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.242+0000 7fb9a46fb700 1 -- start start 2024-05-08T21:21:54.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.243+0000 7fb9a46fb700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb99c0cbf50 0x7fb99c161d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:54.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.243+0000 7fb9a46fb700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb99c162240 0x7fb99c15e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:54.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.243+0000 7fb9a46fb700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb99c0cf970 con 0x7fb99c0d0f10 2024-05-08T21:21:54.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.243+0000 7fb9a46fb700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb99c0cfc50 con 0x7fb99c0cbf50 2024-05-08T21:21:54.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.243+0000 7fb9a46fb700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c0cff30 con 0x7fb99c162240 2024-05-08T21:21:54.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.243+0000 7fb9a2497700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb99c0cbf50 0x7fb99c161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:54.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.243+0000 7fb9a2c98700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb99c0d0f10 0x7fb99c06a960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:32942/0 (socket says 172.21.15.3:32942) 2024-05-08T21:21:54.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.243+0000 7fb9a2c98700 1 -- 172.21.15.3:0/241947491 learned_addr learned my addr 172.21.15.3:0/241947491 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:54.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.243+0000 7fb9a1c96700 1 --2- 172.21.15.3:0/241947491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb99c162240 0x7fb99c15e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:54.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.243+0000 7fb9a2497700 1 -- 172.21.15.3:0/241947491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb99c162240 msgr2=0x7fb99c15e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:54.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.243+0000 7fb9a2497700 1 --2- 172.21.15.3:0/241947491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb99c162240 0x7fb99c15e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:54.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.243+0000 7fb9a2497700 1 -- 172.21.15.3:0/241947491 >> v1:172.21.15.3:6789/0 conn(0x7fb99c0d0f10 legacy=0x7fb99c06a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:54.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.243+0000 7fb9a2497700 1 -- 172.21.15.3:0/241947491 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99c1617e0 con 0x7fb99c0cbf50 2024-05-08T21:21:54.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.243+0000 7fb9a1495700 1 -- 172.21.15.3:0/241947491 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072340300 0 0) 0x7fb99c0cf970 con 0x7fb99c0d0f10 2024-05-08T21:21:54.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.244+0000 7fb9a2497700 1 --2- 172.21.15.3:0/241947491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c0cbf50 0x7fb99c161d00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb998009840 tx=0x7fb998009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a518939dcca6fa90 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:54.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.244+0000 7fb9a1495700 1 -- 172.21.15.3:0/241947491 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb998011830 con 0x7fb99c0cbf50 2024-05-08T21:21:54.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.244+0000 7fb9a1495700 1 -- 172.21.15.3:0/241947491 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb998011990 con 0x7fb99c0cbf50 2024-05-08T21:21:54.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.244+0000 7fb9a1495700 1 -- 172.21.15.3:0/241947491 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb998011d20 con 0x7fb99c0cbf50 2024-05-08T21:21:54.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.244+0000 7fb9a46fb700 1 -- 172.21.15.3:0/241947491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c0cbf50 msgr2=0x7fb99c161d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:54.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.244+0000 7fb9a46fb700 1 --2- 172.21.15.3:0/241947491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c0cbf50 0x7fb99c161d00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb998009840 tx=0x7fb998009b50 comp rx=0 tx=0).stop 2024-05-08T21:21:54.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.245+0000 7fb9a46fb700 1 -- 172.21.15.3:0/241947491 shutdown_connections 2024-05-08T21:21:54.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.245+0000 7fb9a46fb700 1 --2- 172.21.15.3:0/241947491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c0cbf50 0x7fb99c161d00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:54.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.245+0000 7fb9a46fb700 1 --2- 172.21.15.3:0/241947491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb99c162240 0x7fb99c15e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:54.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.245+0000 7fb9a46fb700 1 -- 172.21.15.3:0/241947491 >> 172.21.15.3:0/241947491 conn(0x7fb99c099a50 msgr2=0x7fb99c0d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:54.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.245+0000 7fb9a46fb700 1 -- 172.21.15.3:0/241947491 shutdown_connections 2024-05-08T21:21:54.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.245+0000 7fb9a46fb700 1 -- 172.21.15.3:0/241947491 wait complete. 2024-05-08T21:21:54.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.246+0000 7fb9a46fb700 1 Processor -- start 2024-05-08T21:21:54.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.246+0000 7fb9a46fb700 1 -- start start 2024-05-08T21:21:54.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a46fb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb99c0cbf50 0x7fb99c0ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:54.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a46fb700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb99c0d0f10 0x7fb99c0b4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:54.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a46fb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c162240 0x7fb99c153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:54.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a46fb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c1607b0 con 0x7fb99c162240 2024-05-08T21:21:54.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a46fb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c161530 con 0x7fb99c0cbf50 2024-05-08T21:21:54.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a46fb700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c160a50 con 0x7fb99c0d0f10 2024-05-08T21:21:54.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a2c98700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c162240 0x7fb99c153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:54.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a2c98700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c162240 0x7fb99c153140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52192/0 (socket says 172.21.15.3:52192) 2024-05-08T21:21:54.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a2c98700 1 -- 172.21.15.3:0/720011935 learned_addr learned my addr 172.21.15.3:0/720011935 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:54.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a2497700 1 --2- 172.21.15.3:0/720011935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb99c0cbf50 0x7fb99c0ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:54.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a1c96700 1 --2- 172.21.15.3:0/720011935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb99c0d0f10 0x7fb99c0b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:54.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a2c98700 1 -- 172.21.15.3:0/720011935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb99c0d0f10 msgr2=0x7fb99c0b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:54.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a2c98700 1 --2- 172.21.15.3:0/720011935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb99c0d0f10 0x7fb99c0b4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:54.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a2c98700 1 -- 172.21.15.3:0/720011935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb99c0cbf50 msgr2=0x7fb99c0ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:54.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a2c98700 1 --2- 172.21.15.3:0/720011935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb99c0cbf50 0x7fb99c0ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:54.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a2c98700 1 -- 172.21.15.3:0/720011935 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9980094a0 con 0x7fb99c162240 2024-05-08T21:21:54.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a2c98700 1 --2- 172.21.15.3:0/720011935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c162240 0x7fb99c153140 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb99400cca0 tx=0x7fb99400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:54.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb992ffd700 1 -- 172.21.15.3:0/720011935 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99400ab70 con 0x7fb99c162240 2024-05-08T21:21:54.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb992ffd700 1 -- 172.21.15.3:0/720011935 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99400acd0 con 0x7fb99c162240 2024-05-08T21:21:54.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a46fb700 1 -- 172.21.15.3:0/720011935 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb99c06e030 con 0x7fb99c162240 2024-05-08T21:21:54.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb992ffd700 1 -- 172.21.15.3:0/720011935 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb994007310 con 0x7fb99c162240 2024-05-08T21:21:54.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.247+0000 7fb9a46fb700 1 -- 172.21.15.3:0/720011935 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb99c161010 con 0x7fb99c162240 2024-05-08T21:21:54.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.248+0000 7fb9a46fb700 1 -- 172.21.15.3:0/720011935 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb99c161230 con 0x7fb99c162240 2024-05-08T21:21:54.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.249+0000 7fb992ffd700 1 -- 172.21.15.3:0/720011935 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb994007470 con 0x7fb99c162240 2024-05-08T21:21:54.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.249+0000 7fb992ffd700 1 --2- 172.21.15.3:0/720011935 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb98006ac90 0x7fb98006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:54.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.249+0000 7fb992ffd700 1 -- 172.21.15.3:0/720011935 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb994088e00 con 0x7fb99c162240 2024-05-08T21:21:54.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.250+0000 7fb9a2497700 1 --2- 172.21.15.3:0/720011935 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb98006ac90 0x7fb98006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:54.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.250+0000 7fb9a2497700 1 --2- 172.21.15.3:0/720011935 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb98006ac90 0x7fb98006d150 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb9980090b0 tx=0x7fb998008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:54.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.254+0000 7fb992ffd700 1 -- 172.21.15.3:0/720011935 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb994054070 con 0x7fb99c162240 2024-05-08T21:21:54.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:54.523+0000 7fb9a46fb700 1 -- 172.21.15.3:0/720011935 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb99c074bc0 con 0x7fb99c162240 2024-05-08T21:21:55.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.088+0000 7fb992ffd700 1 -- 172.21.15.3:0/720011935 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb994018070 con 0x7fb99c162240 2024-05-08T21:21:55.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.091+0000 7fb9a46fb700 1 -- 172.21.15.3:0/720011935 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb98006ac90 msgr2=0x7fb98006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:55.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.091+0000 7fb9a46fb700 1 --2- 172.21.15.3:0/720011935 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb98006ac90 0x7fb98006d150 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb9980090b0 tx=0x7fb998008030 comp rx=0 tx=0).stop 2024-05-08T21:21:55.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.091+0000 7fb9a46fb700 1 -- 172.21.15.3:0/720011935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c162240 msgr2=0x7fb99c153140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:55.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.091+0000 7fb9a46fb700 1 --2- 172.21.15.3:0/720011935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c162240 0x7fb99c153140 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb99400cca0 tx=0x7fb99400a060 comp rx=0 tx=0).stop 2024-05-08T21:21:55.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.091+0000 7fb9a46fb700 1 -- 172.21.15.3:0/720011935 shutdown_connections 2024-05-08T21:21:55.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.091+0000 7fb9a46fb700 1 --2- 172.21.15.3:0/720011935 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb98006ac90 0x7fb98006d150 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.091+0000 7fb9a46fb700 1 --2- 172.21.15.3:0/720011935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c162240 0x7fb99c153140 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.091+0000 7fb9a46fb700 1 --2- 172.21.15.3:0/720011935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb99c0cbf50 0x7fb99c0ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.091+0000 7fb9a46fb700 1 --2- 172.21.15.3:0/720011935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb99c0d0f10 0x7fb99c0b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.091+0000 7fb9a46fb700 1 -- 172.21.15.3:0/720011935 >> 172.21.15.3:0/720011935 conn(0x7fb99c099a50 msgr2=0x7fb99c06a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:55.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.091+0000 7fb9a46fb700 1 -- 172.21.15.3:0/720011935 shutdown_connections 2024-05-08T21:21:55.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.091+0000 7fb9a46fb700 1 -- 172.21.15.3:0/720011935 wait complete. 2024-05-08T21:21:55.108 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-08T21:21:55.109 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:21:55.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.247+0000 7f9ba509a700 1 Processor -- start 2024-05-08T21:21:55.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.247+0000 7f9ba509a700 1 -- start start 2024-05-08T21:21:55.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.248+0000 7f9ba509a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ba00ce120 0x7f9ba006e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.248+0000 7f9ba509a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba006fbf0 con 0x7f9ba015db00 2024-05-08T21:21:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.248+0000 7f9ba509a700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba006edb0 con 0x7f9ba00cb2a0 2024-05-08T21:21:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.248+0000 7f9ba509a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba006f090 con 0x7f9ba00ce120 2024-05-08T21:21:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.248+0000 7f9b9ed9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9ba015db00 0x7f9ba006dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:32958/0 (socket says 172.21.15.3:32958) 2024-05-08T21:21:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.248+0000 7f9b9ed9d700 1 -- 172.21.15.3:0/1377803754 learned_addr learned my addr 172.21.15.3:0/1377803754 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.248+0000 7f9b9e59c700 1 --2- 172.21.15.3:0/1377803754 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ba00ce120 0x7f9ba006e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.248+0000 7f9b9dd9b700 1 -- 172.21.15.3:0/1377803754 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199071819 0 0) 0x7f9ba006fbf0 con 0x7f9ba015db00 2024-05-08T21:21:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.248+0000 7f9b9dd9b700 1 -- 172.21.15.3:0/1377803754 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b880035b0 con 0x7f9ba015db00 2024-05-08T21:21:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.248+0000 7f9b9dd9b700 1 -- 172.21.15.3:0/1377803754 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652872593 0 0) 0x7f9ba006edb0 con 0x7f9ba00cb2a0 2024-05-08T21:21:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.248+0000 7f9b9dd9b700 1 -- 172.21.15.3:0/1377803754 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ba006fbf0 con 0x7f9ba00cb2a0 2024-05-08T21:21:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9dd9b700 1 -- 172.21.15.3:0/1377803754 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 322349093 0 0) 0x7f9b880035b0 con 0x7f9ba015db00 2024-05-08T21:21:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9dd9b700 1 -- 172.21.15.3:0/1377803754 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ba00ce120 msgr2=0x7f9ba006e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9dd9b700 1 --2- 172.21.15.3:0/1377803754 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ba00ce120 0x7f9ba006e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9dd9b700 1 -- 172.21.15.3:0/1377803754 >> v1:172.21.15.186:6789/0 conn(0x7f9ba00cb2a0 legacy=0x7f9ba00cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9dd9b700 1 -- 172.21.15.3:0/1377803754 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba006f370 con 0x7f9ba015db00 2024-05-08T21:21:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9dd9b700 1 -- 172.21.15.3:0/1377803754 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f9b90001670 con 0x7f9ba015db00 2024-05-08T21:21:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9dd9b700 1 -- 172.21.15.3:0/1377803754 >> v1:172.21.15.3:6789/0 conn(0x7f9ba015db00 legacy=0x7f9ba006dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9dd9b700 1 --2- 172.21.15.3:0/1377803754 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ba00ce120 0x7f9b88003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9dd9b700 1 --2- 172.21.15.3:0/1377803754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b88004170 0x7f9b88004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9dd9b700 1 --2- 172.21.15.3:0/1377803754 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b88004d90 0x7f9b88005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9dd9b700 1 -- 172.21.15.3:0/1377803754 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba006f090 con 0x7f9b88004d90 2024-05-08T21:21:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9dd9b700 1 -- 172.21.15.3:0/1377803754 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8800bb50 con 0x7f9b88004170 2024-05-08T21:21:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9dd9b700 1 -- 172.21.15.3:0/1377803754 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8800df20 con 0x7f9ba00ce120 2024-05-08T21:21:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9f59e700 1 --2- 172.21.15.3:0/1377803754 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b88004d90 0x7f9b88005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9e59c700 1 --2- 172.21.15.3:0/1377803754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b88004170 0x7f9b88004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9ed9d700 1 --2- 172.21.15.3:0/1377803754 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ba00ce120 0x7f9b88003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9f59e700 1 -- 172.21.15.3:0/1377803754 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ba00ce120 msgr2=0x7f9b88003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9f59e700 1 --2- 172.21.15.3:0/1377803754 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ba00ce120 0x7f9b88003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9f59e700 1 -- 172.21.15.3:0/1377803754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b88004170 msgr2=0x7f9b88004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9f59e700 1 --2- 172.21.15.3:0/1377803754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b88004170 0x7f9b88004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9f59e700 1 -- 172.21.15.3:0/1377803754 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b880035b0 con 0x7f9b88004d90 2024-05-08T21:21:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9f59e700 1 --2- 172.21.15.3:0/1377803754 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b88004d90 0x7f9b88005210 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800c2f0 tx=0x7f9b88005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f611bddc356109b server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9dd9b700 1 -- 172.21.15.3:0/1377803754 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b98001f30 con 0x7f9b88004d90 2024-05-08T21:21:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.249+0000 7f9b9dd9b700 1 -- 172.21.15.3:0/1377803754 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9800ce90 con 0x7f9b88004d90 2024-05-08T21:21:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.250+0000 7f9b9dd9b700 1 -- 172.21.15.3:0/1377803754 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b98003c40 con 0x7f9b88004d90 2024-05-08T21:21:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.250+0000 7f9ba509a700 1 -- 172.21.15.3:0/1377803754 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b88004d90 msgr2=0x7f9b88005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.250+0000 7f9ba509a700 1 --2- 172.21.15.3:0/1377803754 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b88004d90 0x7f9b88005210 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800c2f0 tx=0x7f9b88005880 comp rx=0 tx=0).stop 2024-05-08T21:21:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.250+0000 7f9b9f59e700 1 -- 172.21.15.3:0/1377803754 reap_dead start 2024-05-08T21:21:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.250+0000 7f9ba509a700 1 -- 172.21.15.3:0/1377803754 shutdown_connections 2024-05-08T21:21:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.250+0000 7f9ba509a700 1 -- 172.21.15.3:0/1377803754 >> 172.21.15.3:0/1377803754 conn(0x7f9ba0099a50 msgr2=0x7f9ba00d0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.250+0000 7f9ba509a700 1 -- 172.21.15.3:0/1377803754 shutdown_connections 2024-05-08T21:21:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.250+0000 7f9ba509a700 1 -- 172.21.15.3:0/1377803754 wait complete. 2024-05-08T21:21:55.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.252+0000 7f9ba509a700 1 Processor -- start 2024-05-08T21:21:55.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.252+0000 7f9ba509a700 1 -- start start 2024-05-08T21:21:55.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9ba509a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b88004d90 0x7f9ba00c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:55.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9ba509a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ba00c3f30 0x7f9ba00cca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:55.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9ba509a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ba01ee390 0x7f9ba01ee810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:55.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9ba509a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba00afe20 con 0x7f9b88004d90 2024-05-08T21:21:55.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9ba509a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba01eed50 con 0x7f9ba00c3f30 2024-05-08T21:21:55.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9ba509a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba00b0070 con 0x7f9ba01ee390 2024-05-08T21:21:55.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9b9ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b88004d90 0x7f9ba00c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:55.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9b9ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b88004d90 0x7f9ba00c6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52222/0 (socket says 172.21.15.3:52222) 2024-05-08T21:21:55.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9b9ed9d700 1 -- 172.21.15.3:0/2568133472 learned_addr learned my addr 172.21.15.3:0/2568133472 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:55.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9b9e59c700 1 --2- 172.21.15.3:0/2568133472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ba00c3f30 0x7f9ba00cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:55.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9b9f59e700 1 --2- 172.21.15.3:0/2568133472 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ba01ee390 0x7f9ba01ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:55.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9b9ed9d700 1 -- 172.21.15.3:0/2568133472 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ba01ee390 msgr2=0x7f9ba01ee810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:55.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9b9ed9d700 1 --2- 172.21.15.3:0/2568133472 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ba01ee390 0x7f9ba01ee810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9b9ed9d700 1 -- 172.21.15.3:0/2568133472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ba00c3f30 msgr2=0x7f9ba00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:55.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9b9ed9d700 1 --2- 172.21.15.3:0/2568133472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ba00c3f30 0x7f9ba00cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9b9ed9d700 1 -- 172.21.15.3:0/2568133472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba00b0350 con 0x7f9b88004d90 2024-05-08T21:21:55.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9b9ed9d700 1 --2- 172.21.15.3:0/2568133472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b88004d90 0x7f9ba00c6640 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9b90007290 tx=0x7f9b900075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:55.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9b877fe700 1 -- 172.21.15.3:0/2568133472 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b90012560 con 0x7f9b88004d90 2024-05-08T21:21:55.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9b877fe700 1 -- 172.21.15.3:0/2568133472 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b90012700 con 0x7f9b88004d90 2024-05-08T21:21:55.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9ba509a700 1 -- 172.21.15.3:0/2568133472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba00b0630 con 0x7f9b88004d90 2024-05-08T21:21:55.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9b877fe700 1 -- 172.21.15.3:0/2568133472 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b90012ab0 con 0x7f9b88004d90 2024-05-08T21:21:55.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.253+0000 7f9ba509a700 1 -- 172.21.15.3:0/2568133472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba00c6b80 con 0x7f9b88004d90 2024-05-08T21:21:55.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.254+0000 7f9ba509a700 1 -- 172.21.15.3:0/2568133472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba00b01b0 con 0x7f9b88004d90 2024-05-08T21:21:55.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.255+0000 7f9b877fe700 1 -- 172.21.15.3:0/2568133472 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9b90012c10 con 0x7f9b88004d90 2024-05-08T21:21:55.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.255+0000 7f9b877fe700 1 --2- 172.21.15.3:0/2568133472 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9b8c06ac90 0x7f9b8c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:55.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.255+0000 7f9b877fe700 1 -- 172.21.15.3:0/2568133472 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b90089200 con 0x7f9b88004d90 2024-05-08T21:21:55.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.256+0000 7f9b9e59c700 1 --2- 172.21.15.3:0/2568133472 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9b8c06ac90 0x7f9b8c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:55.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.256+0000 7f9b9e59c700 1 --2- 172.21.15.3:0/2568133472 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9b8c06ac90 0x7f9b8c06d150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9ba00c6f50 tx=0x7f9b94009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:55.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.260+0000 7f9b877fe700 1 -- 172.21.15.3:0/2568133472 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9ba00b01b0 con 0x7f9b88004d90 2024-05-08T21:21:55.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.529+0000 7f9ba509a700 1 -- 172.21.15.3:0/2568133472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ba00b0b50 con 0x7f9b88004d90 2024-05-08T21:21:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.529+0000 7f9b877fe700 1 -- 172.21.15.3:0/2568133472 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f9ba00b0b50 con 0x7f9b88004d90 2024-05-08T21:21:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:21:55.532 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/2839854293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2839854293},{"type":"v1","addr":"172.21.15.3:6834","nonce":2839854293}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1878326567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":1878326567},{"type":"v1","addr":"172.21.15.3:6835","nonce":1878326567}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/3624849830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3624849830},{"type":"v1","addr":"172.21.15.186:6836","nonce":3624849830}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:21:52.062548+0000","modified":"2024-05-08T21:21:55.079806+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6837/1462886680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1462886680},{"type":"v1","addr":"172.21.15.186:6837","nonce":1462886680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-05-08T21:21:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.532+0000 7f9ba509a700 1 -- 172.21.15.3:0/2568133472 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9b8c06ac90 msgr2=0x7f9b8c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.532+0000 7f9ba509a700 1 --2- 172.21.15.3:0/2568133472 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9b8c06ac90 0x7f9b8c06d150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9ba00c6f50 tx=0x7f9b94009210 comp rx=0 tx=0).stop 2024-05-08T21:21:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.532+0000 7f9ba509a700 1 -- 172.21.15.3:0/2568133472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b88004d90 msgr2=0x7f9ba00c6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.532+0000 7f9ba509a700 1 --2- 172.21.15.3:0/2568133472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b88004d90 0x7f9ba00c6640 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9b90007290 tx=0x7f9b900075a0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.532+0000 7f9ba509a700 1 -- 172.21.15.3:0/2568133472 shutdown_connections 2024-05-08T21:21:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.532+0000 7f9ba509a700 1 --2- 172.21.15.3:0/2568133472 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ba01ee390 0x7f9ba01ee810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.532+0000 7f9ba509a700 1 --2- 172.21.15.3:0/2568133472 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9b8c06ac90 0x7f9b8c06d150 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.532+0000 7f9ba509a700 1 --2- 172.21.15.3:0/2568133472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b88004d90 0x7f9ba00c6640 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.532+0000 7f9ba509a700 1 --2- 172.21.15.3:0/2568133472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ba00c3f30 0x7f9ba00cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.532+0000 7f9ba509a700 1 -- 172.21.15.3:0/2568133472 >> 172.21.15.3:0/2568133472 conn(0x7f9ba0099a50 msgr2=0x7f9ba014fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.532+0000 7f9ba509a700 1 -- 172.21.15.3:0/2568133472 shutdown_connections 2024-05-08T21:21:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.532+0000 7f9ba509a700 1 -- 172.21.15.3:0/2568133472 wait complete. 2024-05-08T21:21:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 8 2024-05-08T21:21:55.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:21:55.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.684+0000 7fa48db25700 1 Processor -- start 2024-05-08T21:21:55.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.684+0000 7fa48db25700 1 -- start start 2024-05-08T21:21:55.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.685+0000 7fa48db25700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa48815da10 0x7fa4880d0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:55.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.685+0000 7fa48db25700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa48806c3d0 0x7fa48806c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:55.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.685+0000 7fa48db25700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4880cf990 con 0x7fa48806bd80 2024-05-08T21:21:55.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.685+0000 7fa48db25700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa48809ab40 con 0x7fa48806c3d0 2024-05-08T21:21:55.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.685+0000 7fa48db25700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4880d04b0 con 0x7fa48815da10 2024-05-08T21:21:55.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.685+0000 7fa487fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa48806c3d0 0x7fa48806c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:55.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.686+0000 7fa487fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa48806c3d0 0x7fa48806c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52226/0 (socket says 172.21.15.3:52226) 2024-05-08T21:21:55.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.686+0000 7fa486ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa48806bd80 0x7fa48809a430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:32966/0 (socket says 172.21.15.3:32966) 2024-05-08T21:21:55.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.686+0000 7fa487fff700 1 -- 172.21.15.3:0/1198416738 learned_addr learned my addr 172.21.15.3:0/1198416738 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.686+0000 7fa4877fe700 1 --2- 172.21.15.3:0/1198416738 >> v2:172.21.15.186:3300/0 conn(0x7fa48815da10 0x7fa4880d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.686+0000 7fa487fff700 1 -- 172.21.15.3:0/1198416738 >> v2:172.21.15.186:3300/0 conn(0x7fa48815da10 msgr2=0x7fa4880d0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.686+0000 7fa487fff700 1 --2- 172.21.15.3:0/1198416738 >> v2:172.21.15.186:3300/0 conn(0x7fa48815da10 0x7fa4880d0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.686+0000 7fa487fff700 1 -- 172.21.15.3:0/1198416738 >> v1:172.21.15.3:6789/0 conn(0x7fa48806bd80 legacy=0x7fa48809a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.686+0000 7fa487fff700 1 -- 172.21.15.3:0/1198416738 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48806cd60 con 0x7fa48806c3d0 2024-05-08T21:21:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.686+0000 7fa4867fc700 1 -- 172.21.15.3:0/1198416738 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887547873 0 0) 0x7fa4880cf990 con 0x7fa48806bd80 2024-05-08T21:21:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.686+0000 7fa487fff700 1 --2- 172.21.15.3:0/1198416738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa48806c3d0 0x7fa48806c820 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa488067c90 tx=0x7fa48000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=765eaf632b205eed server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.686+0000 7fa4867fc700 1 -- 172.21.15.3:0/1198416738 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48000e070 con 0x7fa48806c3d0 2024-05-08T21:21:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.686+0000 7fa4867fc700 1 -- 172.21.15.3:0/1198416738 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48000b9c0 con 0x7fa48806c3d0 2024-05-08T21:21:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.686+0000 7fa4867fc700 1 -- 172.21.15.3:0/1198416738 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48000bd70 con 0x7fa48806c3d0 2024-05-08T21:21:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.687+0000 7fa48db25700 1 -- 172.21.15.3:0/1198416738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa48806c3d0 msgr2=0x7fa48806c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.687+0000 7fa48db25700 1 --2- 172.21.15.3:0/1198416738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa48806c3d0 0x7fa48806c820 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa488067c90 tx=0x7fa48000b330 comp rx=0 tx=0).stop 2024-05-08T21:21:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.687+0000 7fa48db25700 1 -- 172.21.15.3:0/1198416738 shutdown_connections 2024-05-08T21:21:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.687+0000 7fa48db25700 1 --2- 172.21.15.3:0/1198416738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa48806c3d0 0x7fa48806c820 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.687+0000 7fa48db25700 1 --2- 172.21.15.3:0/1198416738 >> v2:172.21.15.186:3300/0 conn(0x7fa48815da10 0x7fa4880d0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.687+0000 7fa48db25700 1 -- 172.21.15.3:0/1198416738 >> 172.21.15.3:0/1198416738 conn(0x7fa4880b3010 msgr2=0x7fa4880b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:55.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.687+0000 7fa48db25700 1 -- 172.21.15.3:0/1198416738 shutdown_connections 2024-05-08T21:21:55.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.687+0000 7fa48db25700 1 -- 172.21.15.3:0/1198416738 wait complete. 2024-05-08T21:21:55.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.688+0000 7fa48db25700 1 Processor -- start 2024-05-08T21:21:55.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.688+0000 7fa48db25700 1 -- start start 2024-05-08T21:21:55.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.689+0000 7fa48db25700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa48806bd80 0x7fa4880b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:55.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.689+0000 7fa48db25700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa48806c3d0 0x7fa4880ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:55.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.689+0000 7fa48db25700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa48815da10 0x7fa4880cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.689+0000 7fa48db25700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48814df70 con 0x7fa48806bd80 2024-05-08T21:21:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.689+0000 7fa48db25700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48814e250 con 0x7fa48815da10 2024-05-08T21:21:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.689+0000 7fa48db25700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4880c6920 con 0x7fa48806c3d0 2024-05-08T21:21:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.689+0000 7fa4877fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa48806bd80 0x7fa4880b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.689+0000 7fa4877fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa48806bd80 0x7fa4880b6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52236/0 (socket says 172.21.15.3:52236) 2024-05-08T21:21:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.689+0000 7fa4877fe700 1 -- 172.21.15.3:0/2522137272 learned_addr learned my addr 172.21.15.3:0/2522137272 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.689+0000 7fa487fff700 1 --2- 172.21.15.3:0/2522137272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa48815da10 0x7fa4880cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.689+0000 7fa486ffd700 1 --2- 172.21.15.3:0/2522137272 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa48806c3d0 0x7fa4880ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.690+0000 7fa4877fe700 1 -- 172.21.15.3:0/2522137272 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa48806c3d0 msgr2=0x7fa4880ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.690+0000 7fa4877fe700 1 --2- 172.21.15.3:0/2522137272 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa48806c3d0 0x7fa4880ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.690+0000 7fa4877fe700 1 -- 172.21.15.3:0/2522137272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa48815da10 msgr2=0x7fa4880cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.690+0000 7fa4877fe700 1 --2- 172.21.15.3:0/2522137272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa48815da10 0x7fa4880cd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.690+0000 7fa4877fe700 1 -- 172.21.15.3:0/2522137272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa480009b30 con 0x7fa48806bd80 2024-05-08T21:21:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.690+0000 7fa487fff700 1 --2- 172.21.15.3:0/2522137272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa48815da10 0x7fa4880cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:21:55.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.690+0000 7fa4877fe700 1 --2- 172.21.15.3:0/2522137272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa48806bd80 0x7fa4880b6a30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa47800e810 tx=0x7fa47800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:55.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.690+0000 7fa46ffff700 1 -- 172.21.15.3:0/2522137272 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa478013610 con 0x7fa48806bd80 2024-05-08T21:21:55.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.690+0000 7fa46ffff700 1 -- 172.21.15.3:0/2522137272 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa478013770 con 0x7fa48806bd80 2024-05-08T21:21:55.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.690+0000 7fa48db25700 1 -- 172.21.15.3:0/2522137272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4880c6c00 con 0x7fa48806bd80 2024-05-08T21:21:55.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.690+0000 7fa46ffff700 1 -- 172.21.15.3:0/2522137272 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa478013b20 con 0x7fa48806bd80 2024-05-08T21:21:55.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.690+0000 7fa48db25700 1 -- 172.21.15.3:0/2522137272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa48814b5e0 con 0x7fa48806bd80 2024-05-08T21:21:55.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.691+0000 7fa48db25700 1 -- 172.21.15.3:0/2522137272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4880c6f70 con 0x7fa48806bd80 2024-05-08T21:21:55.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.692+0000 7fa46ffff700 1 -- 172.21.15.3:0/2522137272 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa478013c80 con 0x7fa48806bd80 2024-05-08T21:21:55.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.692+0000 7fa46ffff700 1 --2- 172.21.15.3:0/2522137272 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa47406ac90 0x7fa47406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:55.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.692+0000 7fa46ffff700 1 -- 172.21.15.3:0/2522137272 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa478088fc0 con 0x7fa48806bd80 2024-05-08T21:21:55.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.692+0000 7fa486ffd700 1 --2- 172.21.15.3:0/2522137272 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa47406ac90 0x7fa47406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:55.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.693+0000 7fa486ffd700 1 --2- 172.21.15.3:0/2522137272 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa47406ac90 0x7fa47406d150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa4880cd940 tx=0x7fa47c007c90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:55.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.697+0000 7fa46ffff700 1 -- 172.21.15.3:0/2522137272 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fa4780541b0 con 0x7fa48806bd80 2024-05-08T21:21:55.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.912+0000 7fa48db25700 1 -- 172.21.15.3:0/2522137272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4880b0930 con 0x7fa48806bd80 2024-05-08T21:21:55.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.914+0000 7fa46ffff700 1 -- 172.21.15.3:0/2522137272 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15989 (secure 0 0 0) 0x7fa4780577d0 con 0x7fa48806bd80 2024-05-08T21:21:55.916 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:21:55.916 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:53.064342+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:21:55.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.916+0000 7fa48db25700 1 -- 172.21.15.3:0/2522137272 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa47406ac90 msgr2=0x7fa47406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:55.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.916+0000 7fa48db25700 1 --2- 172.21.15.3:0/2522137272 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa47406ac90 0x7fa47406d150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa4880cd940 tx=0x7fa47c007c90 comp rx=0 tx=0).stop 2024-05-08T21:21:55.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.916+0000 7fa48db25700 1 -- 172.21.15.3:0/2522137272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa48806bd80 msgr2=0x7fa4880b6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:55.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.916+0000 7fa48db25700 1 --2- 172.21.15.3:0/2522137272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa48806bd80 0x7fa4880b6a30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa47800e810 tx=0x7fa47800ebd0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.916+0000 7fa48db25700 1 -- 172.21.15.3:0/2522137272 shutdown_connections 2024-05-08T21:21:55.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.916+0000 7fa48db25700 1 --2- 172.21.15.3:0/2522137272 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa47406ac90 0x7fa47406d150 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.916+0000 7fa48db25700 1 --2- 172.21.15.3:0/2522137272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa48815da10 0x7fa4880cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.916+0000 7fa48db25700 1 --2- 172.21.15.3:0/2522137272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa48806bd80 0x7fa4880b6a30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.916+0000 7fa48db25700 1 --2- 172.21.15.3:0/2522137272 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa48806c3d0 0x7fa4880ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:55.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.916+0000 7fa48db25700 1 -- 172.21.15.3:0/2522137272 >> 172.21.15.3:0/2522137272 conn(0x7fa4880b3010 msgr2=0x7fa488099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:55.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.916+0000 7fa48db25700 1 -- 172.21.15.3:0/2522137272 shutdown_connections 2024-05-08T21:21:55.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:55.916+0000 7fa48db25700 1 -- 172.21.15.3:0/2522137272 wait complete. 2024-05-08T21:21:55.932 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-08T21:21:55.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T21:21:56.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.066+0000 7f2e6c8ba700 1 Processor -- start 2024-05-08T21:21:56.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.066+0000 7f2e6c8ba700 1 -- start start 2024-05-08T21:21:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.067+0000 7f2e6c8ba700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e680cc590 0x7f2e6806f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.067+0000 7f2e6c8ba700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e680cb160 con 0x7f2e6815db00 2024-05-08T21:21:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.067+0000 7f2e6c8ba700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6806ad20 con 0x7f2e6806f7c0 2024-05-08T21:21:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.067+0000 7f2e6c8ba700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6806fc50 con 0x7f2e680cc590 2024-05-08T21:21:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.067+0000 7f2e6659c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2e6815db00 0x7f2e680caa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:32974/0 (socket says 172.21.15.3:32974) 2024-05-08T21:21:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.067+0000 7f2e6659c700 1 -- 172.21.15.3:0/2769549542 learned_addr learned my addr 172.21.15.3:0/2769549542 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.067+0000 7f2e65d9b700 1 --2- 172.21.15.3:0/2769549542 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e680cc590 0x7f2e6806f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.067+0000 7f2e6559a700 1 -- 172.21.15.3:0/2769549542 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274271012 0 0) 0x7f2e680cb160 con 0x7f2e6815db00 2024-05-08T21:21:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.067+0000 7f2e6559a700 1 -- 172.21.15.3:0/2769549542 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e540035b0 con 0x7f2e6815db00 2024-05-08T21:21:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.067+0000 7f2e6559a700 1 -- 172.21.15.3:0/2769549542 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236827314 0 0) 0x7f2e6806ad20 con 0x7f2e6806f7c0 2024-05-08T21:21:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.067+0000 7f2e6559a700 1 -- 172.21.15.3:0/2769549542 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e680cb160 con 0x7f2e6806f7c0 2024-05-08T21:21:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.068+0000 7f2e65d9b700 1 -- 172.21.15.3:0/2769549542 >> v1:172.21.15.186:6789/0 conn(0x7f2e6806f7c0 legacy=0x7f2e6806e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.068+0000 7f2e65d9b700 1 -- 172.21.15.3:0/2769549542 >> v1:172.21.15.3:6789/0 conn(0x7f2e6815db00 legacy=0x7f2e680caa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.068+0000 7f2e65d9b700 1 -- 172.21.15.3:0/2769549542 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6806b000 con 0x7f2e680cc590 2024-05-08T21:21:56.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.068+0000 7f2e65d9b700 1 --2- 172.21.15.3:0/2769549542 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e680cc590 0x7f2e6806f280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2e58009870 tx=0x7f2e58009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f7ba7cd809e6480 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:56.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.068+0000 7f2e6559a700 1 -- 172.21.15.3:0/2769549542 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58011830 con 0x7f2e680cc590 2024-05-08T21:21:56.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.068+0000 7f2e6559a700 1 -- 172.21.15.3:0/2769549542 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e58011990 con 0x7f2e680cc590 2024-05-08T21:21:56.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.068+0000 7f2e6559a700 1 -- 172.21.15.3:0/2769549542 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58011d20 con 0x7f2e680cc590 2024-05-08T21:21:56.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.069+0000 7f2e6c8ba700 1 -- 172.21.15.3:0/2769549542 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e680cc590 msgr2=0x7f2e6806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.069+0000 7f2e6c8ba700 1 --2- 172.21.15.3:0/2769549542 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e680cc590 0x7f2e6806f280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2e58009870 tx=0x7f2e58009b80 comp rx=0 tx=0).stop 2024-05-08T21:21:56.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.069+0000 7f2e6c8ba700 1 -- 172.21.15.3:0/2769549542 shutdown_connections 2024-05-08T21:21:56.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.069+0000 7f2e6c8ba700 1 --2- 172.21.15.3:0/2769549542 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e680cc590 0x7f2e6806f280 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.069+0000 7f2e6c8ba700 1 -- 172.21.15.3:0/2769549542 >> 172.21.15.3:0/2769549542 conn(0x7f2e68099a50 msgr2=0x7f2e680d0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:56.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.069+0000 7f2e6c8ba700 1 -- 172.21.15.3:0/2769549542 shutdown_connections 2024-05-08T21:21:56.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.069+0000 7f2e6c8ba700 1 -- 172.21.15.3:0/2769549542 wait complete. 2024-05-08T21:21:56.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.070+0000 7f2e6c8ba700 1 Processor -- start 2024-05-08T21:21:56.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.070+0000 7f2e6c8ba700 1 -- start start 2024-05-08T21:21:56.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.071+0000 7f2e6c8ba700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e6806f7c0 0x7f2e680b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:56.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.071+0000 7f2e6c8ba700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e680cc590 0x7f2e68160780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:56.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.071+0000 7f2e6c8ba700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e6815db00 0x7f2e681f6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:56.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.071+0000 7f2e6c8ba700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6815f4c0 con 0x7f2e6815db00 2024-05-08T21:21:56.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.071+0000 7f2e6c8ba700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6815e420 con 0x7f2e6806f7c0 2024-05-08T21:21:56.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.071+0000 7f2e6c8ba700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6815e700 con 0x7f2e680cc590 2024-05-08T21:21:56.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.071+0000 7f2e66d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e6815db00 0x7f2e681f6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:56.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.071+0000 7f2e66d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e6815db00 0x7f2e681f6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52244/0 (socket says 172.21.15.3:52244) 2024-05-08T21:21:56.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.071+0000 7f2e66d9d700 1 -- 172.21.15.3:0/943909697 learned_addr learned my addr 172.21.15.3:0/943909697 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:56.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.071+0000 7f2e6659c700 1 --2- 172.21.15.3:0/943909697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e6806f7c0 0x7f2e680b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:56.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.071+0000 7f2e65d9b700 1 --2- 172.21.15.3:0/943909697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e680cc590 0x7f2e68160780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:56.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.072+0000 7f2e66d9d700 1 -- 172.21.15.3:0/943909697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e680cc590 msgr2=0x7f2e68160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.072+0000 7f2e66d9d700 1 --2- 172.21.15.3:0/943909697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e680cc590 0x7f2e68160780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.072+0000 7f2e66d9d700 1 -- 172.21.15.3:0/943909697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e6806f7c0 msgr2=0x7f2e680b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.072+0000 7f2e66d9d700 1 --2- 172.21.15.3:0/943909697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e6806f7c0 0x7f2e680b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.072+0000 7f2e66d9d700 1 -- 172.21.15.3:0/943909697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e580094a0 con 0x7f2e6815db00 2024-05-08T21:21:56.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.072+0000 7f2e66d9d700 1 --2- 172.21.15.3:0/943909697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e6815db00 0x7f2e681f6880 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000cc20 tx=0x7f2e6000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:56.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.072+0000 7f2e5effd700 1 -- 172.21.15.3:0/943909697 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60003c00 con 0x7f2e6815db00 2024-05-08T21:21:56.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.072+0000 7f2e5effd700 1 -- 172.21.15.3:0/943909697 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e60003d60 con 0x7f2e6815db00 2024-05-08T21:21:56.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.072+0000 7f2e6c8ba700 1 -- 172.21.15.3:0/943909697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6815e9e0 con 0x7f2e6815db00 2024-05-08T21:21:56.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.072+0000 7f2e5effd700 1 -- 172.21.15.3:0/943909697 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6000aae0 con 0x7f2e6815db00 2024-05-08T21:21:56.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.072+0000 7f2e6c8ba700 1 -- 172.21.15.3:0/943909697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e6815efa0 con 0x7f2e6815db00 2024-05-08T21:21:56.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.073+0000 7f2e6c8ba700 1 -- 172.21.15.3:0/943909697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e68068000 con 0x7f2e6815db00 2024-05-08T21:21:56.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.074+0000 7f2e5effd700 1 -- 172.21.15.3:0/943909697 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2e6000ac40 con 0x7f2e6815db00 2024-05-08T21:21:56.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.074+0000 7f2e5effd700 1 --2- 172.21.15.3:0/943909697 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2e4806ac90 0x7f2e4806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:56.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.074+0000 7f2e5effd700 1 -- 172.21.15.3:0/943909697 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e600893b0 con 0x7f2e6815db00 2024-05-08T21:21:56.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.075+0000 7f2e6659c700 1 --2- 172.21.15.3:0/943909697 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2e4806ac90 0x7f2e4806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:56.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.075+0000 7f2e6659c700 1 --2- 172.21.15.3:0/943909697 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2e4806ac90 0x7f2e4806d150 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2e50007fd0 tx=0x7f2e50007dc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:56.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.079+0000 7f2e5effd700 1 -- 172.21.15.3:0/943909697 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2e60054590 con 0x7f2e6815db00 2024-05-08T21:21:56.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.293+0000 7f2e6c8ba700 1 -- 172.21.15.3:0/943909697 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2e6806de00 con 0x7f2e4806ac90 2024-05-08T21:21:56.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.301+0000 7f2e5effd700 1 -- 172.21.15.3:0/943909697 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+253923 (secure 0 0 0) 0x7f2e500058e0 con 0x7f2e4806ac90 2024-05-08T21:21:56.306 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:21:56.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.304+0000 7f2e6c8ba700 1 -- 172.21.15.3:0/943909697 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2e4806ac90 msgr2=0x7f2e4806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.304+0000 7f2e6c8ba700 1 --2- 172.21.15.3:0/943909697 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2e4806ac90 0x7f2e4806d150 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2e50007fd0 tx=0x7f2e50007dc0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.304+0000 7f2e6c8ba700 1 -- 172.21.15.3:0/943909697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e6815db00 msgr2=0x7f2e681f6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.304+0000 7f2e6c8ba700 1 --2- 172.21.15.3:0/943909697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e6815db00 0x7f2e681f6880 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000cc20 tx=0x7f2e6000cf30 comp rx=0 tx=0).stop 2024-05-08T21:21:56.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.304+0000 7f2e6c8ba700 1 -- 172.21.15.3:0/943909697 shutdown_connections 2024-05-08T21:21:56.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.304+0000 7f2e6c8ba700 1 --2- 172.21.15.3:0/943909697 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2e4806ac90 0x7f2e4806d150 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.304+0000 7f2e6c8ba700 1 --2- 172.21.15.3:0/943909697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e6815db00 0x7f2e681f6880 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.304+0000 7f2e6c8ba700 1 --2- 172.21.15.3:0/943909697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e6806f7c0 0x7f2e680b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.304+0000 7f2e6c8ba700 1 --2- 172.21.15.3:0/943909697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e680cc590 0x7f2e68160780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.305+0000 7f2e6c8ba700 1 -- 172.21.15.3:0/943909697 >> 172.21.15.3:0/943909697 conn(0x7f2e68099a50 msgr2=0x7f2e680caa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:56.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.305+0000 7f2e6c8ba700 1 -- 172.21.15.3:0/943909697 shutdown_connections 2024-05-08T21:21:56.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.305+0000 7f2e6c8ba700 1 -- 172.21.15.3:0/943909697 wait complete. 2024-05-08T21:21:56.309 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-08T21:21:56.320 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-05-08T21:21:54.531507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219176,"kb_used_data":2376,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755544,"statfs":{"total":773094113280,"available":772869677056,"internally_reserved":0,"allocated":2433024,"data_stored":1546584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000942"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.597209+0000","last_change":"2024-05-08T21:21:50.597209+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.597209+0000","last_undegraded":"2024-05-08T21:21:50.597209+0000","last_fullsized":"2024-05-08T21:21:50.597209+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:00:55.530393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.595397+0000","last_change":"2024-05-08T21:21:50.595397+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.595397+0000","last_undegraded":"2024-05-08T21:21:50.595397+0000","last_fullsized":"2024-05-08T21:21:50.595397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:10:09.131458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.598276+0000","last_change":"2024-05-08T21:21:50.598276+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.598276+0000","last_undegraded":"2024-05-08T21:21:50.598276+0000","last_fullsized":"2024-05-08T21:21:50.598276+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:13.563899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.596797+0000","last_change":"2024-05-08T21:21:50.596797+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.596797+0000","last_undegraded":"2024-05-08T21:21:50.596797+0000","last_fullsized":"2024-05-08T21:21:50.596797+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:05:44.822440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.594861+0000","last_change":"2024-05-08T21:21:50.594861+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.594861+0000","last_undegraded":"2024-05-08T21:21:50.594861+0000","last_fullsized":"2024-05-08T21:21:50.594861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:05.185930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.597979+0000","last_change":"2024-05-08T21:21:50.597979+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.597979+0000","last_undegraded":"2024-05-08T21:21:50.597979+0000","last_fullsized":"2024-05-08T21:21:50.597979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:38:17.091010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.595478+0000","last_change":"2024-05-08T21:21:50.595478+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.595478+0000","last_undegraded":"2024-05-08T21:21:50.595478+0000","last_fullsized":"2024-05-08T21:21:50.595478+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:20:59.795219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.600967+0000","last_change":"2024-05-08T21:21:50.600967+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.600967+0000","last_undegraded":"2024-05-08T21:21:50.600967+0000","last_fullsized":"2024-05-08T21:21:50.600967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:41.009726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.599227+0000","last_change":"2024-05-08T21:21:50.599227+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.599227+0000","last_undegraded":"2024-05-08T21:21:50.599227+0000","last_fullsized":"2024-05-08T21:21:50.599227+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:13.563899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.600117+0000","last_change":"2024-05-08T21:21:50.600117+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.600117+0000","last_undegraded":"2024-05-08T21:21:50.600117+0000","last_fullsized":"2024-05-08T21:21:50.600117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:38:17.091010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.601196+0000","last_change":"2024-05-08T21:21:50.601196+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.601196+0000","last_undegraded":"2024-05-08T21:21:50.601196+0000","last_fullsized":"2024-05-08T21:21:50.601196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:37:21.706277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.597958+0000","last_change":"2024-05-08T21:21:50.597958+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.597958+0000","last_undegraded":"2024-05-08T21:21:50.597958+0000","last_fullsized":"2024-05-08T21:21:50.597958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:00:55.530393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.597139+0000","last_change":"2024-05-08T21:21:50.597139+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.597139+0000","last_undegraded":"2024-05-08T21:21:50.597139+0000","last_fullsized":"2024-05-08T21:21:50.597139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:20:59.795219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.600711+0000","last_change":"2024-05-08T21:21:50.600711+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.600711+0000","last_undegraded":"2024-05-08T21:21:50.600711+0000","last_fullsized":"2024-05-08T21:21:50.600711+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:45:34.421641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.598120+0000","last_change":"2024-05-08T21:21:50.598120+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.598120+0000","last_undegraded":"2024-05-08T21:21:50.598120+0000","last_fullsized":"2024-05-08T21:21:50.598120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:45:34.421641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.603218+0000","last_change":"2024-05-08T21:21:50.603218+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.603218+0000","last_undegraded":"2024-05-08T21:21:50.603218+0000","last_fullsized":"2024-05-08T21:21:50.603218+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:41.009726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.594439+0000","last_change":"2024-05-08T21:21:50.594439+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.594439+0000","last_undegraded":"2024-05-08T21:21:50.594439+0000","last_fullsized":"2024-05-08T21:21:50.594439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:10:09.131458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.600433+0000","last_change":"2024-05-08T21:21:50.600433+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.600433+0000","last_undegraded":"2024-05-08T21:21:50.600433+0000","last_fullsized":"2024-05-08T21:21:50.600433+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:44:53.582574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.599694+0000","last_change":"2024-05-08T21:21:50.599694+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.599694+0000","last_undegraded":"2024-05-08T21:21:50.599694+0000","last_fullsized":"2024-05-08T21:21:50.599694+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:44:53.582574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.603231+0000","last_change":"2024-05-08T21:21:50.603231+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.603231+0000","last_undegraded":"2024-05-08T21:21:50.603231+0000","last_fullsized":"2024-05-08T21:21:50.603231+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:38:17.091010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.600312+0000","last_change":"2024-05-08T21:21:50.600312+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.600312+0000","last_undegraded":"2024-05-08T21:21:50.600312+0000","last_fullsized":"2024-05-08T21:21:50.600312+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:13.563899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":161,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-08T21:21:45.622742+0000","last_change":"2024-05-08T21:21:42.333097+0000","last_active":"2024-05-08T21:21:45.622742+0000","last_peered":"2024-05-08T21:21:45.622742+0000","last_clean":"2024-05-08T21:21:45.622742+0000","last_became_active":"2024-05-08T21:21:42.332033+0000","last_became_peered":"2024-05-08T21:21:42.332033+0000","last_unstale":"2024-05-08T21:21:45.622742+0000","last_undegraded":"2024-05-08T21:21:45.622742+0000","last_fullsized":"2024-05-08T21:21:45.622742+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_clean_scrub_stamp":"2024-05-08T21:21:40.862622+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:39:24.772551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.596616+0000","last_change":"2024-05-08T21:21:50.596616+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.596616+0000","last_undegraded":"2024-05-08T21:21:50.596616+0000","last_fullsized":"2024-05-08T21:21:50.596616+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:14:36.375522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.596745+0000","last_change":"2024-05-08T21:21:50.596745+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.596745+0000","last_undegraded":"2024-05-08T21:21:50.596745+0000","last_fullsized":"2024-05-08T21:21:50.596745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:44:53.582574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.600583+0000","last_change":"2024-05-08T21:21:50.600583+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.600583+0000","last_undegraded":"2024-05-08T21:21:50.600583+0000","last_fullsized":"2024-05-08T21:21:50.600583+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:14:36.375522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.598152+0000","last_change":"2024-05-08T21:21:50.598152+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.598152+0000","last_undegraded":"2024-05-08T21:21:50.598152+0000","last_fullsized":"2024-05-08T21:21:50.598152+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:44:53.582574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.597582+0000","last_change":"2024-05-08T21:21:50.597582+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.597582+0000","last_undegraded":"2024-05-08T21:21:50.597582+0000","last_fullsized":"2024-05-08T21:21:50.597582+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:37:21.706277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.600420+0000","last_change":"2024-05-08T21:21:50.600420+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.600420+0000","last_undegraded":"2024-05-08T21:21:50.600420+0000","last_fullsized":"2024-05-08T21:21:50.600420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:05:44.822440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.600398+0000","last_change":"2024-05-08T21:21:50.600398+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.600398+0000","last_undegraded":"2024-05-08T21:21:50.600398+0000","last_fullsized":"2024-05-08T21:21:50.600398+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:13.563899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.596771+0000","last_change":"2024-05-08T21:21:50.596771+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.596771+0000","last_undegraded":"2024-05-08T21:21:50.596771+0000","last_fullsized":"2024-05-08T21:21:50.596771+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:38:17.091010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.603177+0000","last_change":"2024-05-08T21:21:50.603177+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.603177+0000","last_undegraded":"2024-05-08T21:21:50.603177+0000","last_fullsized":"2024-05-08T21:21:50.603177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:00:55.530393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.600333+0000","last_change":"2024-05-08T21:21:50.600333+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.600333+0000","last_undegraded":"2024-05-08T21:21:50.600333+0000","last_fullsized":"2024-05-08T21:21:50.600333+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:41.009726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-08T21:21:50.603473+0000","last_change":"2024-05-08T21:21:50.603473+0000","last_active":"2024-05-08T21:21:50.586265+0000","last_peered":"2024-05-08T21:21:50.586265+0000","last_clean":"2024-05-08T21:21:50.586265+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:50.603473+0000","last_undegraded":"2024-05-08T21:21:50.603473+0000","last_fullsized":"2024-05-08T21:21:50.603473+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:45:34.421641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-08T21:21:51.591355+0000","last_change":"2024-05-08T21:21:51.591355+0000","last_active":"2024-05-08T21:21:51.591355+0000","last_peered":"2024-05-08T21:21:51.591355+0000","last_clean":"2024-05-08T21:21:51.591355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-08T21:21:51.591355+0000","last_undegraded":"2024-05-08T21:21:51.591355+0000","last_fullsized":"2024-05-08T21:21:51.591355+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28372,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343468,"statfs":{"total":96636764160,"available":96607711232,"internally_reserved":0,"allocated":749568,"data_stored":636099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":45731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":45731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":749568,"data_stored":636099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":45731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":45731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":45731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":45731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T21:21:56.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T21:21:56.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.498+0000 7f84e253f700 1 Processor -- start 2024-05-08T21:21:56.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.498+0000 7f84e253f700 1 -- start start 2024-05-08T21:21:56.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.498+0000 7f84e253f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f84dc0cd7c0 0x7f84dc099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:56.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.498+0000 7f84e253f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84dc09ac00 con 0x7f84dc15b810 2024-05-08T21:21:56.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.498+0000 7f84e253f700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84dc0cc5d0 con 0x7f84dc0cf270 2024-05-08T21:21:56.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.498+0000 7f84e253f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84dc0cf700 con 0x7f84dc0cd7c0 2024-05-08T21:21:56.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.499+0000 7f84e0adc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f84dc0cd7c0 0x7f84dc099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:56.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.499+0000 7f84dbfff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f84dc15b810 0x7f84dc06ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:32980/0 (socket says 172.21.15.3:32980) 2024-05-08T21:21:56.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.499+0000 7f84dbfff700 1 -- 172.21.15.3:0/3947208178 learned_addr learned my addr 172.21.15.3:0/3947208178 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:56.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.499+0000 7f84e0adc700 1 -- 172.21.15.3:0/3947208178 >> v1:172.21.15.186:6789/0 conn(0x7f84dc0cf270 legacy=0x7f84dc0cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.499+0000 7f84e0adc700 1 -- 172.21.15.3:0/3947208178 >> v1:172.21.15.3:6789/0 conn(0x7f84dc15b810 legacy=0x7f84dc06ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.499+0000 7f84e0adc700 1 -- 172.21.15.3:0/3947208178 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc0cc8b0 con 0x7f84dc0cd7c0 2024-05-08T21:21:56.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.499+0000 7f84daffd700 1 -- 172.21.15.3:0/3947208178 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915877697 0 0) 0x7f84dc09ac00 con 0x7f84dc15b810 2024-05-08T21:21:56.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.499+0000 7f84e0adc700 1 --2- 172.21.15.3:0/3947208178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84dc0cd7c0 0x7f84dc099a50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f84d4009d90 tx=0x7f84d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94ab55ab265fa878 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:56.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.499+0000 7f84daffd700 1 -- 172.21.15.3:0/3947208178 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d400e070 con 0x7f84dc0cd7c0 2024-05-08T21:21:56.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.499+0000 7f84daffd700 1 -- 172.21.15.3:0/3947208178 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d400bbd0 con 0x7f84dc0cd7c0 2024-05-08T21:21:56.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.499+0000 7f84daffd700 1 -- 172.21.15.3:0/3947208178 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4012430 con 0x7f84dc0cd7c0 2024-05-08T21:21:56.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.500+0000 7f84e253f700 1 -- 172.21.15.3:0/3947208178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84dc0cd7c0 msgr2=0x7f84dc099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.500+0000 7f84e253f700 1 --2- 172.21.15.3:0/3947208178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84dc0cd7c0 0x7f84dc099a50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f84d4009d90 tx=0x7f84d400b330 comp rx=0 tx=0).stop 2024-05-08T21:21:56.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.500+0000 7f84e253f700 1 -- 172.21.15.3:0/3947208178 shutdown_connections 2024-05-08T21:21:56.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.500+0000 7f84e253f700 1 --2- 172.21.15.3:0/3947208178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84dc0cd7c0 0x7f84dc099a50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.500+0000 7f84e253f700 1 -- 172.21.15.3:0/3947208178 >> 172.21.15.3:0/3947208178 conn(0x7f84dc0b2f80 msgr2=0x7f84dc0b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:56.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.500+0000 7f84e253f700 1 -- 172.21.15.3:0/3947208178 shutdown_connections 2024-05-08T21:21:56.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.500+0000 7f84e253f700 1 -- 172.21.15.3:0/3947208178 wait complete. 2024-05-08T21:21:56.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.501+0000 7f84e253f700 1 Processor -- start 2024-05-08T21:21:56.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.502+0000 7f84e253f700 1 -- start start 2024-05-08T21:21:56.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.502+0000 7f84e253f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84dc0cd7c0 0x7f84dc0c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:56.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.502+0000 7f84e253f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc0cf270 0x7f84dc156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:56.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.502+0000 7f84e253f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84dc15b810 0x7f84dc1f78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:56.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.502+0000 7f84e253f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc0c99d0 con 0x7f84dc15b810 2024-05-08T21:21:56.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.502+0000 7f84e253f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc1f74b0 con 0x7f84dc0cd7c0 2024-05-08T21:21:56.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.502+0000 7f84e253f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc0c8ae0 con 0x7f84dc0cf270 2024-05-08T21:21:56.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.502+0000 7f84e0adc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84dc15b810 0x7f84dc1f78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:56.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.502+0000 7f84e0adc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84dc15b810 0x7f84dc1f78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52262/0 (socket says 172.21.15.3:52262) 2024-05-08T21:21:56.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.503+0000 7f84e0adc700 1 -- 172.21.15.3:0/315726915 learned_addr learned my addr 172.21.15.3:0/315726915 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:56.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.503+0000 7f84dbfff700 1 --2- 172.21.15.3:0/315726915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84dc0cd7c0 0x7f84dc0c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:56.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.503+0000 7f84db7fe700 1 --2- 172.21.15.3:0/315726915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc0cf270 0x7f84dc156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:56.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.503+0000 7f84e0adc700 1 -- 172.21.15.3:0/315726915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc0cf270 msgr2=0x7f84dc156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.503+0000 7f84e0adc700 1 --2- 172.21.15.3:0/315726915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc0cf270 0x7f84dc156650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.503+0000 7f84e0adc700 1 -- 172.21.15.3:0/315726915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84dc0cd7c0 msgr2=0x7f84dc0c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.503+0000 7f84e0adc700 1 --2- 172.21.15.3:0/315726915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84dc0cd7c0 0x7f84dc0c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.503+0000 7f84e0adc700 1 -- 172.21.15.3:0/315726915 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d4009a70 con 0x7f84dc15b810 2024-05-08T21:21:56.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.503+0000 7f84e0adc700 1 --2- 172.21.15.3:0/315726915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84dc15b810 0x7f84dc1f78d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f84d4014040 tx=0x7f84d4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:56.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.503+0000 7f84d8ff9700 1 -- 172.21.15.3:0/315726915 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d401b810 con 0x7f84dc15b810 2024-05-08T21:21:56.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.503+0000 7f84d8ff9700 1 -- 172.21.15.3:0/315726915 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d4020070 con 0x7f84dc15b810 2024-05-08T21:21:56.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.503+0000 7f84e253f700 1 -- 172.21.15.3:0/315726915 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84dc1f4320 con 0x7f84dc15b810 2024-05-08T21:21:56.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.503+0000 7f84d8ff9700 1 -- 172.21.15.3:0/315726915 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d401bbb0 con 0x7f84dc15b810 2024-05-08T21:21:56.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.503+0000 7f84e253f700 1 -- 172.21.15.3:0/315726915 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84dc1f4bc0 con 0x7f84dc15b810 2024-05-08T21:21:56.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.504+0000 7f84e253f700 1 -- 172.21.15.3:0/315726915 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84dc1f75f0 con 0x7f84dc15b810 2024-05-08T21:21:56.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.505+0000 7f84d8ff9700 1 -- 172.21.15.3:0/315726915 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f84d401bd10 con 0x7f84dc15b810 2024-05-08T21:21:56.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.505+0000 7f84d8ff9700 1 --2- 172.21.15.3:0/315726915 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f84c806ac90 0x7f84c806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:56.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.505+0000 7f84d8ff9700 1 -- 172.21.15.3:0/315726915 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84d4022070 con 0x7f84dc15b810 2024-05-08T21:21:56.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.505+0000 7f84dbfff700 1 --2- 172.21.15.3:0/315726915 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f84c806ac90 0x7f84c806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:56.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.506+0000 7f84dbfff700 1 --2- 172.21.15.3:0/315726915 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f84c806ac90 0x7f84c806d150 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f84cc000f60 tx=0x7f84cc004a50 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:56.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.510+0000 7f84d8ff9700 1 -- 172.21.15.3:0/315726915 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f84dc1f75f0 con 0x7f84dc15b810 2024-05-08T21:21:56.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.726+0000 7f84e253f700 1 -- 172.21.15.3:0/315726915 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f84dc0cc280 con 0x7f84c806ac90 2024-05-08T21:21:56.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.734+0000 7f84d8ff9700 1 -- 172.21.15.3:0/315726915 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264668 (secure 0 0 0) 0x7f84cc001280 con 0x7f84c806ac90 2024-05-08T21:21:56.739 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:21:56.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.737+0000 7f84e253f700 1 -- 172.21.15.3:0/315726915 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f84c806ac90 msgr2=0x7f84c806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.737+0000 7f84e253f700 1 --2- 172.21.15.3:0/315726915 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f84c806ac90 0x7f84c806d150 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f84cc000f60 tx=0x7f84cc004a50 comp rx=0 tx=0).stop 2024-05-08T21:21:56.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.737+0000 7f84e253f700 1 -- 172.21.15.3:0/315726915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84dc15b810 msgr2=0x7f84dc1f78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.737+0000 7f84e253f700 1 --2- 172.21.15.3:0/315726915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84dc15b810 0x7f84dc1f78d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f84d4014040 tx=0x7f84d4004ad0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.738+0000 7f84e253f700 1 -- 172.21.15.3:0/315726915 shutdown_connections 2024-05-08T21:21:56.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.738+0000 7f84e253f700 1 --2- 172.21.15.3:0/315726915 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f84c806ac90 0x7f84c806d150 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.738+0000 7f84e253f700 1 --2- 172.21.15.3:0/315726915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84dc15b810 0x7f84dc1f78d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.738+0000 7f84e253f700 1 --2- 172.21.15.3:0/315726915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84dc0cd7c0 0x7f84dc0c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.738+0000 7f84e253f700 1 --2- 172.21.15.3:0/315726915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc0cf270 0x7f84dc156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.738+0000 7f84e253f700 1 -- 172.21.15.3:0/315726915 >> 172.21.15.3:0/315726915 conn(0x7f84dc0b2f80 msgr2=0x7f84dc0ccfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:56.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.738+0000 7f84e253f700 1 -- 172.21.15.3:0/315726915 shutdown_connections 2024-05-08T21:21:56.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.738+0000 7f84e253f700 1 -- 172.21.15.3:0/315726915 wait complete. 2024-05-08T21:21:56.742 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-08T21:21:56.753 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-05-08T21:21:56.531887+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218964,"kb_used_data":2764,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755756,"statfs":{"total":773094113280,"available":772869894144,"internally_reserved":0,"allocated":2830336,"data_stored":1721144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.939556"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604264+0000","last_change":"2024-05-08T21:21:52.093899+0000","last_active":"2024-05-08T21:21:55.604264+0000","last_peered":"2024-05-08T21:21:55.604264+0000","last_clean":"2024-05-08T21:21:55.604264+0000","last_became_active":"2024-05-08T21:21:52.093609+0000","last_became_peered":"2024-05-08T21:21:52.093609+0000","last_unstale":"2024-05-08T21:21:55.604264+0000","last_undegraded":"2024-05-08T21:21:55.604264+0000","last_fullsized":"2024-05-08T21:21:55.604264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:22:03.106073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073419+0000","last_change":"2024-05-08T21:21:52.074817+0000","last_active":"2024-05-08T21:21:53.073419+0000","last_peered":"2024-05-08T21:21:53.073419+0000","last_clean":"2024-05-08T21:21:53.073419+0000","last_became_active":"2024-05-08T21:21:52.074544+0000","last_became_peered":"2024-05-08T21:21:52.074544+0000","last_unstale":"2024-05-08T21:21:53.073419+0000","last_undegraded":"2024-05-08T21:21:53.073419+0000","last_fullsized":"2024-05-08T21:21:53.073419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:22:38.788235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603137+0000","last_change":"2024-05-08T21:21:52.082843+0000","last_active":"2024-05-08T21:21:55.603137+0000","last_peered":"2024-05-08T21:21:55.603137+0000","last_clean":"2024-05-08T21:21:55.603137+0000","last_became_active":"2024-05-08T21:21:52.080398+0000","last_became_peered":"2024-05-08T21:21:52.080398+0000","last_unstale":"2024-05-08T21:21:55.603137+0000","last_undegraded":"2024-05-08T21:21:55.603137+0000","last_fullsized":"2024-05-08T21:21:55.603137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:36:48.359422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603875+0000","last_change":"2024-05-08T21:21:52.086712+0000","last_active":"2024-05-08T21:21:55.603875+0000","last_peered":"2024-05-08T21:21:55.603875+0000","last_clean":"2024-05-08T21:21:55.603875+0000","last_became_active":"2024-05-08T21:21:52.086445+0000","last_became_peered":"2024-05-08T21:21:52.086445+0000","last_unstale":"2024-05-08T21:21:55.603875+0000","last_undegraded":"2024-05-08T21:21:55.603875+0000","last_fullsized":"2024-05-08T21:21:55.603875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:22:38.788235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603913+0000","last_change":"2024-05-08T21:21:52.085542+0000","last_active":"2024-05-08T21:21:55.603913+0000","last_peered":"2024-05-08T21:21:55.603913+0000","last_clean":"2024-05-08T21:21:55.603913+0000","last_became_active":"2024-05-08T21:21:52.085253+0000","last_became_peered":"2024-05-08T21:21:52.085253+0000","last_unstale":"2024-05-08T21:21:55.603913+0000","last_undegraded":"2024-05-08T21:21:55.603913+0000","last_fullsized":"2024-05-08T21:21:55.603913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:54:03.930846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602797+0000","last_change":"2024-05-08T21:21:52.090302+0000","last_active":"2024-05-08T21:21:55.602797+0000","last_peered":"2024-05-08T21:21:55.602797+0000","last_clean":"2024-05-08T21:21:55.602797+0000","last_became_active":"2024-05-08T21:21:52.088764+0000","last_became_peered":"2024-05-08T21:21:52.088764+0000","last_unstale":"2024-05-08T21:21:55.602797+0000","last_undegraded":"2024-05-08T21:21:55.602797+0000","last_fullsized":"2024-05-08T21:21:55.602797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:09:59.496154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603367+0000","last_change":"2024-05-08T21:21:52.095147+0000","last_active":"2024-05-08T21:21:55.603367+0000","last_peered":"2024-05-08T21:21:55.603367+0000","last_clean":"2024-05-08T21:21:55.603367+0000","last_became_active":"2024-05-08T21:21:52.094954+0000","last_became_peered":"2024-05-08T21:21:52.094954+0000","last_unstale":"2024-05-08T21:21:55.603367+0000","last_undegraded":"2024-05-08T21:21:55.603367+0000","last_fullsized":"2024-05-08T21:21:55.603367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:09:59.496154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076360+0000","last_change":"2024-05-08T21:21:52.083974+0000","last_active":"2024-05-08T21:21:53.076360+0000","last_peered":"2024-05-08T21:21:53.076360+0000","last_clean":"2024-05-08T21:21:53.076360+0000","last_became_active":"2024-05-08T21:21:52.083702+0000","last_became_peered":"2024-05-08T21:21:52.083702+0000","last_unstale":"2024-05-08T21:21:53.076360+0000","last_undegraded":"2024-05-08T21:21:53.076360+0000","last_fullsized":"2024-05-08T21:21:53.076360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:54:03.930846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604376+0000","last_change":"2024-05-08T21:21:52.087758+0000","last_active":"2024-05-08T21:21:55.604376+0000","last_peered":"2024-05-08T21:21:55.604376+0000","last_clean":"2024-05-08T21:21:55.604376+0000","last_became_active":"2024-05-08T21:21:52.087489+0000","last_became_peered":"2024-05-08T21:21:52.087489+0000","last_unstale":"2024-05-08T21:21:55.604376+0000","last_undegraded":"2024-05-08T21:21:55.604376+0000","last_fullsized":"2024-05-08T21:21:55.604376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:08.050201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603328+0000","last_change":"2024-05-08T21:21:52.092265+0000","last_active":"2024-05-08T21:21:55.603328+0000","last_peered":"2024-05-08T21:21:55.603328+0000","last_clean":"2024-05-08T21:21:55.603328+0000","last_became_active":"2024-05-08T21:21:52.091813+0000","last_became_peered":"2024-05-08T21:21:52.091813+0000","last_unstale":"2024-05-08T21:21:55.603328+0000","last_undegraded":"2024-05-08T21:21:55.603328+0000","last_fullsized":"2024-05-08T21:21:55.603328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:04:56.104718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602830+0000","last_change":"2024-05-08T21:21:52.087586+0000","last_active":"2024-05-08T21:21:55.602830+0000","last_peered":"2024-05-08T21:21:55.602830+0000","last_clean":"2024-05-08T21:21:55.602830+0000","last_became_active":"2024-05-08T21:21:52.087302+0000","last_became_peered":"2024-05-08T21:21:52.087302+0000","last_unstale":"2024-05-08T21:21:55.602830+0000","last_undegraded":"2024-05-08T21:21:55.602830+0000","last_fullsized":"2024-05-08T21:21:55.602830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:41:45.740288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076636+0000","last_change":"2024-05-08T21:21:52.094218+0000","last_active":"2024-05-08T21:21:53.076636+0000","last_peered":"2024-05-08T21:21:53.076636+0000","last_clean":"2024-05-08T21:21:53.076636+0000","last_became_active":"2024-05-08T21:21:52.093855+0000","last_became_peered":"2024-05-08T21:21:52.093855+0000","last_unstale":"2024-05-08T21:21:53.076636+0000","last_undegraded":"2024-05-08T21:21:53.076636+0000","last_fullsized":"2024-05-08T21:21:53.076636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:18:36.173550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076649+0000","last_change":"2024-05-08T21:21:52.093685+0000","last_active":"2024-05-08T21:21:53.076649+0000","last_peered":"2024-05-08T21:21:53.076649+0000","last_clean":"2024-05-08T21:21:53.076649+0000","last_became_active":"2024-05-08T21:21:52.093380+0000","last_became_peered":"2024-05-08T21:21:52.093380+0000","last_unstale":"2024-05-08T21:21:53.076649+0000","last_undegraded":"2024-05-08T21:21:53.076649+0000","last_fullsized":"2024-05-08T21:21:53.076649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:39:11.021535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603501+0000","last_change":"2024-05-08T21:21:52.077438+0000","last_active":"2024-05-08T21:21:55.603501+0000","last_peered":"2024-05-08T21:21:55.603501+0000","last_clean":"2024-05-08T21:21:55.603501+0000","last_became_active":"2024-05-08T21:21:52.077129+0000","last_became_peered":"2024-05-08T21:21:52.077129+0000","last_unstale":"2024-05-08T21:21:55.603501+0000","last_undegraded":"2024-05-08T21:21:55.603501+0000","last_fullsized":"2024-05-08T21:21:55.603501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:28:07.575847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073038+0000","last_change":"2024-05-08T21:21:52.096939+0000","last_active":"2024-05-08T21:21:53.073038+0000","last_peered":"2024-05-08T21:21:53.073038+0000","last_clean":"2024-05-08T21:21:53.073038+0000","last_became_active":"2024-05-08T21:21:52.096629+0000","last_became_peered":"2024-05-08T21:21:52.096629+0000","last_unstale":"2024-05-08T21:21:53.073038+0000","last_undegraded":"2024-05-08T21:21:53.073038+0000","last_fullsized":"2024-05-08T21:21:53.073038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:04:56.104718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603067+0000","last_change":"2024-05-08T21:21:52.088031+0000","last_active":"2024-05-08T21:21:55.603067+0000","last_peered":"2024-05-08T21:21:55.603067+0000","last_clean":"2024-05-08T21:21:55.603067+0000","last_became_active":"2024-05-08T21:21:52.087748+0000","last_became_peered":"2024-05-08T21:21:52.087748+0000","last_unstale":"2024-05-08T21:21:55.603067+0000","last_undegraded":"2024-05-08T21:21:55.603067+0000","last_fullsized":"2024-05-08T21:21:55.603067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:46:26.988461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604380+0000","last_change":"2024-05-08T21:21:52.096438+0000","last_active":"2024-05-08T21:21:55.604380+0000","last_peered":"2024-05-08T21:21:55.604380+0000","last_clean":"2024-05-08T21:21:55.604380+0000","last_became_active":"2024-05-08T21:21:52.095936+0000","last_became_peered":"2024-05-08T21:21:52.095936+0000","last_unstale":"2024-05-08T21:21:55.604380+0000","last_undegraded":"2024-05-08T21:21:55.604380+0000","last_fullsized":"2024-05-08T21:21:55.604380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:38:46.924064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603006+0000","last_change":"2024-05-08T21:21:52.095114+0000","last_active":"2024-05-08T21:21:55.603006+0000","last_peered":"2024-05-08T21:21:55.603006+0000","last_clean":"2024-05-08T21:21:55.603006+0000","last_became_active":"2024-05-08T21:21:52.094795+0000","last_became_peered":"2024-05-08T21:21:52.094795+0000","last_unstale":"2024-05-08T21:21:55.603006+0000","last_undegraded":"2024-05-08T21:21:55.603006+0000","last_fullsized":"2024-05-08T21:21:55.603006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:14:37.380612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603538+0000","last_change":"2024-05-08T21:21:52.084672+0000","last_active":"2024-05-08T21:21:55.603538+0000","last_peered":"2024-05-08T21:21:55.603538+0000","last_clean":"2024-05-08T21:21:55.603538+0000","last_became_active":"2024-05-08T21:21:52.084373+0000","last_became_peered":"2024-05-08T21:21:52.084373+0000","last_unstale":"2024-05-08T21:21:55.603538+0000","last_undegraded":"2024-05-08T21:21:55.603538+0000","last_fullsized":"2024-05-08T21:21:55.603538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:10:50.287200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.095184+0000","last_change":"2024-05-08T21:21:51.614469+0000","last_active":"2024-05-08T21:21:52.095184+0000","last_peered":"2024-05-08T21:21:52.095184+0000","last_clean":"2024-05-08T21:21:52.095184+0000","last_became_active":"2024-05-08T21:21:51.614173+0000","last_became_peered":"2024-05-08T21:21:51.614173+0000","last_unstale":"2024-05-08T21:21:52.095184+0000","last_undegraded":"2024-05-08T21:21:52.095184+0000","last_fullsized":"2024-05-08T21:21:52.095184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:00:55.530393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076753+0000","last_change":"2024-05-08T21:21:52.095181+0000","last_active":"2024-05-08T21:21:53.076753+0000","last_peered":"2024-05-08T21:21:53.076753+0000","last_clean":"2024-05-08T21:21:53.076753+0000","last_became_active":"2024-05-08T21:21:52.095007+0000","last_became_peered":"2024-05-08T21:21:52.095007+0000","last_unstale":"2024-05-08T21:21:53.076753+0000","last_undegraded":"2024-05-08T21:21:53.076753+0000","last_fullsized":"2024-05-08T21:21:53.076753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:28:07.575847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.091195+0000","last_change":"2024-05-08T21:21:51.615481+0000","last_active":"2024-05-08T21:21:52.091195+0000","last_peered":"2024-05-08T21:21:52.091195+0000","last_clean":"2024-05-08T21:21:52.091195+0000","last_became_active":"2024-05-08T21:21:51.614841+0000","last_became_peered":"2024-05-08T21:21:51.614841+0000","last_unstale":"2024-05-08T21:21:52.091195+0000","last_undegraded":"2024-05-08T21:21:52.091195+0000","last_fullsized":"2024-05-08T21:21:52.091195+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:10:09.131458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603223+0000","last_change":"2024-05-08T21:21:52.083798+0000","last_active":"2024-05-08T21:21:55.603223+0000","last_peered":"2024-05-08T21:21:55.603223+0000","last_clean":"2024-05-08T21:21:55.603223+0000","last_became_active":"2024-05-08T21:21:52.083361+0000","last_became_peered":"2024-05-08T21:21:52.083361+0000","last_unstale":"2024-05-08T21:21:55.603223+0000","last_undegraded":"2024-05-08T21:21:55.603223+0000","last_fullsized":"2024-05-08T21:21:55.603223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:38.782318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.094224+0000","last_change":"2024-05-08T21:21:51.610838+0000","last_active":"2024-05-08T21:21:52.094224+0000","last_peered":"2024-05-08T21:21:52.094224+0000","last_clean":"2024-05-08T21:21:52.094224+0000","last_became_active":"2024-05-08T21:21:51.610168+0000","last_became_peered":"2024-05-08T21:21:51.610168+0000","last_unstale":"2024-05-08T21:21:52.094224+0000","last_undegraded":"2024-05-08T21:21:52.094224+0000","last_fullsized":"2024-05-08T21:21:52.094224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:13.563899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602928+0000","last_change":"2024-05-08T21:21:52.101000+0000","last_active":"2024-05-08T21:21:55.602928+0000","last_peered":"2024-05-08T21:21:55.602928+0000","last_clean":"2024-05-08T21:21:55.602928+0000","last_became_active":"2024-05-08T21:21:52.100589+0000","last_became_peered":"2024-05-08T21:21:52.100589+0000","last_unstale":"2024-05-08T21:21:55.602928+0000","last_undegraded":"2024-05-08T21:21:55.602928+0000","last_fullsized":"2024-05-08T21:21:55.602928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:22:03.106073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.094794+0000","last_change":"2024-05-08T21:21:51.617528+0000","last_active":"2024-05-08T21:21:52.094794+0000","last_peered":"2024-05-08T21:21:52.094794+0000","last_clean":"2024-05-08T21:21:52.094794+0000","last_became_active":"2024-05-08T21:21:51.617225+0000","last_became_peered":"2024-05-08T21:21:51.617225+0000","last_unstale":"2024-05-08T21:21:52.094794+0000","last_undegraded":"2024-05-08T21:21:52.094794+0000","last_fullsized":"2024-05-08T21:21:52.094794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:05:44.822440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604076+0000","last_change":"2024-05-08T21:21:52.085423+0000","last_active":"2024-05-08T21:21:55.604076+0000","last_peered":"2024-05-08T21:21:55.604076+0000","last_clean":"2024-05-08T21:21:55.604076+0000","last_became_active":"2024-05-08T21:21:52.085144+0000","last_became_peered":"2024-05-08T21:21:52.085144+0000","last_unstale":"2024-05-08T21:21:55.604076+0000","last_undegraded":"2024-05-08T21:21:55.604076+0000","last_fullsized":"2024-05-08T21:21:55.604076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:54:03.930846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.092454+0000","last_change":"2024-05-08T21:21:51.611668+0000","last_active":"2024-05-08T21:21:52.092454+0000","last_peered":"2024-05-08T21:21:52.092454+0000","last_clean":"2024-05-08T21:21:52.092454+0000","last_became_active":"2024-05-08T21:21:51.611363+0000","last_became_peered":"2024-05-08T21:21:51.611363+0000","last_unstale":"2024-05-08T21:21:52.092454+0000","last_undegraded":"2024-05-08T21:21:52.092454+0000","last_fullsized":"2024-05-08T21:21:52.092454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:05.185930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076192+0000","last_change":"2024-05-08T21:21:52.092056+0000","last_active":"2024-05-08T21:21:53.076192+0000","last_peered":"2024-05-08T21:21:53.076192+0000","last_clean":"2024-05-08T21:21:53.076192+0000","last_became_active":"2024-05-08T21:21:52.086766+0000","last_became_peered":"2024-05-08T21:21:52.086766+0000","last_unstale":"2024-05-08T21:21:53.076192+0000","last_undegraded":"2024-05-08T21:21:53.076192+0000","last_fullsized":"2024-05-08T21:21:53.076192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:05:45.827530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.101711+0000","last_change":"2024-05-08T21:21:51.609576+0000","last_active":"2024-05-08T21:21:52.101711+0000","last_peered":"2024-05-08T21:21:52.101711+0000","last_clean":"2024-05-08T21:21:52.101711+0000","last_became_active":"2024-05-08T21:21:51.609004+0000","last_became_peered":"2024-05-08T21:21:51.609004+0000","last_unstale":"2024-05-08T21:21:52.101711+0000","last_undegraded":"2024-05-08T21:21:52.101711+0000","last_fullsized":"2024-05-08T21:21:52.101711+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:38:17.091010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603223+0000","last_change":"2024-05-08T21:21:52.093264+0000","last_active":"2024-05-08T21:21:55.603223+0000","last_peered":"2024-05-08T21:21:55.603223+0000","last_clean":"2024-05-08T21:21:55.603223+0000","last_became_active":"2024-05-08T21:21:52.092898+0000","last_became_peered":"2024-05-08T21:21:52.092898+0000","last_unstale":"2024-05-08T21:21:55.603223+0000","last_undegraded":"2024-05-08T21:21:55.603223+0000","last_fullsized":"2024-05-08T21:21:55.603223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:41:45.740288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073983+0000","last_change":"2024-05-08T21:21:51.609633+0000","last_active":"2024-05-08T21:21:52.073983+0000","last_peered":"2024-05-08T21:21:52.073983+0000","last_clean":"2024-05-08T21:21:52.073983+0000","last_became_active":"2024-05-08T21:21:51.609304+0000","last_became_peered":"2024-05-08T21:21:51.609304+0000","last_unstale":"2024-05-08T21:21:52.073983+0000","last_undegraded":"2024-05-08T21:21:52.073983+0000","last_fullsized":"2024-05-08T21:21:52.073983+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:20:59.795219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076482+0000","last_change":"2024-05-08T21:21:52.078368+0000","last_active":"2024-05-08T21:21:53.076482+0000","last_peered":"2024-05-08T21:21:53.076482+0000","last_clean":"2024-05-08T21:21:53.076482+0000","last_became_active":"2024-05-08T21:21:52.078095+0000","last_became_peered":"2024-05-08T21:21:52.078095+0000","last_unstale":"2024-05-08T21:21:53.076482+0000","last_undegraded":"2024-05-08T21:21:53.076482+0000","last_fullsized":"2024-05-08T21:21:53.076482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:38.782318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073269+0000","last_change":"2024-05-08T21:21:51.612841+0000","last_active":"2024-05-08T21:21:52.073269+0000","last_peered":"2024-05-08T21:21:52.073269+0000","last_clean":"2024-05-08T21:21:52.073269+0000","last_became_active":"2024-05-08T21:21:51.612563+0000","last_became_peered":"2024-05-08T21:21:51.612563+0000","last_unstale":"2024-05-08T21:21:52.073269+0000","last_undegraded":"2024-05-08T21:21:52.073269+0000","last_fullsized":"2024-05-08T21:21:52.073269+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:41.009726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073158+0000","last_change":"2024-05-08T21:21:52.088955+0000","last_active":"2024-05-08T21:21:53.073158+0000","last_peered":"2024-05-08T21:21:53.073158+0000","last_clean":"2024-05-08T21:21:53.073158+0000","last_became_active":"2024-05-08T21:21:52.088686+0000","last_became_peered":"2024-05-08T21:21:52.088686+0000","last_unstale":"2024-05-08T21:21:53.073158+0000","last_undegraded":"2024-05-08T21:21:53.073158+0000","last_fullsized":"2024-05-08T21:21:53.073158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:38.782318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.075045+0000","last_change":"2024-05-08T21:21:51.620459+0000","last_active":"2024-05-08T21:21:52.075045+0000","last_peered":"2024-05-08T21:21:52.075045+0000","last_clean":"2024-05-08T21:21:52.075045+0000","last_became_active":"2024-05-08T21:21:51.620147+0000","last_became_peered":"2024-05-08T21:21:51.620147+0000","last_unstale":"2024-05-08T21:21:52.075045+0000","last_undegraded":"2024-05-08T21:21:52.075045+0000","last_fullsized":"2024-05-08T21:21:52.075045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:13.563899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604099+0000","last_change":"2024-05-08T21:21:52.096162+0000","last_active":"2024-05-08T21:21:55.604099+0000","last_peered":"2024-05-08T21:21:55.604099+0000","last_clean":"2024-05-08T21:21:55.604099+0000","last_became_active":"2024-05-08T21:21:52.095548+0000","last_became_peered":"2024-05-08T21:21:52.095548+0000","last_unstale":"2024-05-08T21:21:55.604099+0000","last_undegraded":"2024-05-08T21:21:55.604099+0000","last_fullsized":"2024-05-08T21:21:55.604099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:33:44.560880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073170+0000","last_change":"2024-05-08T21:21:51.617174+0000","last_active":"2024-05-08T21:21:52.073170+0000","last_peered":"2024-05-08T21:21:52.073170+0000","last_clean":"2024-05-08T21:21:52.073170+0000","last_became_active":"2024-05-08T21:21:51.616894+0000","last_became_peered":"2024-05-08T21:21:51.616894+0000","last_unstale":"2024-05-08T21:21:52.073170+0000","last_undegraded":"2024-05-08T21:21:52.073170+0000","last_fullsized":"2024-05-08T21:21:52.073170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:38:17.091010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603455+0000","last_change":"2024-05-08T21:21:52.080625+0000","last_active":"2024-05-08T21:21:55.603455+0000","last_peered":"2024-05-08T21:21:55.603455+0000","last_clean":"2024-05-08T21:21:55.603455+0000","last_became_active":"2024-05-08T21:21:52.080347+0000","last_became_peered":"2024-05-08T21:21:52.080347+0000","last_unstale":"2024-05-08T21:21:55.603455+0000","last_undegraded":"2024-05-08T21:21:55.603455+0000","last_fullsized":"2024-05-08T21:21:55.603455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:04:56.104718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.087805+0000","last_change":"2024-05-08T21:21:51.617911+0000","last_active":"2024-05-08T21:21:52.087805+0000","last_peered":"2024-05-08T21:21:52.087805+0000","last_clean":"2024-05-08T21:21:52.087805+0000","last_became_active":"2024-05-08T21:21:51.617651+0000","last_became_peered":"2024-05-08T21:21:51.617651+0000","last_unstale":"2024-05-08T21:21:52.087805+0000","last_undegraded":"2024-05-08T21:21:52.087805+0000","last_fullsized":"2024-05-08T21:21:52.087805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:37:21.706277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076579+0000","last_change":"2024-05-08T21:21:52.090657+0000","last_active":"2024-05-08T21:21:53.076579+0000","last_peered":"2024-05-08T21:21:53.076579+0000","last_clean":"2024-05-08T21:21:53.076579+0000","last_became_active":"2024-05-08T21:21:52.090381+0000","last_became_peered":"2024-05-08T21:21:52.090381+0000","last_unstale":"2024-05-08T21:21:53.076579+0000","last_undegraded":"2024-05-08T21:21:53.076579+0000","last_fullsized":"2024-05-08T21:21:53.076579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:37:22.711367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.105960+0000","last_change":"2024-05-08T21:21:51.613700+0000","last_active":"2024-05-08T21:21:52.105960+0000","last_peered":"2024-05-08T21:21:52.105960+0000","last_clean":"2024-05-08T21:21:52.105960+0000","last_became_active":"2024-05-08T21:21:51.613457+0000","last_became_peered":"2024-05-08T21:21:51.613457+0000","last_unstale":"2024-05-08T21:21:52.105960+0000","last_undegraded":"2024-05-08T21:21:52.105960+0000","last_fullsized":"2024-05-08T21:21:52.105960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:00:55.530393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603663+0000","last_change":"2024-05-08T21:21:52.093097+0000","last_active":"2024-05-08T21:21:55.603663+0000","last_peered":"2024-05-08T21:21:55.603663+0000","last_clean":"2024-05-08T21:21:55.603663+0000","last_became_active":"2024-05-08T21:21:52.092769+0000","last_became_peered":"2024-05-08T21:21:52.092769+0000","last_unstale":"2024-05-08T21:21:55.603663+0000","last_undegraded":"2024-05-08T21:21:55.603663+0000","last_fullsized":"2024-05-08T21:21:55.603663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:38:46.924064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.100944+0000","last_change":"2024-05-08T21:21:51.614173+0000","last_active":"2024-05-08T21:21:52.100944+0000","last_peered":"2024-05-08T21:21:52.100944+0000","last_clean":"2024-05-08T21:21:52.100944+0000","last_became_active":"2024-05-08T21:21:51.613664+0000","last_became_peered":"2024-05-08T21:21:51.613664+0000","last_unstale":"2024-05-08T21:21:52.100944+0000","last_undegraded":"2024-05-08T21:21:52.100944+0000","last_fullsized":"2024-05-08T21:21:52.100944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:20:59.795219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602872+0000","last_change":"2024-05-08T21:21:52.081761+0000","last_active":"2024-05-08T21:21:55.602872+0000","last_peered":"2024-05-08T21:21:55.602872+0000","last_clean":"2024-05-08T21:21:55.602872+0000","last_became_active":"2024-05-08T21:21:52.081477+0000","last_became_peered":"2024-05-08T21:21:52.081477+0000","last_unstale":"2024-05-08T21:21:55.602872+0000","last_undegraded":"2024-05-08T21:21:55.602872+0000","last_fullsized":"2024-05-08T21:21:55.602872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:39:11.021535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.075759+0000","last_change":"2024-05-08T21:21:51.616956+0000","last_active":"2024-05-08T21:21:52.075759+0000","last_peered":"2024-05-08T21:21:52.075759+0000","last_clean":"2024-05-08T21:21:52.075759+0000","last_became_active":"2024-05-08T21:21:51.616670+0000","last_became_peered":"2024-05-08T21:21:51.616670+0000","last_unstale":"2024-05-08T21:21:52.075759+0000","last_undegraded":"2024-05-08T21:21:52.075759+0000","last_fullsized":"2024-05-08T21:21:52.075759+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:45:34.421641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603797+0000","last_change":"2024-05-08T21:21:52.100972+0000","last_active":"2024-05-08T21:21:55.603797+0000","last_peered":"2024-05-08T21:21:55.603797+0000","last_clean":"2024-05-08T21:21:55.603797+0000","last_became_active":"2024-05-08T21:21:52.100711+0000","last_became_peered":"2024-05-08T21:21:52.100711+0000","last_unstale":"2024-05-08T21:21:55.603797+0000","last_undegraded":"2024-05-08T21:21:55.603797+0000","last_fullsized":"2024-05-08T21:21:55.603797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:22:38.788235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073127+0000","last_change":"2024-05-08T21:21:51.609380+0000","last_active":"2024-05-08T21:21:52.073127+0000","last_peered":"2024-05-08T21:21:52.073127+0000","last_clean":"2024-05-08T21:21:52.073127+0000","last_became_active":"2024-05-08T21:21:51.608973+0000","last_became_peered":"2024-05-08T21:21:51.608973+0000","last_unstale":"2024-05-08T21:21:52.073127+0000","last_undegraded":"2024-05-08T21:21:52.073127+0000","last_fullsized":"2024-05-08T21:21:52.073127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:45:34.421641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602586+0000","last_change":"2024-05-08T21:21:52.088958+0000","last_active":"2024-05-08T21:21:55.602586+0000","last_peered":"2024-05-08T21:21:55.602586+0000","last_clean":"2024-05-08T21:21:55.602586+0000","last_became_active":"2024-05-08T21:21:52.088270+0000","last_became_peered":"2024-05-08T21:21:52.088270+0000","last_unstale":"2024-05-08T21:21:55.602586+0000","last_undegraded":"2024-05-08T21:21:55.602586+0000","last_fullsized":"2024-05-08T21:21:55.602586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:28:07.575847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.090786+0000","last_change":"2024-05-08T21:21:51.614824+0000","last_active":"2024-05-08T21:21:52.090786+0000","last_peered":"2024-05-08T21:21:52.090786+0000","last_clean":"2024-05-08T21:21:52.090786+0000","last_became_active":"2024-05-08T21:21:51.614526+0000","last_became_peered":"2024-05-08T21:21:51.614526+0000","last_unstale":"2024-05-08T21:21:52.090786+0000","last_undegraded":"2024-05-08T21:21:52.090786+0000","last_fullsized":"2024-05-08T21:21:52.090786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:41.009726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603517+0000","last_change":"2024-05-08T21:21:52.097367+0000","last_active":"2024-05-08T21:21:55.603517+0000","last_peered":"2024-05-08T21:21:55.603517+0000","last_clean":"2024-05-08T21:21:55.603517+0000","last_became_active":"2024-05-08T21:21:52.097047+0000","last_became_peered":"2024-05-08T21:21:52.097047+0000","last_unstale":"2024-05-08T21:21:55.603517+0000","last_undegraded":"2024-05-08T21:21:55.603517+0000","last_fullsized":"2024-05-08T21:21:55.603517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:46:26.988461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.100630+0000","last_change":"2024-05-08T21:21:51.615362+0000","last_active":"2024-05-08T21:21:52.100630+0000","last_peered":"2024-05-08T21:21:52.100630+0000","last_clean":"2024-05-08T21:21:52.100630+0000","last_became_active":"2024-05-08T21:21:51.615119+0000","last_became_peered":"2024-05-08T21:21:51.615119+0000","last_unstale":"2024-05-08T21:21:52.100630+0000","last_undegraded":"2024-05-08T21:21:52.100630+0000","last_fullsized":"2024-05-08T21:21:52.100630+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:10:09.131458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603677+0000","last_change":"2024-05-08T21:21:52.086491+0000","last_active":"2024-05-08T21:21:55.603677+0000","last_peered":"2024-05-08T21:21:55.603677+0000","last_clean":"2024-05-08T21:21:55.603677+0000","last_became_active":"2024-05-08T21:21:52.086134+0000","last_became_peered":"2024-05-08T21:21:52.086134+0000","last_unstale":"2024-05-08T21:21:55.603677+0000","last_undegraded":"2024-05-08T21:21:55.603677+0000","last_fullsized":"2024-05-08T21:21:55.603677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:41.471400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.070417+0000","last_change":"2024-05-08T21:21:51.617080+0000","last_active":"2024-05-08T21:21:52.070417+0000","last_peered":"2024-05-08T21:21:52.070417+0000","last_clean":"2024-05-08T21:21:52.070417+0000","last_became_active":"2024-05-08T21:21:51.615569+0000","last_became_peered":"2024-05-08T21:21:51.615569+0000","last_unstale":"2024-05-08T21:21:52.070417+0000","last_undegraded":"2024-05-08T21:21:52.070417+0000","last_fullsized":"2024-05-08T21:21:52.070417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:44:53.582574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604152+0000","last_change":"2024-05-08T21:21:52.089644+0000","last_active":"2024-05-08T21:21:55.604152+0000","last_peered":"2024-05-08T21:21:55.604152+0000","last_clean":"2024-05-08T21:21:55.604152+0000","last_became_active":"2024-05-08T21:21:52.089161+0000","last_became_peered":"2024-05-08T21:21:52.089161+0000","last_unstale":"2024-05-08T21:21:55.604152+0000","last_undegraded":"2024-05-08T21:21:55.604152+0000","last_fullsized":"2024-05-08T21:21:55.604152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:28:07.575847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.075661+0000","last_change":"2024-05-08T21:21:51.618799+0000","last_active":"2024-05-08T21:21:52.075661+0000","last_peered":"2024-05-08T21:21:52.075661+0000","last_clean":"2024-05-08T21:21:52.075661+0000","last_became_active":"2024-05-08T21:21:51.618584+0000","last_became_peered":"2024-05-08T21:21:51.618584+0000","last_unstale":"2024-05-08T21:21:52.075661+0000","last_undegraded":"2024-05-08T21:21:52.075661+0000","last_fullsized":"2024-05-08T21:21:52.075661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:44:53.582574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076748+0000","last_change":"2024-05-08T21:21:52.090935+0000","last_active":"2024-05-08T21:21:53.076748+0000","last_peered":"2024-05-08T21:21:53.076748+0000","last_clean":"2024-05-08T21:21:53.076748+0000","last_became_active":"2024-05-08T21:21:52.090545+0000","last_became_peered":"2024-05-08T21:21:52.090545+0000","last_unstale":"2024-05-08T21:21:53.076748+0000","last_undegraded":"2024-05-08T21:21:53.076748+0000","last_fullsized":"2024-05-08T21:21:53.076748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:14:37.380612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073078+0000","last_change":"2024-05-08T21:21:52.079445+0000","last_active":"2024-05-08T21:21:53.073078+0000","last_peered":"2024-05-08T21:21:53.073078+0000","last_clean":"2024-05-08T21:21:53.073078+0000","last_became_active":"2024-05-08T21:21:52.079055+0000","last_became_peered":"2024-05-08T21:21:52.079055+0000","last_unstale":"2024-05-08T21:21:53.073078+0000","last_undegraded":"2024-05-08T21:21:53.073078+0000","last_fullsized":"2024-05-08T21:21:53.073078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:28:07.575847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073936+0000","last_change":"2024-05-08T21:21:51.616872+0000","last_active":"2024-05-08T21:21:52.073936+0000","last_peered":"2024-05-08T21:21:52.073936+0000","last_clean":"2024-05-08T21:21:52.073936+0000","last_became_active":"2024-05-08T21:21:51.616576+0000","last_became_peered":"2024-05-08T21:21:51.616576+0000","last_unstale":"2024-05-08T21:21:52.073936+0000","last_undegraded":"2024-05-08T21:21:52.073936+0000","last_fullsized":"2024-05-08T21:21:52.073936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:38:17.091010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076646+0000","last_change":"2024-05-08T21:21:52.079441+0000","last_active":"2024-05-08T21:21:53.076646+0000","last_peered":"2024-05-08T21:21:53.076646+0000","last_clean":"2024-05-08T21:21:53.076646+0000","last_became_active":"2024-05-08T21:21:52.079170+0000","last_became_peered":"2024-05-08T21:21:52.079170+0000","last_unstale":"2024-05-08T21:21:53.076646+0000","last_undegraded":"2024-05-08T21:21:53.076646+0000","last_fullsized":"2024-05-08T21:21:53.076646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:41.471400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.072713+0000","last_change":"2024-05-08T21:21:51.613447+0000","last_active":"2024-05-08T21:21:52.072713+0000","last_peered":"2024-05-08T21:21:52.072713+0000","last_clean":"2024-05-08T21:21:52.072713+0000","last_became_active":"2024-05-08T21:21:51.613150+0000","last_became_peered":"2024-05-08T21:21:51.613150+0000","last_unstale":"2024-05-08T21:21:52.072713+0000","last_undegraded":"2024-05-08T21:21:52.072713+0000","last_fullsized":"2024-05-08T21:21:52.072713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:13.563899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073528+0000","last_change":"2024-05-08T21:21:42.333097+0000","last_active":"2024-05-08T21:21:52.073528+0000","last_peered":"2024-05-08T21:21:52.073528+0000","last_clean":"2024-05-08T21:21:52.073528+0000","last_became_active":"2024-05-08T21:21:42.332033+0000","last_became_peered":"2024-05-08T21:21:42.332033+0000","last_unstale":"2024-05-08T21:21:52.073528+0000","last_undegraded":"2024-05-08T21:21:52.073528+0000","last_fullsized":"2024-05-08T21:21:52.073528+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_clean_scrub_stamp":"2024-05-08T21:21:40.862622+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:39:24.772551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602793+0000","last_change":"2024-05-08T21:21:52.096265+0000","last_active":"2024-05-08T21:21:55.602793+0000","last_peered":"2024-05-08T21:21:55.602793+0000","last_clean":"2024-05-08T21:21:55.602793+0000","last_became_active":"2024-05-08T21:21:52.095929+0000","last_became_peered":"2024-05-08T21:21:52.095929+0000","last_unstale":"2024-05-08T21:21:55.602793+0000","last_undegraded":"2024-05-08T21:21:55.602793+0000","last_fullsized":"2024-05-08T21:21:55.602793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:42.014816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073633+0000","last_change":"2024-05-08T21:21:51.614632+0000","last_active":"2024-05-08T21:21:52.073633+0000","last_peered":"2024-05-08T21:21:52.073633+0000","last_clean":"2024-05-08T21:21:52.073633+0000","last_became_active":"2024-05-08T21:21:51.614368+0000","last_became_peered":"2024-05-08T21:21:51.614368+0000","last_unstale":"2024-05-08T21:21:52.073633+0000","last_undegraded":"2024-05-08T21:21:52.073633+0000","last_fullsized":"2024-05-08T21:21:52.073633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:14:36.375522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.075513+0000","last_change":"2024-05-08T21:21:51.615266+0000","last_active":"2024-05-08T21:21:52.075513+0000","last_peered":"2024-05-08T21:21:52.075513+0000","last_clean":"2024-05-08T21:21:52.075513+0000","last_became_active":"2024-05-08T21:21:51.614949+0000","last_became_peered":"2024-05-08T21:21:51.614949+0000","last_unstale":"2024-05-08T21:21:52.075513+0000","last_undegraded":"2024-05-08T21:21:52.075513+0000","last_fullsized":"2024-05-08T21:21:52.075513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:44:53.582574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073133+0000","last_change":"2024-05-08T21:21:52.079810+0000","last_active":"2024-05-08T21:21:53.073133+0000","last_peered":"2024-05-08T21:21:53.073133+0000","last_clean":"2024-05-08T21:21:53.073133+0000","last_became_active":"2024-05-08T21:21:52.079291+0000","last_became_peered":"2024-05-08T21:21:52.079291+0000","last_unstale":"2024-05-08T21:21:53.073133+0000","last_undegraded":"2024-05-08T21:21:53.073133+0000","last_fullsized":"2024-05-08T21:21:53.073133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:41:45.740288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.068776+0000","last_change":"2024-05-08T21:21:51.613365+0000","last_active":"2024-05-08T21:21:52.068776+0000","last_peered":"2024-05-08T21:21:52.068776+0000","last_clean":"2024-05-08T21:21:52.068776+0000","last_became_active":"2024-05-08T21:21:51.613022+0000","last_became_peered":"2024-05-08T21:21:51.613022+0000","last_unstale":"2024-05-08T21:21:52.068776+0000","last_undegraded":"2024-05-08T21:21:52.068776+0000","last_fullsized":"2024-05-08T21:21:52.068776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:14:36.375522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076917+0000","last_change":"2024-05-08T21:21:52.082814+0000","last_active":"2024-05-08T21:21:53.076917+0000","last_peered":"2024-05-08T21:21:53.076917+0000","last_clean":"2024-05-08T21:21:53.076917+0000","last_became_active":"2024-05-08T21:21:52.082539+0000","last_became_peered":"2024-05-08T21:21:52.082539+0000","last_unstale":"2024-05-08T21:21:53.076917+0000","last_undegraded":"2024-05-08T21:21:53.076917+0000","last_fullsized":"2024-05-08T21:21:53.076917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:36:48.359422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.093644+0000","last_change":"2024-05-08T21:21:51.612996+0000","last_active":"2024-05-08T21:21:52.093644+0000","last_peered":"2024-05-08T21:21:52.093644+0000","last_clean":"2024-05-08T21:21:52.093644+0000","last_became_active":"2024-05-08T21:21:51.612671+0000","last_became_peered":"2024-05-08T21:21:51.612671+0000","last_unstale":"2024-05-08T21:21:52.093644+0000","last_undegraded":"2024-05-08T21:21:52.093644+0000","last_fullsized":"2024-05-08T21:21:52.093644+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:44:53.582574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603051+0000","last_change":"2024-05-08T21:21:52.075455+0000","last_active":"2024-05-08T21:21:55.603051+0000","last_peered":"2024-05-08T21:21:55.603051+0000","last_clean":"2024-05-08T21:21:55.603051+0000","last_became_active":"2024-05-08T21:21:52.075082+0000","last_became_peered":"2024-05-08T21:21:52.075082+0000","last_unstale":"2024-05-08T21:21:55.603051+0000","last_undegraded":"2024-05-08T21:21:55.603051+0000","last_fullsized":"2024-05-08T21:21:55.603051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:02.395932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073416+0000","last_change":"2024-05-08T21:21:51.614333+0000","last_active":"2024-05-08T21:21:52.073416+0000","last_peered":"2024-05-08T21:21:52.073416+0000","last_clean":"2024-05-08T21:21:52.073416+0000","last_became_active":"2024-05-08T21:21:51.614048+0000","last_became_peered":"2024-05-08T21:21:51.614048+0000","last_unstale":"2024-05-08T21:21:52.073416+0000","last_undegraded":"2024-05-08T21:21:52.073416+0000","last_fullsized":"2024-05-08T21:21:52.073416+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:37:21.706277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076323+0000","last_change":"2024-05-08T21:21:52.094009+0000","last_active":"2024-05-08T21:21:53.076323+0000","last_peered":"2024-05-08T21:21:53.076323+0000","last_clean":"2024-05-08T21:21:53.076323+0000","last_became_active":"2024-05-08T21:21:52.093730+0000","last_became_peered":"2024-05-08T21:21:52.093730+0000","last_unstale":"2024-05-08T21:21:53.076323+0000","last_undegraded":"2024-05-08T21:21:53.076323+0000","last_fullsized":"2024-05-08T21:21:53.076323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:14.568989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.068928+0000","last_change":"2024-05-08T21:21:51.620291+0000","last_active":"2024-05-08T21:21:52.068928+0000","last_peered":"2024-05-08T21:21:52.068928+0000","last_clean":"2024-05-08T21:21:52.068928+0000","last_became_active":"2024-05-08T21:21:51.619965+0000","last_became_peered":"2024-05-08T21:21:51.619965+0000","last_unstale":"2024-05-08T21:21:52.068928+0000","last_undegraded":"2024-05-08T21:21:52.068928+0000","last_fullsized":"2024-05-08T21:21:52.068928+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:05:44.822440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076439+0000","last_change":"2024-05-08T21:21:52.086946+0000","last_active":"2024-05-08T21:21:53.076439+0000","last_peered":"2024-05-08T21:21:53.076439+0000","last_clean":"2024-05-08T21:21:53.076439+0000","last_became_active":"2024-05-08T21:21:52.086651+0000","last_became_peered":"2024-05-08T21:21:52.086651+0000","last_unstale":"2024-05-08T21:21:53.076439+0000","last_undegraded":"2024-05-08T21:21:53.076439+0000","last_fullsized":"2024-05-08T21:21:53.076439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:22:38.788235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.068205+0000","last_change":"2024-05-08T21:21:51.616253+0000","last_active":"2024-05-08T21:21:52.068205+0000","last_peered":"2024-05-08T21:21:52.068205+0000","last_clean":"2024-05-08T21:21:52.068205+0000","last_became_active":"2024-05-08T21:21:51.615883+0000","last_became_peered":"2024-05-08T21:21:51.615883+0000","last_unstale":"2024-05-08T21:21:52.068205+0000","last_undegraded":"2024-05-08T21:21:52.068205+0000","last_fullsized":"2024-05-08T21:21:52.068205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:13.563899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076873+0000","last_change":"2024-05-08T21:21:52.081159+0000","last_active":"2024-05-08T21:21:53.076873+0000","last_peered":"2024-05-08T21:21:53.076873+0000","last_clean":"2024-05-08T21:21:53.076873+0000","last_became_active":"2024-05-08T21:21:52.080823+0000","last_became_peered":"2024-05-08T21:21:52.080823+0000","last_unstale":"2024-05-08T21:21:53.076873+0000","last_undegraded":"2024-05-08T21:21:53.076873+0000","last_fullsized":"2024-05-08T21:21:53.076873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:41:45.740288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.100647+0000","last_change":"2024-05-08T21:21:51.609896+0000","last_active":"2024-05-08T21:21:52.100647+0000","last_peered":"2024-05-08T21:21:52.100647+0000","last_clean":"2024-05-08T21:21:52.100647+0000","last_became_active":"2024-05-08T21:21:51.609409+0000","last_became_peered":"2024-05-08T21:21:51.609409+0000","last_unstale":"2024-05-08T21:21:52.100647+0000","last_undegraded":"2024-05-08T21:21:52.100647+0000","last_fullsized":"2024-05-08T21:21:52.100647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:38:17.091010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603571+0000","last_change":"2024-05-08T21:21:52.101873+0000","last_active":"2024-05-08T21:21:55.603571+0000","last_peered":"2024-05-08T21:21:55.603571+0000","last_clean":"2024-05-08T21:21:55.603571+0000","last_became_active":"2024-05-08T21:21:52.101487+0000","last_became_peered":"2024-05-08T21:21:52.101487+0000","last_unstale":"2024-05-08T21:21:55.603571+0000","last_undegraded":"2024-05-08T21:21:55.603571+0000","last_fullsized":"2024-05-08T21:21:55.603571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:02.395932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073673+0000","last_change":"2024-05-08T21:21:51.620697+0000","last_active":"2024-05-08T21:21:52.073673+0000","last_peered":"2024-05-08T21:21:52.073673+0000","last_clean":"2024-05-08T21:21:52.073673+0000","last_became_active":"2024-05-08T21:21:51.620370+0000","last_became_peered":"2024-05-08T21:21:51.620370+0000","last_unstale":"2024-05-08T21:21:52.073673+0000","last_undegraded":"2024-05-08T21:21:52.073673+0000","last_fullsized":"2024-05-08T21:21:52.073673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:00:55.530393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603751+0000","last_change":"2024-05-08T21:21:52.089908+0000","last_active":"2024-05-08T21:21:55.603751+0000","last_peered":"2024-05-08T21:21:55.603751+0000","last_clean":"2024-05-08T21:21:55.603751+0000","last_became_active":"2024-05-08T21:21:52.089610+0000","last_became_peered":"2024-05-08T21:21:52.089610+0000","last_unstale":"2024-05-08T21:21:55.603751+0000","last_undegraded":"2024-05-08T21:21:55.603751+0000","last_fullsized":"2024-05-08T21:21:55.603751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:26:01.523710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.089213+0000","last_change":"2024-05-08T21:21:51.610168+0000","last_active":"2024-05-08T21:21:52.089213+0000","last_peered":"2024-05-08T21:21:52.089213+0000","last_clean":"2024-05-08T21:21:52.089213+0000","last_became_active":"2024-05-08T21:21:51.609850+0000","last_became_peered":"2024-05-08T21:21:51.609850+0000","last_unstale":"2024-05-08T21:21:52.089213+0000","last_undegraded":"2024-05-08T21:21:52.089213+0000","last_fullsized":"2024-05-08T21:21:52.089213+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:41.009726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603664+0000","last_change":"2024-05-08T21:21:52.090014+0000","last_active":"2024-05-08T21:21:55.603664+0000","last_peered":"2024-05-08T21:21:55.603664+0000","last_clean":"2024-05-08T21:21:55.603664+0000","last_became_active":"2024-05-08T21:21:52.089630+0000","last_became_peered":"2024-05-08T21:21:52.089630+0000","last_unstale":"2024-05-08T21:21:55.603664+0000","last_undegraded":"2024-05-08T21:21:55.603664+0000","last_fullsized":"2024-05-08T21:21:55.603664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:26:01.523710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.088752+0000","last_change":"2024-05-08T21:21:51.613229+0000","last_active":"2024-05-08T21:21:52.088752+0000","last_peered":"2024-05-08T21:21:52.088752+0000","last_clean":"2024-05-08T21:21:52.088752+0000","last_became_active":"2024-05-08T21:21:51.612871+0000","last_became_peered":"2024-05-08T21:21:51.612871+0000","last_unstale":"2024-05-08T21:21:52.088752+0000","last_undegraded":"2024-05-08T21:21:52.088752+0000","last_fullsized":"2024-05-08T21:21:52.088752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:45:34.421641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604321+0000","last_change":"2024-05-08T21:21:52.097257+0000","last_active":"2024-05-08T21:21:55.604321+0000","last_peered":"2024-05-08T21:21:55.604321+0000","last_clean":"2024-05-08T21:21:55.604321+0000","last_became_active":"2024-05-08T21:21:52.096970+0000","last_became_peered":"2024-05-08T21:21:52.096970+0000","last_unstale":"2024-05-08T21:21:55.604321+0000","last_undegraded":"2024-05-08T21:21:55.604321+0000","last_fullsized":"2024-05-08T21:21:55.604321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:33:44.560880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602552+0000","last_change":"2024-05-08T21:21:52.089064+0000","last_active":"2024-05-08T21:21:55.602552+0000","last_peered":"2024-05-08T21:21:55.602552+0000","last_clean":"2024-05-08T21:21:55.602552+0000","last_became_active":"2024-05-08T21:21:52.088633+0000","last_became_peered":"2024-05-08T21:21:52.088633+0000","last_unstale":"2024-05-08T21:21:55.602552+0000","last_undegraded":"2024-05-08T21:21:55.602552+0000","last_fullsized":"2024-05-08T21:21:55.602552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:39:11.021535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602557+0000","last_change":"2024-05-08T21:21:52.087199+0000","last_active":"2024-05-08T21:21:55.602557+0000","last_peered":"2024-05-08T21:21:55.602557+0000","last_clean":"2024-05-08T21:21:55.602557+0000","last_became_active":"2024-05-08T21:21:52.086874+0000","last_became_peered":"2024-05-08T21:21:52.086874+0000","last_unstale":"2024-05-08T21:21:55.602557+0000","last_undegraded":"2024-05-08T21:21:55.602557+0000","last_fullsized":"2024-05-08T21:21:55.602557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:18:36.173550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603028+0000","last_change":"2024-05-08T21:21:52.090034+0000","last_active":"2024-05-08T21:21:55.603028+0000","last_peered":"2024-05-08T21:21:55.603028+0000","last_clean":"2024-05-08T21:21:55.603028+0000","last_became_active":"2024-05-08T21:21:52.089251+0000","last_became_peered":"2024-05-08T21:21:52.089251+0000","last_unstale":"2024-05-08T21:21:55.603028+0000","last_undegraded":"2024-05-08T21:21:55.603028+0000","last_fullsized":"2024-05-08T21:21:55.603028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:41.471400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603108+0000","last_change":"2024-05-08T21:21:52.085409+0000","last_active":"2024-05-08T21:21:55.603108+0000","last_peered":"2024-05-08T21:21:55.603108+0000","last_clean":"2024-05-08T21:21:55.603108+0000","last_became_active":"2024-05-08T21:21:52.085050+0000","last_became_peered":"2024-05-08T21:21:52.085050+0000","last_unstale":"2024-05-08T21:21:55.603108+0000","last_undegraded":"2024-05-08T21:21:55.603108+0000","last_fullsized":"2024-05-08T21:21:55.603108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:40:06.048132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076692+0000","last_change":"2024-05-08T21:21:52.100833+0000","last_active":"2024-05-08T21:21:53.076692+0000","last_peered":"2024-05-08T21:21:53.076692+0000","last_clean":"2024-05-08T21:21:53.076692+0000","last_became_active":"2024-05-08T21:21:52.100640+0000","last_became_peered":"2024-05-08T21:21:52.100640+0000","last_unstale":"2024-05-08T21:21:53.076692+0000","last_undegraded":"2024-05-08T21:21:53.076692+0000","last_fullsized":"2024-05-08T21:21:53.076692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:02.395932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073296+0000","last_change":"2024-05-08T21:21:52.080879+0000","last_active":"2024-05-08T21:21:53.073296+0000","last_peered":"2024-05-08T21:21:53.073296+0000","last_clean":"2024-05-08T21:21:53.073296+0000","last_became_active":"2024-05-08T21:21:52.080592+0000","last_became_peered":"2024-05-08T21:21:52.080592+0000","last_unstale":"2024-05-08T21:21:53.073296+0000","last_undegraded":"2024-05-08T21:21:53.073296+0000","last_fullsized":"2024-05-08T21:21:53.073296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:39:11.021535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603171+0000","last_change":"2024-05-08T21:21:52.092091+0000","last_active":"2024-05-08T21:21:55.603171+0000","last_peered":"2024-05-08T21:21:55.603171+0000","last_clean":"2024-05-08T21:21:55.603171+0000","last_became_active":"2024-05-08T21:21:52.091489+0000","last_became_peered":"2024-05-08T21:21:52.091489+0000","last_unstale":"2024-05-08T21:21:55.603171+0000","last_undegraded":"2024-05-08T21:21:55.603171+0000","last_fullsized":"2024-05-08T21:21:55.603171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:38.782318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076796+0000","last_change":"2024-05-08T21:21:52.087595+0000","last_active":"2024-05-08T21:21:53.076796+0000","last_peered":"2024-05-08T21:21:53.076796+0000","last_clean":"2024-05-08T21:21:53.076796+0000","last_became_active":"2024-05-08T21:21:52.087302+0000","last_became_peered":"2024-05-08T21:21:52.087302+0000","last_unstale":"2024-05-08T21:21:53.076796+0000","last_undegraded":"2024-05-08T21:21:53.076796+0000","last_fullsized":"2024-05-08T21:21:53.076796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:40:06.048132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603267+0000","last_change":"2024-05-08T21:21:52.089760+0000","last_active":"2024-05-08T21:21:55.603267+0000","last_peered":"2024-05-08T21:21:55.603267+0000","last_clean":"2024-05-08T21:21:55.603267+0000","last_became_active":"2024-05-08T21:21:52.088814+0000","last_became_peered":"2024-05-08T21:21:52.088814+0000","last_unstale":"2024-05-08T21:21:55.603267+0000","last_undegraded":"2024-05-08T21:21:55.603267+0000","last_fullsized":"2024-05-08T21:21:55.603267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:04:56.104718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076502+0000","last_change":"2024-05-08T21:21:52.089062+0000","last_active":"2024-05-08T21:21:53.076502+0000","last_peered":"2024-05-08T21:21:53.076502+0000","last_clean":"2024-05-08T21:21:53.076502+0000","last_became_active":"2024-05-08T21:21:52.088653+0000","last_became_peered":"2024-05-08T21:21:52.088653+0000","last_unstale":"2024-05-08T21:21:53.076502+0000","last_undegraded":"2024-05-08T21:21:53.076502+0000","last_fullsized":"2024-05-08T21:21:53.076502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:56:56.333047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602914+0000","last_change":"2024-05-08T21:21:52.096639+0000","last_active":"2024-05-08T21:21:55.602914+0000","last_peered":"2024-05-08T21:21:55.602914+0000","last_clean":"2024-05-08T21:21:55.602914+0000","last_became_active":"2024-05-08T21:21:52.096444+0000","last_became_peered":"2024-05-08T21:21:52.096444+0000","last_unstale":"2024-05-08T21:21:55.602914+0000","last_undegraded":"2024-05-08T21:21:55.602914+0000","last_fullsized":"2024-05-08T21:21:55.602914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:10:50.287200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603006+0000","last_change":"2024-05-08T21:21:52.082022+0000","last_active":"2024-05-08T21:21:55.603006+0000","last_peered":"2024-05-08T21:21:55.603006+0000","last_clean":"2024-05-08T21:21:55.603006+0000","last_became_active":"2024-05-08T21:21:52.081764+0000","last_became_peered":"2024-05-08T21:21:52.081764+0000","last_unstale":"2024-05-08T21:21:55.603006+0000","last_undegraded":"2024-05-08T21:21:55.603006+0000","last_fullsized":"2024-05-08T21:21:55.603006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:22:38.788235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602807+0000","last_change":"2024-05-08T21:21:52.101119+0000","last_active":"2024-05-08T21:21:55.602807+0000","last_peered":"2024-05-08T21:21:55.602807+0000","last_clean":"2024-05-08T21:21:55.602807+0000","last_became_active":"2024-05-08T21:21:52.100863+0000","last_became_peered":"2024-05-08T21:21:52.100863+0000","last_unstale":"2024-05-08T21:21:55.602807+0000","last_undegraded":"2024-05-08T21:21:55.602807+0000","last_fullsized":"2024-05-08T21:21:55.602807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:40:06.048132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":790528,"data_stored":655548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":66162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":655604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":64598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":64688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T21:21:56.757 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-08T21:21:56.757 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-08T21:21:56.757 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-08T21:21:56.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-08T21:21:56.758 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-08T21:21:56.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.932+0000 7fa9445c3700 1 Processor -- start 2024-05-08T21:21:56.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.932+0000 7fa9445c3700 1 -- start start 2024-05-08T21:21:56.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.933+0000 7fa9445c3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa93c0af030 0x7fa93c15e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:56.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.933+0000 7fa9445c3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa93c06e4d0 0x7fa93c06a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:56.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.933+0000 7fa9445c3700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93c0b4a80 con 0x7fa93c0b3990 2024-05-08T21:21:56.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.933+0000 7fa9445c3700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa93c15ede0 con 0x7fa93c0af030 2024-05-08T21:21:56.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.933+0000 7fa9445c3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c06d800 con 0x7fa93c06e4d0 2024-05-08T21:21:56.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.933+0000 7fa94235f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa93c0af030 0x7fa93c15e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:56.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.933+0000 7fa94235f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa93c0af030 0x7fa93c15e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52264/0 (socket says 172.21.15.3:52264) 2024-05-08T21:21:56.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.933+0000 7fa94235f700 1 -- 172.21.15.3:0/1336918125 learned_addr learned my addr 172.21.15.3:0/1336918125 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:56.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.933+0000 7fa942b60700 1 --2- 172.21.15.3:0/1336918125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa93c06e4d0 0x7fa93c06a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:56.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.933+0000 7fa94235f700 1 -- 172.21.15.3:0/1336918125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa93c06e4d0 msgr2=0x7fa93c06a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:56.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.933+0000 7fa94235f700 1 --2- 172.21.15.3:0/1336918125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa93c06e4d0 0x7fa93c06a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.933+0000 7fa94235f700 1 -- 172.21.15.3:0/1336918125 >> v1:172.21.15.186:6789/0 conn(0x7fa93c0b3990 legacy=0x7fa93c06ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.933+0000 7fa94235f700 1 -- 172.21.15.3:0/1336918125 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c0b3e70 con 0x7fa93c0af030 2024-05-08T21:21:56.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.934+0000 7fa94235f700 1 --2- 172.21.15.3:0/1336918125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa93c0af030 0x7fa93c15e8a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa9380098f0 tx=0x7fa938009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16f55774aeac8006 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:56.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.934+0000 7fa94135d700 1 -- 172.21.15.3:0/1336918125 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93800b8a0 con 0x7fa93c0af030 2024-05-08T21:21:56.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.934+0000 7fa94135d700 1 -- 172.21.15.3:0/1336918125 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93800ba00 con 0x7fa93c0af030 2024-05-08T21:21:56.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.934+0000 7fa94135d700 1 -- 172.21.15.3:0/1336918125 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93800bd90 con 0x7fa93c0af030 2024-05-08T21:21:56.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.934+0000 7fa9445c3700 1 -- 172.21.15.3:0/1336918125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa93c0af030 msgr2=0x7fa93c15e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.934+0000 7fa9445c3700 1 --2- 172.21.15.3:0/1336918125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa93c0af030 0x7fa93c15e8a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa9380098f0 tx=0x7fa938009c00 comp rx=0 tx=0).stop 2024-05-08T21:21:56.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.934+0000 7fa9445c3700 1 -- 172.21.15.3:0/1336918125 shutdown_connections 2024-05-08T21:21:56.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.934+0000 7fa9445c3700 1 --2- 172.21.15.3:0/1336918125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa93c06e4d0 0x7fa93c06a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.934+0000 7fa9445c3700 1 --2- 172.21.15.3:0/1336918125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa93c0af030 0x7fa93c15e8a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.934+0000 7fa9445c3700 1 -- 172.21.15.3:0/1336918125 >> 172.21.15.3:0/1336918125 conn(0x7fa93c0cfce0 msgr2=0x7fa93c0d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:56.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.934+0000 7fa9445c3700 1 -- 172.21.15.3:0/1336918125 shutdown_connections 2024-05-08T21:21:56.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.934+0000 7fa9445c3700 1 -- 172.21.15.3:0/1336918125 wait complete. 2024-05-08T21:21:56.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.936+0000 7fa9445c3700 1 Processor -- start 2024-05-08T21:21:56.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.936+0000 7fa9445c3700 1 -- start start 2024-05-08T21:21:56.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.936+0000 7fa9445c3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa93c06e4d0 0x7fa93c145780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:56.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.936+0000 7fa9445c3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93c0af030 0x7fa93c1f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:56.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa9445c3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa93c0b3990 0x7fa93c143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:56.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa9445c3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c158880 con 0x7fa93c06e4d0 2024-05-08T21:21:56.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa9445c3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c1f7930 con 0x7fa93c0af030 2024-05-08T21:21:56.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa9445c3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c1f3500 con 0x7fa93c0b3990 2024-05-08T21:21:56.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa94235f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa93c06e4d0 0x7fa93c145780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:56.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa94235f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa93c06e4d0 0x7fa93c145780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52280/0 (socket says 172.21.15.3:52280) 2024-05-08T21:21:56.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa94235f700 1 -- 172.21.15.3:0/2839378711 learned_addr learned my addr 172.21.15.3:0/2839378711 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:56.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa941b5e700 1 --2- 172.21.15.3:0/2839378711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93c0af030 0x7fa93c1f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:56.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa942b60700 1 --2- 172.21.15.3:0/2839378711 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa93c0b3990 0x7fa93c143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:56.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa94235f700 1 -- 172.21.15.3:0/2839378711 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa93c0b3990 msgr2=0x7fa93c143730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa94235f700 1 --2- 172.21.15.3:0/2839378711 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa93c0b3990 0x7fa93c143730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa94235f700 1 -- 172.21.15.3:0/2839378711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93c0af030 msgr2=0x7fa93c1f73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:56.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa94235f700 1 --2- 172.21.15.3:0/2839378711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93c0af030 0x7fa93c1f73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:56.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa94235f700 1 -- 172.21.15.3:0/2839378711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa938009550 con 0x7fa93c06e4d0 2024-05-08T21:21:56.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa942b60700 1 --2- 172.21.15.3:0/2839378711 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa93c0b3990 0x7fa93c143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:21:56.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa94235f700 1 --2- 172.21.15.3:0/2839378711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa93c06e4d0 0x7fa93c145780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa938009c40 tx=0x7fa938004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:56.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa932ffd700 1 -- 172.21.15.3:0/2839378711 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa938004480 con 0x7fa93c06e4d0 2024-05-08T21:21:56.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa932ffd700 1 -- 172.21.15.3:0/2839378711 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93801f070 con 0x7fa93c06e4d0 2024-05-08T21:21:56.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa9445c3700 1 -- 172.21.15.3:0/2839378711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa93c145da0 con 0x7fa93c06e4d0 2024-05-08T21:21:56.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa932ffd700 1 -- 172.21.15.3:0/2839378711 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9380038c0 con 0x7fa93c06e4d0 2024-05-08T21:21:56.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.937+0000 7fa9445c3700 1 -- 172.21.15.3:0/2839378711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa93c146640 con 0x7fa93c06e4d0 2024-05-08T21:21:56.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.938+0000 7fa9445c3700 1 -- 172.21.15.3:0/2839378711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa93c068000 con 0x7fa93c06e4d0 2024-05-08T21:21:56.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.939+0000 7fa932ffd700 1 -- 172.21.15.3:0/2839378711 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa938021030 con 0x7fa93c06e4d0 2024-05-08T21:21:56.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.939+0000 7fa932ffd700 1 --2- 172.21.15.3:0/2839378711 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa92006ac90 0x7fa92006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:56.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.939+0000 7fa932ffd700 1 -- 172.21.15.3:0/2839378711 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa93808a680 con 0x7fa93c06e4d0 2024-05-08T21:21:56.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.939+0000 7fa941b5e700 1 --2- 172.21.15.3:0/2839378711 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa92006ac90 0x7fa92006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:56.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.940+0000 7fa941b5e700 1 --2- 172.21.15.3:0/2839378711 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa92006ac90 0x7fa92006d150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa93c157770 tx=0x7fa92c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:56.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:56.944+0000 7fa932ffd700 1 -- 172.21.15.3:0/2839378711 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fa938055980 con 0x7fa93c06e4d0 2024-05-08T21:21:57.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.229+0000 7fa9445c3700 1 -- 172.21.15.3:0/2839378711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fa93c0b3380 con 0x7fa93c06e4d0 2024-05-08T21:21:57.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.233+0000 7fa932ffd700 1 -- 172.21.15.3:0/2839378711 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170943 (secure 0 0 0) 0x7fa938058fa0 con 0x7fa93c06e4d0 2024-05-08T21:21:57.236 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:21:57.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.236+0000 7fa9445c3700 1 -- 172.21.15.3:0/2839378711 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa92006ac90 msgr2=0x7fa92006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:57.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.236+0000 7fa9445c3700 1 --2- 172.21.15.3:0/2839378711 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa92006ac90 0x7fa92006d150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa93c157770 tx=0x7fa92c009040 comp rx=0 tx=0).stop 2024-05-08T21:21:57.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.236+0000 7fa9445c3700 1 -- 172.21.15.3:0/2839378711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa93c06e4d0 msgr2=0x7fa93c145780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:57.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.236+0000 7fa9445c3700 1 --2- 172.21.15.3:0/2839378711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa93c06e4d0 0x7fa93c145780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa938009c40 tx=0x7fa938004750 comp rx=0 tx=0).stop 2024-05-08T21:21:57.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.236+0000 7fa9445c3700 1 -- 172.21.15.3:0/2839378711 shutdown_connections 2024-05-08T21:21:57.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.236+0000 7fa9445c3700 1 --2- 172.21.15.3:0/2839378711 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa93c0b3990 0x7fa93c143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.236+0000 7fa9445c3700 1 --2- 172.21.15.3:0/2839378711 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa92006ac90 0x7fa92006d150 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.236+0000 7fa9445c3700 1 --2- 172.21.15.3:0/2839378711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa93c06e4d0 0x7fa93c145780 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.236+0000 7fa9445c3700 1 --2- 172.21.15.3:0/2839378711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93c0af030 0x7fa93c1f73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.236+0000 7fa9445c3700 1 -- 172.21.15.3:0/2839378711 >> 172.21.15.3:0/2839378711 conn(0x7fa93c0cfce0 msgr2=0x7fa93c15fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:57.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.236+0000 7fa9445c3700 1 -- 172.21.15.3:0/2839378711 shutdown_connections 2024-05-08T21:21:57.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.236+0000 7fa9445c3700 1 -- 172.21.15.3:0/2839378711 wait complete. 2024-05-08T21:21:57.251 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":5,"flags":0,"active_gid":4111,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":46682},{"type":"v1","addr":"172.21.15.186:6833","nonce":46682}]},"active_addr":"172.21.15.186:6833/46682","active_change":"2024-05-08T21:21:34.520029+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4101,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":3823244936}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":2804554265}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":269942971}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":3089537195}]}]} 2024-05-08T21:21:57.254 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-08T21:21:57.255 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-08T21:21:57.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:21:57.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.389+0000 7f2f55189700 1 Processor -- start 2024-05-08T21:21:57.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.389+0000 7f2f55189700 1 -- start start 2024-05-08T21:21:57.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.390+0000 7f2f55189700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2f5006bd80 0x7f2f500d0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:57.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.390+0000 7f2f55189700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f5006c3d0 0x7f2f5006c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:57.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.390+0000 7f2f55189700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f500cf990 con 0x7f2f5015da10 2024-05-08T21:21:57.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.390+0000 7f2f55189700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5009ab40 con 0x7f2f5006bd80 2024-05-08T21:21:57.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.390+0000 7f2f55189700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5006cda0 con 0x7f2f5006c3d0 2024-05-08T21:21:57.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.391+0000 7f2f4659c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2f5006bd80 0x7f2f500d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:57.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.391+0000 7f2f4f59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f5006c3d0 0x7f2f5006c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:57.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.391+0000 7f2f4ed9d700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f2f5015da10 0x7f2f5009a430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:44278/0 (socket says 172.21.15.3:44278) 2024-05-08T21:21:57.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.391+0000 7f2f4659c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2f5006bd80 0x7f2f500d0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:38294/0 (socket says 172.21.15.3:38294) 2024-05-08T21:21:57.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.391+0000 7f2f4659c700 1 -- 172.21.15.3:0/1110031748 learned_addr learned my addr 172.21.15.3:0/1110031748 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:57.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.391+0000 7f2f4659c700 1 -- 172.21.15.3:0/1110031748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f5006c3d0 msgr2=0x7f2f5006c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:57.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.391+0000 7f2f4659c700 1 --2- 172.21.15.3:0/1110031748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f5006c3d0 0x7f2f5006c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.391+0000 7f2f4659c700 1 -- 172.21.15.3:0/1110031748 >> v1:172.21.15.186:6789/0 conn(0x7f2f5015da10 legacy=0x7f2f5009a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:57.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.391+0000 7f2f4659c700 1 -- 172.21.15.3:0/1110031748 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f500d04b0 con 0x7f2f5006bd80 2024-05-08T21:21:57.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.391+0000 7f2f4e59c700 1 -- 172.21.15.3:0/1110031748 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002082664 0 0) 0x7f2f500cf990 con 0x7f2f5015da10 2024-05-08T21:21:57.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.391+0000 7f2f4f59e700 1 --2- 172.21.15.3:0/1110031748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f5006c3d0 0x7f2f5006c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:21:57.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.391+0000 7f2f4659c700 1 --2- 172.21.15.3:0/1110031748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f5006bd80 0x7f2f500d0940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2f38009920 tx=0x7f2f38009c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee3e06269b194e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:57.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.391+0000 7f2f4e59c700 1 -- 172.21.15.3:0/1110031748 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38011830 con 0x7f2f5006bd80 2024-05-08T21:21:57.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.391+0000 7f2f4e59c700 1 -- 172.21.15.3:0/1110031748 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f38011990 con 0x7f2f5006bd80 2024-05-08T21:21:57.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.391+0000 7f2f4e59c700 1 -- 172.21.15.3:0/1110031748 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38011d20 con 0x7f2f5006bd80 2024-05-08T21:21:57.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.392+0000 7f2f55189700 1 -- 172.21.15.3:0/1110031748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f5006bd80 msgr2=0x7f2f500d0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:57.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.392+0000 7f2f55189700 1 --2- 172.21.15.3:0/1110031748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f5006bd80 0x7f2f500d0940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2f38009920 tx=0x7f2f38009c30 comp rx=0 tx=0).stop 2024-05-08T21:21:57.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.392+0000 7f2f55189700 1 -- 172.21.15.3:0/1110031748 shutdown_connections 2024-05-08T21:21:57.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.392+0000 7f2f55189700 1 --2- 172.21.15.3:0/1110031748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f5006c3d0 0x7f2f5006c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.392+0000 7f2f55189700 1 --2- 172.21.15.3:0/1110031748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f5006bd80 0x7f2f500d0940 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.392+0000 7f2f55189700 1 -- 172.21.15.3:0/1110031748 >> 172.21.15.3:0/1110031748 conn(0x7f2f500b3010 msgr2=0x7f2f500b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:57.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.392+0000 7f2f55189700 1 -- 172.21.15.3:0/1110031748 shutdown_connections 2024-05-08T21:21:57.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.392+0000 7f2f55189700 1 -- 172.21.15.3:0/1110031748 wait complete. 2024-05-08T21:21:57.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.393+0000 7f2f55189700 1 Processor -- start 2024-05-08T21:21:57.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.394+0000 7f2f55189700 1 -- start start 2024-05-08T21:21:57.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.394+0000 7f2f55189700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f5006bd80 0x7f2f50155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:57.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.394+0000 7f2f55189700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f5006c3d0 0x7f2f500c5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:57.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.394+0000 7f2f55189700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f5015da10 0x7f2f500b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:57.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.394+0000 7f2f55189700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f500b5ae0 con 0x7f2f5015da10 2024-05-08T21:21:57.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.394+0000 7f2f55189700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f500c8b80 con 0x7f2f5006bd80 2024-05-08T21:21:57.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.394+0000 7f2f55189700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f500c8e20 con 0x7f2f5006c3d0 2024-05-08T21:21:57.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.395+0000 7f2f4f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f5015da10 0x7f2f500b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:57.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.395+0000 7f2f4f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f5015da10 0x7f2f500b5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52294/0 (socket says 172.21.15.3:52294) 2024-05-08T21:21:57.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.395+0000 7f2f4f59e700 1 -- 172.21.15.3:0/2754356923 learned_addr learned my addr 172.21.15.3:0/2754356923 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:57.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.395+0000 7f2f4ed9d700 1 --2- 172.21.15.3:0/2754356923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f5006bd80 0x7f2f50155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:57.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.395+0000 7f2f4659c700 1 --2- 172.21.15.3:0/2754356923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f5006c3d0 0x7f2f500c5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:57.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.395+0000 7f2f4f59e700 1 -- 172.21.15.3:0/2754356923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f5006c3d0 msgr2=0x7f2f500c5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:57.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.395+0000 7f2f4f59e700 1 --2- 172.21.15.3:0/2754356923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f5006c3d0 0x7f2f500c5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.395+0000 7f2f4f59e700 1 -- 172.21.15.3:0/2754356923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f5006bd80 msgr2=0x7f2f50155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:57.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.395+0000 7f2f4f59e700 1 --2- 172.21.15.3:0/2754356923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f5006bd80 0x7f2f50155c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.395+0000 7f2f4f59e700 1 -- 172.21.15.3:0/2754356923 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f48009a70 con 0x7f2f5015da10 2024-05-08T21:21:57.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.395+0000 7f2f4f59e700 1 --2- 172.21.15.3:0/2754356923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f5015da10 0x7f2f500b5020 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2f4800efc0 tx=0x7f2f4800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:57.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.395+0000 7f2f47fff700 1 -- 172.21.15.3:0/2754356923 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4800ca90 con 0x7f2f5015da10 2024-05-08T21:21:57.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.395+0000 7f2f47fff700 1 -- 172.21.15.3:0/2754356923 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4800f040 con 0x7f2f5015da10 2024-05-08T21:21:57.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.395+0000 7f2f55189700 1 -- 172.21.15.3:0/2754356923 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f38009550 con 0x7f2f5015da10 2024-05-08T21:21:57.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.395+0000 7f2f47fff700 1 -- 172.21.15.3:0/2754356923 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4800ce10 con 0x7f2f5015da10 2024-05-08T21:21:57.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.396+0000 7f2f55189700 1 -- 172.21.15.3:0/2754356923 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f500c93e0 con 0x7f2f5015da10 2024-05-08T21:21:57.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.397+0000 7f2f55189700 1 -- 172.21.15.3:0/2754356923 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f500c9600 con 0x7f2f5015da10 2024-05-08T21:21:57.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.397+0000 7f2f47fff700 1 -- 172.21.15.3:0/2754356923 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2f48014480 con 0x7f2f5015da10 2024-05-08T21:21:57.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.397+0000 7f2f47fff700 1 --2- 172.21.15.3:0/2754356923 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2f3006ac90 0x7f2f3006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:57.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.398+0000 7f2f47fff700 1 -- 172.21.15.3:0/2754356923 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f480899e0 con 0x7f2f5015da10 2024-05-08T21:21:57.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.398+0000 7f2f4ed9d700 1 --2- 172.21.15.3:0/2754356923 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2f3006ac90 0x7f2f3006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:57.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.398+0000 7f2f4ed9d700 1 --2- 172.21.15.3:0/2754356923 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2f3006ac90 0x7f2f3006d150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2f40007fd0 tx=0x7f2f40007de0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:57.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.402+0000 7f2f47fff700 1 -- 172.21.15.3:0/2754356923 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2f48054bc0 con 0x7f2f5015da10 2024-05-08T21:21:57.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.624+0000 7f2f55189700 1 -- 172.21.15.3:0/2754356923 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f50068000 con 0x7f2f5015da10 2024-05-08T21:21:57.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.625+0000 7f2f47fff700 1 -- 172.21.15.3:0/2754356923 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15989 (secure 0 0 0) 0x7f2f48009d70 con 0x7f2f5015da10 2024-05-08T21:21:57.627 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:21:57.627 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:53.064342+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:21:57.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.627+0000 7f2f55189700 1 -- 172.21.15.3:0/2754356923 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2f3006ac90 msgr2=0x7f2f3006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:57.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.627+0000 7f2f55189700 1 --2- 172.21.15.3:0/2754356923 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2f3006ac90 0x7f2f3006d150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2f40007fd0 tx=0x7f2f40007de0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.627+0000 7f2f55189700 1 -- 172.21.15.3:0/2754356923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f5015da10 msgr2=0x7f2f500b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:57.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.627+0000 7f2f55189700 1 --2- 172.21.15.3:0/2754356923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f5015da10 0x7f2f500b5020 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2f4800efc0 tx=0x7f2f4800c3a0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.628+0000 7f2f55189700 1 -- 172.21.15.3:0/2754356923 shutdown_connections 2024-05-08T21:21:57.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.628+0000 7f2f55189700 1 --2- 172.21.15.3:0/2754356923 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2f3006ac90 0x7f2f3006d150 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.628+0000 7f2f55189700 1 --2- 172.21.15.3:0/2754356923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f5015da10 0x7f2f500b5020 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.628+0000 7f2f55189700 1 --2- 172.21.15.3:0/2754356923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f5006bd80 0x7f2f50155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.628+0000 7f2f55189700 1 --2- 172.21.15.3:0/2754356923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f5006c3d0 0x7f2f500c5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.628+0000 7f2f55189700 1 -- 172.21.15.3:0/2754356923 >> 172.21.15.3:0/2754356923 conn(0x7f2f500b3010 msgr2=0x7f2f500b2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:57.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.628+0000 7f2f55189700 1 -- 172.21.15.3:0/2754356923 shutdown_connections 2024-05-08T21:21:57.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.628+0000 7f2f55189700 1 -- 172.21.15.3:0/2754356923 wait complete. 2024-05-08T21:21:57.643 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-08T21:21:57.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:21:57.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.778+0000 7fb6f7273700 1 Processor -- start 2024-05-08T21:21:57.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.778+0000 7fb6f7273700 1 -- start start 2024-05-08T21:21:57.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.779+0000 7fb6f7273700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb6f0161b60 0x7fb6f006f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:57.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.779+0000 7fb6f7273700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f0166760 con 0x7fb6f0161f40 2024-05-08T21:21:57.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.779+0000 7fb6f7273700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f006e360 con 0x7fb6f00c3a00 2024-05-08T21:21:57.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.779+0000 7fb6f7273700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f006e640 con 0x7fb6f0161b60 2024-05-08T21:21:57.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.779+0000 7fb6f500f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb6f0161b60 0x7fb6f006f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:57.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.779+0000 7fb6f480e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb6f0161f40 0x7fb6f00c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:32984/0 (socket says 172.21.15.3:32984) 2024-05-08T21:21:57.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.779+0000 7fb6f480e700 1 -- 172.21.15.3:0/4211103747 learned_addr learned my addr 172.21.15.3:0/4211103747 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:57.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.779+0000 7fb6f500f700 1 --2- 172.21.15.3:0/4211103747 >> v2:172.21.15.3:3300/0 conn(0x7fb6f0161b60 0x7fb6f006f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52296/0 (socket says 172.21.15.3:52296) 2024-05-08T21:21:57.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.779+0000 7fb6f500f700 1 -- 172.21.15.3:0/4211103747 >> v1:172.21.15.186:6789/0 conn(0x7fb6f00c3a00 legacy=0x7fb6f01659a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:57.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.779+0000 7fb6f500f700 1 -- 172.21.15.3:0/4211103747 >> v1:172.21.15.3:6789/0 conn(0x7fb6f0161f40 legacy=0x7fb6f00c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:57.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.779+0000 7fb6f500f700 1 -- 172.21.15.3:0/4211103747 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f006e920 con 0x7fb6f0161b60 2024-05-08T21:21:57.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.779+0000 7fb6e7fff700 1 -- 172.21.15.3:0/4211103747 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044627778 0 0) 0x7fb6f0166760 con 0x7fb6f0161f40 2024-05-08T21:21:57.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.779+0000 7fb6f500f700 1 --2- 172.21.15.3:0/4211103747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6f0161b60 0x7fb6f006f120 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec009840 tx=0x7fb6ec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7915537869e28ec7 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:57.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.780+0000 7fb6e7fff700 1 -- 172.21.15.3:0/4211103747 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec004230 con 0x7fb6f0161b60 2024-05-08T21:21:57.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.780+0000 7fb6e7fff700 1 -- 172.21.15.3:0/4211103747 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ec004390 con 0x7fb6f0161b60 2024-05-08T21:21:57.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.780+0000 7fb6e7fff700 1 -- 172.21.15.3:0/4211103747 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec0036a0 con 0x7fb6f0161b60 2024-05-08T21:21:57.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.780+0000 7fb6f7273700 1 -- 172.21.15.3:0/4211103747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6f0161b60 msgr2=0x7fb6f006f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:57.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.780+0000 7fb6f7273700 1 --2- 172.21.15.3:0/4211103747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6f0161b60 0x7fb6f006f120 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec009840 tx=0x7fb6ec009b50 comp rx=0 tx=0).stop 2024-05-08T21:21:57.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.780+0000 7fb6f7273700 1 -- 172.21.15.3:0/4211103747 shutdown_connections 2024-05-08T21:21:57.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.780+0000 7fb6f7273700 1 --2- 172.21.15.3:0/4211103747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6f0161b60 0x7fb6f006f120 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.780+0000 7fb6f7273700 1 -- 172.21.15.3:0/4211103747 >> 172.21.15.3:0/4211103747 conn(0x7fb6f00d0320 msgr2=0x7fb6f00d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:57.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.780+0000 7fb6f7273700 1 -- 172.21.15.3:0/4211103747 shutdown_connections 2024-05-08T21:21:57.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.780+0000 7fb6f7273700 1 -- 172.21.15.3:0/4211103747 wait complete. 2024-05-08T21:21:57.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.782+0000 7fb6f7273700 1 Processor -- start 2024-05-08T21:21:57.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.782+0000 7fb6f7273700 1 -- start start 2024-05-08T21:21:57.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.782+0000 7fb6f7273700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6f00c3a00 0x7fb6f01f7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:57.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.782+0000 7fb6f7273700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6f0161b60 0x7fb6f00c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:57.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.782+0000 7fb6f7273700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6f0161f40 0x7fb6f00c5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:57.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.782+0000 7fb6f7273700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f00b3df0 con 0x7fb6f00c3a00 2024-05-08T21:21:57.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.782+0000 7fb6f7273700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f00b3f80 con 0x7fb6f0161f40 2024-05-08T21:21:57.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.782+0000 7fb6f7273700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f00c9480 con 0x7fb6f0161b60 2024-05-08T21:21:57.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.782+0000 7fb6f500f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6f00c3a00 0x7fb6f01f7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:57.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.782+0000 7fb6f500f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6f00c3a00 0x7fb6f01f7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52298/0 (socket says 172.21.15.3:52298) 2024-05-08T21:21:57.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.782+0000 7fb6f500f700 1 -- 172.21.15.3:0/3184903985 learned_addr learned my addr 172.21.15.3:0/3184903985 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:57.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.783+0000 7fb6f5810700 1 --2- 172.21.15.3:0/3184903985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6f0161f40 0x7fb6f00c5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:57.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.783+0000 7fb6f480e700 1 --2- 172.21.15.3:0/3184903985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6f0161b60 0x7fb6f00c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:57.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.783+0000 7fb6f500f700 1 -- 172.21.15.3:0/3184903985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6f0161b60 msgr2=0x7fb6f00c8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:57.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.783+0000 7fb6f500f700 1 --2- 172.21.15.3:0/3184903985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6f0161b60 0x7fb6f00c8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.783+0000 7fb6f500f700 1 -- 172.21.15.3:0/3184903985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6f0161f40 msgr2=0x7fb6f00c5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:57.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.783+0000 7fb6f500f700 1 --2- 172.21.15.3:0/3184903985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6f0161f40 0x7fb6f00c5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:57.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.783+0000 7fb6f500f700 1 -- 172.21.15.3:0/3184903985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ec0094a0 con 0x7fb6f00c3a00 2024-05-08T21:21:57.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.783+0000 7fb6f5810700 1 --2- 172.21.15.3:0/3184903985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6f0161f40 0x7fb6f00c5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:21:57.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.783+0000 7fb6f480e700 1 --2- 172.21.15.3:0/3184903985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6f0161b60 0x7fb6f00c8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:21:57.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.783+0000 7fb6f500f700 1 --2- 172.21.15.3:0/3184903985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6f00c3a00 0x7fb6f01f7030 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb6f006f6b0 tx=0x7fb6ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:57.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.783+0000 7fb6e5ffb700 1 -- 172.21.15.3:0/3184903985 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec011630 con 0x7fb6f00c3a00 2024-05-08T21:21:57.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.783+0000 7fb6e5ffb700 1 -- 172.21.15.3:0/3184903985 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ec01e070 con 0x7fb6f00c3a00 2024-05-08T21:21:57.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.783+0000 7fb6f7273700 1 -- 172.21.15.3:0/3184903985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f00c9760 con 0x7fb6f00c3a00 2024-05-08T21:21:57.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.783+0000 7fb6e5ffb700 1 -- 172.21.15.3:0/3184903985 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec0119f0 con 0x7fb6f00c3a00 2024-05-08T21:21:57.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.783+0000 7fb6f7273700 1 -- 172.21.15.3:0/3184903985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f00c9d20 con 0x7fb6f00c3a00 2024-05-08T21:21:57.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.785+0000 7fb6f7273700 1 -- 172.21.15.3:0/3184903985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f00b62f0 con 0x7fb6f00c3a00 2024-05-08T21:21:57.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.785+0000 7fb6e5ffb700 1 -- 172.21.15.3:0/3184903985 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb6ec011b50 con 0x7fb6f00c3a00 2024-05-08T21:21:57.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.786+0000 7fb6e5ffb700 1 --2- 172.21.15.3:0/3184903985 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb6d406ac90 0x7fb6d406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:57.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.786+0000 7fb6e5ffb700 1 -- 172.21.15.3:0/3184903985 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6ec08a6e0 con 0x7fb6f00c3a00 2024-05-08T21:21:57.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.786+0000 7fb6f480e700 1 --2- 172.21.15.3:0/3184903985 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb6d406ac90 0x7fb6d406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:57.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.786+0000 7fb6f480e700 1 --2- 172.21.15.3:0/3184903985 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb6d406ac90 0x7fb6d406d150 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb6f00ca460 tx=0x7fb6e0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:57.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:57.790+0000 7fb6e5ffb700 1 -- 172.21.15.3:0/3184903985 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb6ec055950 con 0x7fb6f00c3a00 2024-05-08T21:21:57.966 INFO:tasks.ceph.mds.d.smithi186.stderr:2024-05-08T21:21:57.965+0000 7fb4def6e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-08T21:21:58.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.009+0000 7fb6f7273700 1 -- 172.21.15.3:0/3184903985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6f0068000 con 0x7fb6f00c3a00 2024-05-08T21:21:58.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.010+0000 7fb6e5ffb700 1 -- 172.21.15.3:0/3184903985 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15989 (secure 0 0 0) 0x7fb6ec020090 con 0x7fb6f00c3a00 2024-05-08T21:21:58.012 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:21:58.012 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:53.064342+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:21:58.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.012+0000 7fb6f7273700 1 -- 172.21.15.3:0/3184903985 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb6d406ac90 msgr2=0x7fb6d406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.012+0000 7fb6f7273700 1 --2- 172.21.15.3:0/3184903985 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb6d406ac90 0x7fb6d406d150 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb6f00ca460 tx=0x7fb6e0009040 comp rx=0 tx=0).stop 2024-05-08T21:21:58.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.012+0000 7fb6f7273700 1 -- 172.21.15.3:0/3184903985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6f00c3a00 msgr2=0x7fb6f01f7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.012+0000 7fb6f7273700 1 --2- 172.21.15.3:0/3184903985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6f00c3a00 0x7fb6f01f7030 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb6f006f6b0 tx=0x7fb6ec004750 comp rx=0 tx=0).stop 2024-05-08T21:21:58.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.012+0000 7fb6f7273700 1 -- 172.21.15.3:0/3184903985 shutdown_connections 2024-05-08T21:21:58.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.012+0000 7fb6f7273700 1 --2- 172.21.15.3:0/3184903985 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb6d406ac90 0x7fb6d406d150 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.012+0000 7fb6f7273700 1 --2- 172.21.15.3:0/3184903985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6f0161f40 0x7fb6f00c5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.012+0000 7fb6f7273700 1 --2- 172.21.15.3:0/3184903985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6f00c3a00 0x7fb6f01f7030 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.012+0000 7fb6f7273700 1 --2- 172.21.15.3:0/3184903985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6f0161b60 0x7fb6f00c8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.012+0000 7fb6f7273700 1 -- 172.21.15.3:0/3184903985 >> 172.21.15.3:0/3184903985 conn(0x7fb6f00d0320 msgr2=0x7fb6f00afd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.012+0000 7fb6f7273700 1 -- 172.21.15.3:0/3184903985 shutdown_connections 2024-05-08T21:21:58.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.012+0000 7fb6f7273700 1 -- 172.21.15.3:0/3184903985 wait complete. 2024-05-08T21:21:58.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-08T21:21:58.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-08T21:21:58.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-08T21:21:58.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-08T21:21:58.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-08T21:21:58.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-08T21:21:58.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-05-08T21:21:58.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-05-08T21:21:58.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.269+0000 7f890f11f700 1 Processor -- start 2024-05-08T21:21:58.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.267+0000 7fb4e13ae700 1 Processor -- start 2024-05-08T21:21:58.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.267+0000 7fb4e13ae700 1 -- start start 2024-05-08T21:21:58.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.268+0000 7fb4e13ae700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb4dc098a80 0x7fb4dc06ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.268+0000 7fb4e13ae700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4dc06d4b0 con 0x7fb4dc0b1510 2024-05-08T21:21:58.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.268+0000 7fb4e13ae700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4dc06d790 con 0x7fb4dc06e4f0 2024-05-08T21:21:58.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.268+0000 7fb4e13ae700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4dc06da70 con 0x7fb4dc098a80 2024-05-08T21:21:58.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.269+0000 7f890f11f700 1 -- start start 2024-05-08T21:21:58.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.270+0000 7f890f11f700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8908142bc0 0x7f89080b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.270+0000 7f890f11f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f890806c080 0x7f89080cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.270+0000 7f890f11f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f890806ed70 con 0x7f89080cda40 2024-05-08T21:21:58.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.270+0000 7f890f11f700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89080ce570 con 0x7f8908142bc0 2024-05-08T21:21:58.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.270+0000 7f890f11f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89080ce850 con 0x7f890806c080 2024-05-08T21:21:58.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.270+0000 7fb4da7fc700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fb4dc06e4f0 0x7fb4dc0b0e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:44292/0 (socket says 172.21.15.3:44292) 2024-05-08T21:21:58.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.270+0000 7fb4da7fc700 1 -- 172.21.15.3:0/3874428306 learned_addr learned my addr 172.21.15.3:0/3874428306 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:58.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.270+0000 7f890cebb700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8908142bc0 0x7f89080b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.270+0000 7f890cebb700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8908142bc0 0x7f89080b2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:38312/0 (socket says 172.21.15.3:38312) 2024-05-08T21:21:58.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.270+0000 7f890cebb700 1 -- 172.21.15.3:0/1244462650 learned_addr learned my addr 172.21.15.3:0/1244462650 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.270+0000 7f890d6bc700 1 --2- 172.21.15.3:0/1244462650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f890806c080 0x7f89080cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.270+0000 7f890cebb700 1 -- 172.21.15.3:0/1244462650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f890806c080 msgr2=0x7f89080cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.270+0000 7f890cebb700 1 --2- 172.21.15.3:0/1244462650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f890806c080 0x7f89080cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.270+0000 7f890cebb700 1 -- 172.21.15.3:0/1244462650 >> v1:172.21.15.3:6789/0 conn(0x7f89080cda40 legacy=0x7f890806b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.270+0000 7f890cebb700 1 -- 172.21.15.3:0/1244462650 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890806ea00 con 0x7f8908142bc0 2024-05-08T21:21:58.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.271+0000 7f890cebb700 1 --2- 172.21.15.3:0/1244462650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8908142bc0 0x7f89080b2bf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f88f400bb90 tx=0x7f88f400bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2778a5ec25f60b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.272+0000 7f89077fe700 1 -- 172.21.15.3:0/1244462650 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f400dbe0 con 0x7f8908142bc0 2024-05-08T21:21:58.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.272+0000 7f89077fe700 1 -- 172.21.15.3:0/1244462650 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f400dd40 con 0x7f8908142bc0 2024-05-08T21:21:58.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.272+0000 7f89077fe700 1 -- 172.21.15.3:0/1244462650 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f4004270 con 0x7f8908142bc0 2024-05-08T21:21:58.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.272+0000 7fb4d9ffb700 1 -- 172.21.15.3:0/3874428306 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520697118 0 0) 0x7fb4dc06d4b0 con 0x7fb4dc0b1510 2024-05-08T21:21:58.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.272+0000 7fb4d9ffb700 1 -- 172.21.15.3:0/3874428306 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4c40035b0 con 0x7fb4dc0b1510 2024-05-08T21:21:58.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7fb4daffd700 1 --2- 172.21.15.3:0/3874428306 >> v2:172.21.15.3:3300/0 conn(0x7fb4dc098a80 0x7fb4dc06ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7fb4d9ffb700 1 -- 172.21.15.3:0/3874428306 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 856012575 0 0) 0x7fb4c40035b0 con 0x7fb4dc0b1510 2024-05-08T21:21:58.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7fb4d9ffb700 1 -- 172.21.15.3:0/3874428306 >> v2:172.21.15.3:3300/0 conn(0x7fb4dc098a80 msgr2=0x7fb4dc06ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7fb4d9ffb700 1 --2- 172.21.15.3:0/3874428306 >> v2:172.21.15.3:3300/0 conn(0x7fb4dc098a80 0x7fb4dc06ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7fb4d9ffb700 1 -- 172.21.15.3:0/3874428306 >> v1:172.21.15.186:6789/0 conn(0x7fb4dc06e4f0 legacy=0x7fb4dc0b0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7fb4d9ffb700 1 -- 172.21.15.3:0/3874428306 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc06dd50 con 0x7fb4dc0b1510 2024-05-08T21:21:58.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7fb4d9ffb700 1 -- 172.21.15.3:0/3874428306 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7fb4d40019f0 con 0x7fb4dc0b1510 2024-05-08T21:21:58.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7fb4d9ffb700 1 -- 172.21.15.3:0/3874428306 >> v1:172.21.15.3:6789/0 conn(0x7fb4dc0b1510 legacy=0x7fb4dc0ac3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7fb4d9ffb700 1 --2- 172.21.15.3:0/3874428306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4c4003bd0 0x7fb4c4003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7fb4d9ffb700 1 --2- 172.21.15.3:0/3874428306 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4c40047d0 0x7fb4c400b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7fb4d9ffb700 1 --2- 172.21.15.3:0/3874428306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4c400b690 0x7fb4c400db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7fb4d9ffb700 1 -- 172.21.15.3:0/3874428306 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc06da70 con 0x7fb4c400b690 2024-05-08T21:21:58.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7fb4d9ffb700 1 -- 172.21.15.3:0/3874428306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c400e480 con 0x7fb4c4003bd0 2024-05-08T21:21:58.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7fb4d9ffb700 1 -- 172.21.15.3:0/3874428306 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c400e640 con 0x7fb4c40047d0 2024-05-08T21:21:58.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7fb4daffd700 1 --2- 172.21.15.3:0/3874428306 >> v2:172.21.15.3:3300/0 conn(0x7fb4dc098a80 0x7fb4dc06ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:21:58.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7f890f11f700 1 -- 172.21.15.3:0/1244462650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8908142bc0 msgr2=0x7f89080b2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7f890f11f700 1 --2- 172.21.15.3:0/1244462650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8908142bc0 0x7f89080b2bf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f88f400bb90 tx=0x7f88f400bea0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7fb4db7fe700 1 --2- 172.21.15.3:0/3874428306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4c400b690 0x7fb4c400db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7f890f11f700 1 -- 172.21.15.3:0/1244462650 shutdown_connections 2024-05-08T21:21:58.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7f890f11f700 1 --2- 172.21.15.3:0/1244462650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f890806c080 0x7f89080cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7f890f11f700 1 --2- 172.21.15.3:0/1244462650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8908142bc0 0x7f89080b2bf0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.273+0000 7f890f11f700 1 -- 172.21.15.3:0/1244462650 >> 172.21.15.3:0/1244462650 conn(0x7f89080cd550 msgr2=0x7f89080713c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.274+0000 7f890f11f700 1 -- 172.21.15.3:0/1244462650 shutdown_connections 2024-05-08T21:21:58.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.274+0000 7fb4db7fe700 1 -- 172.21.15.3:0/3874428306 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4c40047d0 msgr2=0x7fb4c400b0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-08T21:21:58.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.274+0000 7fb4db7fe700 1 --2- 172.21.15.3:0/3874428306 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4c40047d0 0x7fb4c400b0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.274+0000 7fb4db7fe700 1 -- 172.21.15.3:0/3874428306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4c4003bd0 msgr2=0x7fb4c4003fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-08T21:21:58.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.274+0000 7fb4db7fe700 1 --2- 172.21.15.3:0/3874428306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4c4003bd0 0x7fb4c4003fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.274+0000 7fb4db7fe700 1 -- 172.21.15.3:0/3874428306 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc06dd50 con 0x7fb4c400b690 2024-05-08T21:21:58.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.274+0000 7fb4db7fe700 1 -- 172.21.15.3:0/3874428306 reap_dead start 2024-05-08T21:21:58.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.274+0000 7fb4db7fe700 1 --2- 172.21.15.3:0/3874428306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4c400b690 0x7fb4c400db50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400a970 tx=0x7fb4c4003860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bdb9191a8f68ae7 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.274+0000 7fb4d9ffb700 1 -- 172.21.15.3:0/3874428306 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4012620 con 0x7fb4c400b690 2024-05-08T21:21:58.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.274+0000 7fb4d9ffb700 1 -- 172.21.15.3:0/3874428306 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d4012780 con 0x7fb4c400b690 2024-05-08T21:21:58.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.274+0000 7fb4d9ffb700 1 -- 172.21.15.3:0/3874428306 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4012b30 con 0x7fb4c400b690 2024-05-08T21:21:58.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.274+0000 7f890f11f700 1 -- 172.21.15.3:0/1244462650 wait complete. 2024-05-08T21:21:58.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.276+0000 7f890f11f700 1 Processor -- start 2024-05-08T21:21:58.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.275+0000 7f15ac4d6700 1 Processor -- start 2024-05-08T21:21:58.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.275+0000 7f15ac4d6700 1 -- start start 2024-05-08T21:21:58.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.275+0000 7fb4e13ae700 1 -- 172.21.15.3:0/3874428306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4c400b690 msgr2=0x7fb4c400db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.275+0000 7fb4e13ae700 1 --2- 172.21.15.3:0/3874428306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4c400b690 0x7fb4c400db50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400a970 tx=0x7fb4c4003860 comp rx=0 tx=0).stop 2024-05-08T21:21:58.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.276+0000 7f15ac4d6700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f15a40cdac0 0x7f15a416c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.276+0000 7f15ac4d6700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a40b35a0 con 0x7f15a406c080 2024-05-08T21:21:58.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.276+0000 7f15ac4d6700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a40b3880 con 0x7f15a4161cc0 2024-05-08T21:21:58.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.276+0000 7f15ac4d6700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a40b3f40 con 0x7f15a40cdac0 2024-05-08T21:21:58.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.276+0000 7f890f11f700 1 -- start start 2024-05-08T21:21:58.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.276+0000 7f2fd78d2700 1 Processor -- start 2024-05-08T21:21:58.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.276+0000 7fb4e13ae700 1 -- 172.21.15.3:0/3874428306 shutdown_connections 2024-05-08T21:21:58.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.276+0000 7fb4e13ae700 1 --2- 172.21.15.3:0/3874428306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4c400b690 0x7fb4c400db50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.276+0000 7fb4e13ae700 1 -- 172.21.15.3:0/3874428306 >> 172.21.15.3:0/3874428306 conn(0x7fb4dc0b4690 msgr2=0x7fb4dc0b4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.277+0000 7f15a9a71700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f15a406c080 0x7f15a406d450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:33030/0 (socket says 172.21.15.3:33030) 2024-05-08T21:21:58.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.277+0000 7f15aaa73700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f15a40cdac0 0x7f15a416c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.277+0000 7f15a9a71700 1 -- 172.21.15.3:0/2827962235 learned_addr learned my addr 172.21.15.3:0/2827962235 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:58.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.277+0000 7f15aaa73700 1 -- 172.21.15.3:0/2827962235 >> v1:172.21.15.186:6789/0 conn(0x7f15a4161cc0 legacy=0x7f15a406b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.277+0000 7f15aaa73700 1 -- 172.21.15.3:0/2827962235 >> v1:172.21.15.3:6789/0 conn(0x7f15a406c080 legacy=0x7f15a406d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.277+0000 7f15aaa73700 1 -- 172.21.15.3:0/2827962235 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40b3b60 con 0x7f15a40cdac0 2024-05-08T21:21:58.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.277+0000 7f890f11f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89080cda40 0x7f8908156c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.277+0000 7f890f11f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8908142bc0 0x7f8908157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.277+0000 7f890f11f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89081577d0 0x7f89080b3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.277+0000 7f890f11f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890813f540 con 0x7f8908142bc0 2024-05-08T21:21:58.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.277+0000 7f890f11f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89081e53c0 con 0x7f89081577d0 2024-05-08T21:21:58.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.277+0000 7f890f11f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8908135b70 con 0x7f89080cda40 2024-05-08T21:21:58.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.278+0000 7f890d6bc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89081577d0 0x7f89080b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.278+0000 7f890d6bc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89081577d0 0x7f89080b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:38334/0 (socket says 172.21.15.3:38334) 2024-05-08T21:21:58.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.278+0000 7f890d6bc700 1 -- 172.21.15.3:0/74003060 learned_addr learned my addr 172.21.15.3:0/74003060 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.278+0000 7f890cebb700 1 --2- 172.21.15.3:0/74003060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89080cda40 0x7f8908156c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.278+0000 7f8907fff700 1 --2- 172.21.15.3:0/74003060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8908142bc0 0x7f8908157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.277+0000 7f2fd78d2700 1 -- start start 2024-05-08T21:21:58.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.278+0000 7f2fd78d2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fd015d7a0 0x7f2fd0161f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.277+0000 7f15aaa73700 1 --2- 172.21.15.3:0/2827962235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15a40cdac0 0x7f15a416c8b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009840 tx=0x7f15a0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50f812ef59676162 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.277+0000 7fb4e13ae700 1 -- 172.21.15.3:0/3874428306 shutdown_connections 2024-05-08T21:21:58.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.277+0000 7fb4e13ae700 1 -- 172.21.15.3:0/3874428306 wait complete. 2024-05-08T21:21:58.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.278+0000 7f2fd78d2700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd0162450 con 0x7f2fd015db80 2024-05-08T21:21:58.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.278+0000 7f2fd78d2700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd00cee00 con 0x7f2fd00d3f40 2024-05-08T21:21:58.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.278+0000 7f2fd78d2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd00b27e0 con 0x7f2fd015d7a0 2024-05-08T21:21:58.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.278+0000 7f2fd566e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fd015d7a0 0x7f2fd0161f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.278+0000 7f2fd566e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fd015d7a0 0x7f2fd0161f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.3:55688/0 (socket says 172.21.15.3:55688) 2024-05-08T21:21:58.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.278+0000 7f2fd566e700 1 -- 172.21.15.3:0/1392702690 learned_addr learned my addr 172.21.15.3:0/1392702690 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.279+0000 7ff6daf99700 1 Processor -- start 2024-05-08T21:21:58.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.279+0000 7f2fd566e700 1 -- 172.21.15.3:0/1392702690 >> v1:172.21.15.186:6789/0 conn(0x7f2fd00d3f40 legacy=0x7f2fd00b0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.279+0000 7f2fd566e700 1 -- 172.21.15.3:0/1392702690 >> v1:172.21.15.3:6789/0 conn(0x7f2fd015db80 legacy=0x7f2fd00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.279+0000 7f2fd566e700 1 -- 172.21.15.3:0/1392702690 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd00d4400 con 0x7f2fd015d7a0 2024-05-08T21:21:58.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.279+0000 7f15a9270700 1 -- 172.21.15.3:0/2827962235 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a0004230 con 0x7f15a40cdac0 2024-05-08T21:21:58.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.279+0000 7f15a9270700 1 -- 172.21.15.3:0/2827962235 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a0004390 con 0x7f15a40cdac0 2024-05-08T21:21:58.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.279+0000 7ff6daf99700 1 -- start start 2024-05-08T21:21:58.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.279+0000 7fb4e13ae700 1 Processor -- start 2024-05-08T21:21:58.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.279+0000 7f2fd566e700 1 --2- 172.21.15.3:0/1392702690 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fd015d7a0 0x7f2fd0161f10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc009920 tx=0x7f2fcc009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7fa0e83e3b92c2f server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.279+0000 7f8907fff700 1 -- 172.21.15.3:0/74003060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89080cda40 msgr2=0x7f8908156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.279+0000 7f8907fff700 1 --2- 172.21.15.3:0/74003060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89080cda40 0x7f8908156c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.279+0000 7f8907fff700 1 -- 172.21.15.3:0/74003060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89081577d0 msgr2=0x7f89080b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.279+0000 7f8907fff700 1 --2- 172.21.15.3:0/74003060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89081577d0 0x7f89080b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.279+0000 7f8907fff700 1 -- 172.21.15.3:0/74003060 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f400b7f0 con 0x7f8908142bc0 2024-05-08T21:21:58.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.280+0000 7f2b682dd700 1 Processor -- start 2024-05-08T21:21:58.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7f8907fff700 1 --2- 172.21.15.3:0/74003060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8908142bc0 0x7f8908157290 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f88fc00c3e0 tx=0x7f88fc00c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7f89057fa700 1 -- 172.21.15.3:0/74003060 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc0052d0 con 0x7f8908142bc0 2024-05-08T21:21:58.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.282+0000 7f89057fa700 1 -- 172.21.15.3:0/74003060 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88fc005450 con 0x7f8908142bc0 2024-05-08T21:21:58.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.282+0000 7f89057fa700 1 -- 172.21.15.3:0/74003060 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc0038c0 con 0x7f8908142bc0 2024-05-08T21:21:58.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.280+0000 7f2b682dd700 1 -- start start 2024-05-08T21:21:58.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7f2b682dd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b60142bc0 0x7f2b600b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7f2b682dd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2b600cda40 0x7f2b600cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7f2b682dd700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b6006ed70 con 0x7f2b6006b970 2024-05-08T21:21:58.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7f2b682dd700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b600ce570 con 0x7f2b600cda40 2024-05-08T21:21:58.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7f2b682dd700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b600ce850 con 0x7f2b60142bc0 2024-05-08T21:21:58.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7f2b6687a700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f2b6006b970 0x7f2b6006f200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:44314/0 (socket says 172.21.15.3:44314) 2024-05-08T21:21:58.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7f2b6687a700 1 -- 172.21.15.3:0/4219783973 learned_addr learned my addr 172.21.15.3:0/4219783973 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:58.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.282+0000 7f2b65878700 1 --2- 172.21.15.3:0/4219783973 >> v2:172.21.15.186:3300/0 conn(0x7f2b600cda40 0x7f2b600cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7f2fc7fff700 1 -- 172.21.15.3:0/1392702690 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc004230 con 0x7f2fd015d7a0 2024-05-08T21:21:58.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7f2fc7fff700 1 -- 172.21.15.3:0/1392702690 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fcc004390 con 0x7f2fd015d7a0 2024-05-08T21:21:58.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7f2fc7fff700 1 -- 172.21.15.3:0/1392702690 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc009e70 con 0x7f2fd015d7a0 2024-05-08T21:21:58.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.282+0000 7f2fd78d2700 1 -- 172.21.15.3:0/1392702690 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fd015d7a0 msgr2=0x7f2fd0161f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.282+0000 7f2fd78d2700 1 --2- 172.21.15.3:0/1392702690 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fd015d7a0 0x7f2fd0161f10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc009920 tx=0x7f2fcc009c30 comp rx=0 tx=0).stop 2024-05-08T21:21:58.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.280+0000 7f15a9270700 1 -- 172.21.15.3:0/2827962235 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a00036a0 con 0x7f15a40cdac0 2024-05-08T21:21:58.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7f15ac4d6700 1 -- 172.21.15.3:0/2827962235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15a40cdac0 msgr2=0x7f15a416c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7f15ac4d6700 1 --2- 172.21.15.3:0/2827962235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15a40cdac0 0x7f15a416c8b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009840 tx=0x7f15a0009b50 comp rx=0 tx=0).stop 2024-05-08T21:21:58.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.282+0000 7f15ac4d6700 1 -- 172.21.15.3:0/2827962235 shutdown_connections 2024-05-08T21:21:58.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.282+0000 7f15ac4d6700 1 --2- 172.21.15.3:0/2827962235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15a40cdac0 0x7f15a416c8b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.282+0000 7f15ac4d6700 1 -- 172.21.15.3:0/2827962235 >> 172.21.15.3:0/2827962235 conn(0x7f15a4071370 msgr2=0x7f15a4071780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.282+0000 7f15ac4d6700 1 -- 172.21.15.3:0/2827962235 shutdown_connections 2024-05-08T21:21:58.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.282+0000 7f15ac4d6700 1 -- 172.21.15.3:0/2827962235 wait complete. 2024-05-08T21:21:58.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.279+0000 7fb4e13ae700 1 -- start start 2024-05-08T21:21:58.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.280+0000 7fb4e13ae700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4c400b690 0x7fb4dc16d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.280+0000 7fb4e13ae700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4dc2151b0 0x7fb4dc0ad720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.280+0000 7fb4e13ae700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4dc215630 0x7fb4dc0adc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.280+0000 7fb4e13ae700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc16dda0 con 0x7fb4dc215630 2024-05-08T21:21:58.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.280+0000 7fb4e13ae700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc0cf1e0 con 0x7fb4c400b690 2024-05-08T21:21:58.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.280+0000 7fb4e13ae700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc0cf4c0 con 0x7fb4dc2151b0 2024-05-08T21:21:58.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7fb4daffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4c400b690 0x7fb4dc16d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7fb4da7fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4dc2151b0 0x7fb4dc0ad720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7fb4da7fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4dc2151b0 0x7fb4dc0ad720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.3:55696/0 (socket says 172.21.15.3:55696) 2024-05-08T21:21:58.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7fb4da7fc700 1 -- 172.21.15.3:0/4124529186 learned_addr learned my addr 172.21.15.3:0/4124529186 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7fb4da7fc700 1 -- 172.21.15.3:0/4124529186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4c400b690 msgr2=0x7fb4dc16d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7fb4da7fc700 1 --2- 172.21.15.3:0/4124529186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4c400b690 0x7fb4dc16d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7fb4da7fc700 1 -- 172.21.15.3:0/4124529186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4dc215630 msgr2=0x7fb4dc0adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7fb4da7fc700 1 --2- 172.21.15.3:0/4124529186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4dc215630 0x7fb4dc0adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7fb4da7fc700 1 -- 172.21.15.3:0/4124529186 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc0cf7a0 con 0x7fb4dc2151b0 2024-05-08T21:21:58.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.282+0000 7fb4da7fc700 1 --2- 172.21.15.3:0/4124529186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4dc2151b0 0x7fb4dc0ad720 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc06e8d0 tx=0x7fb4d000b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.280+0000 7ff6daf99700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff6d40d4040 0x7ff6d40b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.280+0000 7ff6daf99700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff6d40ce3f0 0x7ff6d40ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.280+0000 7ff6daf99700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d40d2930 con 0x7ff6d40d1ae0 2024-05-08T21:21:58.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.280+0000 7ff6daf99700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d40d0970 con 0x7ff6d40d4040 2024-05-08T21:21:58.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.280+0000 7ff6daf99700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d40b4e40 con 0x7ff6d40ce3f0 2024-05-08T21:21:58.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7ff6d3fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff6d40d4040 0x7ff6d40b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7ff6d3fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff6d40d4040 0x7ff6d40b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52326/0 (socket says 172.21.15.3:52326) 2024-05-08T21:21:58.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7ff6d3fff700 1 -- 172.21.15.3:0/654820653 learned_addr learned my addr 172.21.15.3:0/654820653 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7ff6d9536700 1 --2- 172.21.15.3:0/654820653 >> v2:172.21.15.186:3300/0 conn(0x7ff6d40ce3f0 0x7ff6d40ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7ff6d9536700 1 -- 172.21.15.3:0/654820653 >> v2:172.21.15.3:3300/0 conn(0x7ff6d40d4040 msgr2=0x7ff6d40b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7ff6d9536700 1 --2- 172.21.15.3:0/654820653 >> v2:172.21.15.3:3300/0 conn(0x7ff6d40d4040 0x7ff6d40b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7ff6d9536700 1 -- 172.21.15.3:0/654820653 >> v1:172.21.15.3:6789/0 conn(0x7ff6d40d1ae0 legacy=0x7ff6d40d1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7ff6d9536700 1 -- 172.21.15.3:0/654820653 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d40d0040 con 0x7ff6d40ce3f0 2024-05-08T21:21:58.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.281+0000 7ff6d9536700 1 --2- 172.21.15.3:0/654820653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6d40ce3f0 0x7ff6d40ce800 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff6d408d2e0 tx=0x7ff6cc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59777360fc935f68 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.282+0000 7fb4c37fe700 1 -- 172.21.15.3:0/4124529186 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0010850 con 0x7fb4dc2151b0 2024-05-08T21:21:58.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.282+0000 7fb4e13ae700 1 -- 172.21.15.3:0/4124529186 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4dc0cfa80 con 0x7fb4dc2151b0 2024-05-08T21:21:58.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.282+0000 7fb4e13ae700 1 -- 172.21.15.3:0/4124529186 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4dc214b50 con 0x7fb4dc2151b0 2024-05-08T21:21:58.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.283+0000 7f2fd78d2700 1 -- 172.21.15.3:0/1392702690 shutdown_connections 2024-05-08T21:21:58.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.283+0000 7f2fd78d2700 1 --2- 172.21.15.3:0/1392702690 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fd015d7a0 0x7f2fd0161f10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.283+0000 7f2fd78d2700 1 -- 172.21.15.3:0/1392702690 >> 172.21.15.3:0/1392702690 conn(0x7f2fd008afe0 msgr2=0x7f2fd00d3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.283+0000 7fb4c37fe700 1 -- 172.21.15.3:0/4124529186 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d000c040 con 0x7fb4dc2151b0 2024-05-08T21:21:58.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.283+0000 7fb4c37fe700 1 -- 172.21.15.3:0/4124529186 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0010c10 con 0x7fb4dc2151b0 2024-05-08T21:21:58.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.283+0000 7fb4e13ae700 1 -- 172.21.15.3:0/4124529186 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb4dc0cf630 con 0x7fb4dc2151b0 2024-05-08T21:21:58.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.282+0000 7f890f11f700 1 -- 172.21.15.3:0/74003060 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8908135e50 con 0x7f8908142bc0 2024-05-08T21:21:58.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.282+0000 7f890f11f700 1 -- 172.21.15.3:0/74003060 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8908136410 con 0x7f8908142bc0 2024-05-08T21:21:58.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.283+0000 7f2fd78d2700 1 -- 172.21.15.3:0/1392702690 shutdown_connections 2024-05-08T21:21:58.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.283+0000 7ff6d37fe700 1 -- 172.21.15.3:0/654820653 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc003c20 con 0x7ff6d40ce3f0 2024-05-08T21:21:58.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.283+0000 7ff6d37fe700 1 -- 172.21.15.3:0/654820653 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6cc01f070 con 0x7ff6d40ce3f0 2024-05-08T21:21:58.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.283+0000 7ff6d37fe700 1 -- 172.21.15.3:0/654820653 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc00b980 con 0x7ff6d40ce3f0 2024-05-08T21:21:58.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.284+0000 7f890f11f700 1 -- 172.21.15.3:0/74003060 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f88e8000fc0 con 0x7f8908142bc0 2024-05-08T21:21:58.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.284+0000 7f15ac4d6700 1 Processor -- start 2024-05-08T21:21:58.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.283+0000 7f2b66079700 1 --2- 172.21.15.3:0/4219783973 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b60142bc0 0x7f2b600b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.283+0000 7f2b65878700 1 -- 172.21.15.3:0/4219783973 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b60142bc0 msgr2=0x7f2b600b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.283+0000 7f2b65878700 1 --2- 172.21.15.3:0/4219783973 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b60142bc0 0x7f2b600b2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.283+0000 7f2b65878700 1 -- 172.21.15.3:0/4219783973 >> v1:172.21.15.186:6789/0 conn(0x7f2b6006b970 legacy=0x7f2b6006f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.283+0000 7f2b65878700 1 -- 172.21.15.3:0/4219783973 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6006ea00 con 0x7f2b600cda40 2024-05-08T21:21:58.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.283+0000 7f2fd78d2700 1 -- 172.21.15.3:0/1392702690 wait complete. 2024-05-08T21:21:58.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.284+0000 7ff6daf99700 1 -- 172.21.15.3:0/654820653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6d40ce3f0 msgr2=0x7ff6d40ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.284+0000 7ff6daf99700 1 --2- 172.21.15.3:0/654820653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6d40ce3f0 0x7ff6d40ce800 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff6d408d2e0 tx=0x7ff6cc00b330 comp rx=0 tx=0).stop 2024-05-08T21:21:58.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.284+0000 7f70f533b700 1 Processor -- start 2024-05-08T21:21:58.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.285+0000 7fb4c37fe700 1 -- 172.21.15.3:0/4124529186 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb4d000e070 con 0x7fb4dc2151b0 2024-05-08T21:21:58.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.285+0000 7ff6daf99700 1 -- 172.21.15.3:0/654820653 shutdown_connections 2024-05-08T21:21:58.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.285+0000 7ff6daf99700 1 --2- 172.21.15.3:0/654820653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6d40ce3f0 0x7ff6d40ce800 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.285+0000 7ff6daf99700 1 --2- 172.21.15.3:0/654820653 >> v2:172.21.15.3:3300/0 conn(0x7ff6d40d4040 0x7ff6d40b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.285+0000 7ff6daf99700 1 -- 172.21.15.3:0/654820653 >> 172.21.15.3:0/654820653 conn(0x7ff6d408afe0 msgr2=0x7ff6d40d3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.285+0000 7f2fd78d2700 1 Processor -- start 2024-05-08T21:21:58.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.285+0000 7fb4c37fe700 1 --2- 172.21.15.3:0/4124529186 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4c806ac90 0x7fb4c806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.285+0000 7fb4c37fe700 1 -- 172.21.15.3:0/4124529186 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4d0089370 con 0x7fb4dc2151b0 2024-05-08T21:21:58.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.285+0000 7fb4c37fe700 1 --2- 172.21.15.3:0/4124529186 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x7fb4c8072e30 0x7fb4c8075250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7fb4c37fe700 1 -- 172.21.15.3:0/4124529186 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb4c8075900 con 0x7fb4c8072e30 2024-05-08T21:21:58.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7fb4c37fe700 1 -- 172.21.15.3:0/4124529186 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4d0089790 con 0x7fb4dc2151b0 2024-05-08T21:21:58.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.284+0000 7f15ac4d6700 1 -- start start 2024-05-08T21:21:58.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.284+0000 7f70f533b700 1 -- start start 2024-05-08T21:21:58.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.285+0000 7f15ac4d6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15a406c080 0x7f15a415bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f15ac4d6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15a40cdac0 0x7f15a4158400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f15ac4d6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15a4161cc0 0x7f15a4156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f70f533b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70f0142bc0 0x7f70f00b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f70f533b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f70f00cda40 0x7f70f00cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f70f533b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f70f006b970 0x7f70f006be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f70f533b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70f006c340 con 0x7f70f006b970 2024-05-08T21:21:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f70f533b700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70f006ed70 con 0x7f70f00cda40 2024-05-08T21:21:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f15ac4d6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a41579b0 con 0x7f15a40cdac0 2024-05-08T21:21:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f15ac4d6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a4157380 con 0x7f15a406c080 2024-05-08T21:21:58.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f15ac4d6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a415c670 con 0x7f15a4161cc0 2024-05-08T21:21:58.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f70f533b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f00ce570 con 0x7f70f0142bc0 2024-05-08T21:21:58.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7fb4db7fe700 1 --2- 172.21.15.3:0/4124529186 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x7fb4c8072e30 0x7fb4c8075250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7fb4daffd700 1 --2- 172.21.15.3:0/4124529186 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4c806ac90 0x7fb4c806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f15aa272700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15a406c080 0x7f15a415bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f70ee7fc700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f70f00cda40 0x7f70f00cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f70ee7fc700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f70f00cda40 0x7f70f00cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:38374/0 (socket says 172.21.15.3:38374) 2024-05-08T21:21:58.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f70ee7fc700 1 -- 172.21.15.3:0/1931365513 learned_addr learned my addr 172.21.15.3:0/1931365513 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f70eeffd700 1 --2- 172.21.15.3:0/1931365513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70f0142bc0 0x7f70f00b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f15a9a71700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15a40cdac0 0x7f15a4158400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f15aa272700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15a406c080 0x7f15a415bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:38364/0 (socket says 172.21.15.3:38364) 2024-05-08T21:21:58.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f15aa272700 1 -- 172.21.15.3:0/1380565961 learned_addr learned my addr 172.21.15.3:0/1380565961 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f15aaa73700 1 --2- 172.21.15.3:0/1380565961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15a4161cc0 0x7f15a4156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f2b65878700 1 --2- 172.21.15.3:0/4219783973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b600cda40 0x7f2b600cdec0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c009840 tx=0x7f2b5c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68b32af8f2c98a9e server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f70ee7fc700 1 -- 172.21.15.3:0/1931365513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70f0142bc0 msgr2=0x7f70f00b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f70ee7fc700 1 --2- 172.21.15.3:0/1931365513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70f0142bc0 0x7f70f00b2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f70ee7fc700 1 -- 172.21.15.3:0/1931365513 >> v2:172.21.15.3:3300/0 conn(0x7f70f006b970 msgr2=0x7f70f006be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f70ee7fc700 1 --2- 172.21.15.3:0/1931365513 >> v2:172.21.15.3:3300/0 conn(0x7f70f006b970 0x7f70f006be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f70ee7fc700 1 -- 172.21.15.3:0/1931365513 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f00ce850 con 0x7f70f00cda40 2024-05-08T21:21:58.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.287+0000 7fb4db7fe700 1 --2- 172.21.15.3:0/4124529186 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x7fb4c8072e30 0x7fb4c8075250 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.287+0000 7ff6daf99700 1 -- 172.21.15.3:0/654820653 shutdown_connections 2024-05-08T21:21:58.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.287+0000 7f70ee7fc700 1 --2- 172.21.15.3:0/1931365513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f00cda40 0x7f70f00cdec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f70f0071010 tx=0x7f70e800d430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3391cfd0857e7bf7 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f15aa272700 1 -- 172.21.15.3:0/1380565961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15a4161cc0 msgr2=0x7f15a4156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f15aa272700 1 --2- 172.21.15.3:0/1380565961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15a4161cc0 0x7f15a4156960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f15aa272700 1 -- 172.21.15.3:0/1380565961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15a40cdac0 msgr2=0x7f15a4158400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f15aa272700 1 --2- 172.21.15.3:0/1380565961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15a40cdac0 0x7f15a4158400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.286+0000 7f15aa272700 1 -- 172.21.15.3:0/1380565961 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a00094a0 con 0x7f15a406c080 2024-05-08T21:21:58.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.287+0000 7fb4daffd700 1 --2- 172.21.15.3:0/4124529186 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4c806ac90 0x7fb4c806d150 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc006fd0 tx=0x7fb4cc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.287+0000 7ff6daf99700 1 -- 172.21.15.3:0/654820653 wait complete. 2024-05-08T21:21:58.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.287+0000 7fb4c37fe700 1 -- 172.21.15.3:0/4124529186 <== osd.6 v2:172.21.15.186:6808/47351 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fb4c8075900 con 0x7fb4c8072e30 2024-05-08T21:21:58.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.287+0000 7f2b65077700 1 -- 172.21.15.3:0/4219783973 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c004230 con 0x7f2b600cda40 2024-05-08T21:21:58.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.287+0000 7f2b65077700 1 -- 172.21.15.3:0/4219783973 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5c004390 con 0x7f2b600cda40 2024-05-08T21:21:58.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.287+0000 7f2b65077700 1 -- 172.21.15.3:0/4219783973 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c0036a0 con 0x7f2b600cda40 2024-05-08T21:21:58.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f2b682dd700 1 -- 172.21.15.3:0/4219783973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b600cda40 msgr2=0x7f2b600cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f2b682dd700 1 --2- 172.21.15.3:0/4219783973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b600cda40 0x7f2b600cdec0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c009840 tx=0x7f2b5c009b50 comp rx=0 tx=0).stop 2024-05-08T21:21:58.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f2b682dd700 1 -- 172.21.15.3:0/4219783973 shutdown_connections 2024-05-08T21:21:58.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f2b682dd700 1 --2- 172.21.15.3:0/4219783973 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b60142bc0 0x7f2b600b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f2b682dd700 1 --2- 172.21.15.3:0/4219783973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b600cda40 0x7f2b600cdec0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f2b682dd700 1 -- 172.21.15.3:0/4219783973 >> 172.21.15.3:0/4219783973 conn(0x7f2b600cd550 msgr2=0x7f2b600713c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f15aa272700 1 --2- 172.21.15.3:0/1380565961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15a406c080 0x7f15a415bfe0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f159400b4a0 tx=0x7f159400b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f2b682dd700 1 -- 172.21.15.3:0/4219783973 shutdown_connections 2024-05-08T21:21:58.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.287+0000 7f70edffb700 1 -- 172.21.15.3:0/1931365513 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8010070 con 0x7f70f00cda40 2024-05-08T21:21:58.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.287+0000 7f70edffb700 1 -- 172.21.15.3:0/1931365513 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e800db80 con 0x7f70f00cda40 2024-05-08T21:21:58.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.287+0000 7f70edffb700 1 -- 172.21.15.3:0/1931365513 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8016400 con 0x7f70f00cda40 2024-05-08T21:21:58.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f70f533b700 1 -- 172.21.15.3:0/1931365513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f00cda40 msgr2=0x7f70f00cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f70f533b700 1 --2- 172.21.15.3:0/1931365513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f00cda40 0x7f70f00cdec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f70f0071010 tx=0x7f70e800d430 comp rx=0 tx=0).stop 2024-05-08T21:21:58.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7ff6daf99700 1 Processor -- start 2024-05-08T21:21:58.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7ff6daf99700 1 -- start start 2024-05-08T21:21:58.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f89057fa700 1 -- 172.21.15.3:0/74003060 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f88fc003a20 con 0x7f8908142bc0 2024-05-08T21:21:58.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f89057fa700 1 --2- 172.21.15.3:0/74003060 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f88ec06ac90 0x7f88ec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f89057fa700 1 -- 172.21.15.3:0/74003060 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88fc0885a0 con 0x7f8908142bc0 2024-05-08T21:21:58.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f159affd700 1 -- 172.21.15.3:0/1380565961 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594011830 con 0x7f15a406c080 2024-05-08T21:21:58.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f89057fa700 1 --2- 172.21.15.3:0/74003060 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x7f88ec072eb0 0x7f88ec0752d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f89057fa700 1 -- 172.21.15.3:0/74003060 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88ec075980 con 0x7f88ec072eb0 2024-05-08T21:21:58.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f89057fa700 1 -- 172.21.15.3:0/74003060 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88fc088950 con 0x7f8908142bc0 2024-05-08T21:21:58.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f890cebb700 1 --2- 172.21.15.3:0/74003060 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f88ec06ac90 0x7f88ec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.287+0000 7f2fd78d2700 1 -- start start 2024-05-08T21:21:58.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f2fd78d2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2fd00d3f40 0x7f2fd00c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f2fd78d2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fd015d7a0 0x7f2fd00c7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f2fd78d2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd015db80 0x7f2fd0160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f2fd78d2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd00c79a0 con 0x7f2fd00d3f40 2024-05-08T21:21:58.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f2fd78d2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd00c7c80 con 0x7f2fd015db80 2024-05-08T21:21:58.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f2fd78d2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd00c7f60 con 0x7f2fd015d7a0 2024-05-08T21:21:58.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f2fd5e6f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd015db80 0x7f2fd0160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f2fd5e6f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd015db80 0x7f2fd0160730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:38382/0 (socket says 172.21.15.3:38382) 2024-05-08T21:21:58.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f2fd5e6f700 1 -- 172.21.15.3:0/2621787563 learned_addr learned my addr 172.21.15.3:0/2621787563 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f2fd5e6f700 1 -- 172.21.15.3:0/2621787563 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fd015d7a0 msgr2=0x7f2fd00c7150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-08T21:21:58.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f2fd5e6f700 1 --2- 172.21.15.3:0/2621787563 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fd015d7a0 0x7f2fd00c7150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f2fd5e6f700 1 -- 172.21.15.3:0/2621787563 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2fd00d3f40 msgr2=0x7f2fd00c8a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-08T21:21:58.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f2fd5e6f700 1 --2- 172.21.15.3:0/2621787563 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2fd00d3f40 0x7f2fd00c8a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f2fd5e6f700 1 -- 172.21.15.3:0/2621787563 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc009550 con 0x7f2fd015db80 2024-05-08T21:21:58.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f2fd5e6f700 1 --2- 172.21.15.3:0/2621787563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd015db80 0x7f2fd0160730 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800cb40 tx=0x7f2fc800cf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f159affd700 1 -- 172.21.15.3:0/1380565961 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1594011990 con 0x7f15a406c080 2024-05-08T21:21:58.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.289+0000 7f159affd700 1 -- 172.21.15.3:0/1380565961 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594011d40 con 0x7f15a406c080 2024-05-08T21:21:58.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.290+0000 7f70f533b700 1 -- 172.21.15.3:0/1931365513 shutdown_connections 2024-05-08T21:21:58.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.290+0000 7f70f533b700 1 --2- 172.21.15.3:0/1931365513 >> v2:172.21.15.3:3300/0 conn(0x7f70f006b970 0x7f70f006be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.290+0000 7f70f533b700 1 --2- 172.21.15.3:0/1931365513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70f0142bc0 0x7f70f00b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.290+0000 7f70f533b700 1 --2- 172.21.15.3:0/1931365513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f00cda40 0x7f70f00cdec0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.290+0000 7f70f533b700 1 -- 172.21.15.3:0/1931365513 >> 172.21.15.3:0/1931365513 conn(0x7f70f00cd550 msgr2=0x7f70f00713c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.290+0000 7f890cebb700 1 --2- 172.21.15.3:0/74003060 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f88ec06ac90 0x7f88ec06d150 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f88f4019040 tx=0x7f88f4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.290+0000 7ff6daf99700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6d40ce3f0 0x7ff6d41e3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.290+0000 7ff6daf99700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6d40d1ae0 0x7ff6d41da430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.290+0000 7ff6daf99700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6d40d4040 0x7ff6d41da970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.290+0000 7ff6daf99700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d41daeb0 con 0x7ff6d40d1ae0 2024-05-08T21:21:58.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.290+0000 7ff6daf99700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d40c8010 con 0x7ff6d40d4040 2024-05-08T21:21:58.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.290+0000 7ff6daf99700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d41dc8a0 con 0x7ff6d40ce3f0 2024-05-08T21:21:58.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.291+0000 7ff6d8d35700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6d40ce3f0 0x7ff6d41e3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.288+0000 7f2b682dd700 1 -- 172.21.15.3:0/4219783973 wait complete. 2024-05-08T21:21:58.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.290+0000 7f2b682dd700 1 Processor -- start 2024-05-08T21:21:58.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.290+0000 7f15ac4d6700 1 -- 172.21.15.3:0/1380565961 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a415c950 con 0x7f15a406c080 2024-05-08T21:21:58.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.290+0000 7f15ac4d6700 1 -- 172.21.15.3:0/1380565961 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a4157ee0 con 0x7f15a406c080 2024-05-08T21:21:58.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.291+0000 7ff6d3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6d40d1ae0 0x7ff6d41da430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.291+0000 7ff6d3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6d40d1ae0 0x7ff6d41da430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52372/0 (socket says 172.21.15.3:52372) 2024-05-08T21:21:58.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.291+0000 7ff6d3fff700 1 -- 172.21.15.3:0/4201326527 learned_addr learned my addr 172.21.15.3:0/4201326527 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.291+0000 7ff6d9536700 1 --2- 172.21.15.3:0/4201326527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6d40d4040 0x7ff6d41da970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.291+0000 7f2fc5ffb700 1 -- 172.21.15.3:0/2621787563 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8003c00 con 0x7f2fd015db80 2024-05-08T21:21:58.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.291+0000 7f2fc5ffb700 1 -- 172.21.15.3:0/2621787563 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc8003d60 con 0x7f2fd015db80 2024-05-08T21:21:58.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.291+0000 7f2fc5ffb700 1 -- 172.21.15.3:0/2621787563 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc800aaf0 con 0x7f2fd015db80 2024-05-08T21:21:58.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.291+0000 7ff6d3fff700 1 -- 172.21.15.3:0/4201326527 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6d40ce3f0 msgr2=0x7ff6d41e3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.291+0000 7ff6d3fff700 1 --2- 172.21.15.3:0/4201326527 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6d40ce3f0 0x7ff6d41e3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.291+0000 7ff6d3fff700 1 -- 172.21.15.3:0/4201326527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6d40d4040 msgr2=0x7ff6d41da970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.291+0000 7ff6d3fff700 1 --2- 172.21.15.3:0/4201326527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6d40d4040 0x7ff6d41da970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.291+0000 7ff6d3fff700 1 -- 172.21.15.3:0/4201326527 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc009a70 con 0x7ff6d40d1ae0 2024-05-08T21:21:58.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.291+0000 7ff6d9536700 1 --2- 172.21.15.3:0/4201326527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6d40d4040 0x7ff6d41da970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:21:58.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7ff6d3fff700 1 --2- 172.21.15.3:0/4201326527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6d40d1ae0 0x7ff6d41da430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800e7e0 tx=0x7ff6c800eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2fd78d2700 1 -- 172.21.15.3:0/2621787563 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd00c8240 con 0x7f2fd015db80 2024-05-08T21:21:58.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.291+0000 7f2b682dd700 1 -- start start 2024-05-08T21:21:58.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2b682dd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b600cda40 0x7f2b601ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2b682dd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b60142bc0 0x7f2b601633e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2b682dd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b600c5870 0x7f2b60158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2b682dd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b60164680 con 0x7f2b600c5870 2024-05-08T21:21:58.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2b682dd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b600c4e00 con 0x7f2b600cda40 2024-05-08T21:21:58.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2b682dd700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b60163a70 con 0x7f2b60142bc0 2024-05-08T21:21:58.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2b6687a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b600c5870 0x7f2b60158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2b6687a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b600c5870 0x7f2b60158ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52378/0 (socket says 172.21.15.3:52378) 2024-05-08T21:21:58.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2b6687a700 1 -- 172.21.15.3:0/430449891 learned_addr learned my addr 172.21.15.3:0/430449891 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2b66079700 1 --2- 172.21.15.3:0/430449891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b600cda40 0x7f2b601ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2b65878700 1 --2- 172.21.15.3:0/430449891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b60142bc0 0x7f2b601633e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2b66079700 1 -- 172.21.15.3:0/430449891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b60142bc0 msgr2=0x7f2b601633e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2b66079700 1 --2- 172.21.15.3:0/430449891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b60142bc0 0x7f2b601633e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2b66079700 1 -- 172.21.15.3:0/430449891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b600c5870 msgr2=0x7f2b60158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2b66079700 1 --2- 172.21.15.3:0/430449891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b600c5870 0x7f2b60158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2b66079700 1 -- 172.21.15.3:0/430449891 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c0094a0 con 0x7f2b600cda40 2024-05-08T21:21:58.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f2fd78d2700 1 -- 172.21.15.3:0/2621787563 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd00c7690 con 0x7f2fd015db80 2024-05-08T21:21:58.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7f15ac4d6700 1 -- 172.21.15.3:0/1380565961 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f15a406fc20 con 0x7f15a406c080 2024-05-08T21:21:58.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7ff6d17fa700 1 -- 172.21.15.3:0/4201326527 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c800c070 con 0x7ff6d40d1ae0 2024-05-08T21:21:58.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7ff6d17fa700 1 -- 172.21.15.3:0/4201326527 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c8013610 con 0x7ff6d40d1ae0 2024-05-08T21:21:58.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7ff6d17fa700 1 -- 172.21.15.3:0/4201326527 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c80139c0 con 0x7ff6d40d1ae0 2024-05-08T21:21:58.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.293+0000 7f2b66079700 1 --2- 172.21.15.3:0/430449891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b600cda40 0x7f2b601ffca0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2b5000d5c0 tx=0x7f2b5000d8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.293+0000 7f2b56ffd700 1 -- 172.21.15.3:0/430449891 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b500125f0 con 0x7f2b600cda40 2024-05-08T21:21:58.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.293+0000 7f2b56ffd700 1 -- 172.21.15.3:0/430449891 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b50012750 con 0x7f2b600cda40 2024-05-08T21:21:58.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.293+0000 7f2b56ffd700 1 -- 172.21.15.3:0/430449891 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b50012ae0 con 0x7f2b600cda40 2024-05-08T21:21:58.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.293+0000 7f2b682dd700 1 -- 172.21.15.3:0/430449891 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b60163d50 con 0x7f2b600cda40 2024-05-08T21:21:58.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.293+0000 7f2b682dd700 1 -- 172.21.15.3:0/430449891 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b60164310 con 0x7f2b600cda40 2024-05-08T21:21:58.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.293+0000 7f159affd700 1 -- 172.21.15.3:0/1380565961 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f159401a400 con 0x7f15a406c080 2024-05-08T21:21:58.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.294+0000 7f159affd700 1 --2- 172.21.15.3:0/1380565961 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f158806ad60 0x7f158806d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.294+0000 7f159affd700 1 -- 172.21.15.3:0/1380565961 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1594088f30 con 0x7f15a406c080 2024-05-08T21:21:58.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.294+0000 7f159affd700 1 --2- 172.21.15.3:0/1380565961 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x7f1588072f80 0x7f15880753a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.294+0000 7f159affd700 1 -- 172.21.15.3:0/1380565961 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1588075a50 con 0x7f1588072f80 2024-05-08T21:21:58.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.294+0000 7f159affd700 1 -- 172.21.15.3:0/1380565961 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f159408b550 con 0x7f15a406c080 2024-05-08T21:21:58.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.292+0000 7ff6daf99700 1 -- 172.21.15.3:0/4201326527 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d41dcb80 con 0x7ff6d40d1ae0 2024-05-08T21:21:58.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.293+0000 7ff6daf99700 1 -- 172.21.15.3:0/4201326527 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d40c72c0 con 0x7ff6d40d1ae0 2024-05-08T21:21:58.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.294+0000 7f15a9a71700 1 --2- 172.21.15.3:0/1380565961 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f158806ad60 0x7f158806d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.294+0000 7f15aaa73700 1 --2- 172.21.15.3:0/1380565961 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x7f1588072f80 0x7f15880753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.295+0000 7f15a9a71700 1 --2- 172.21.15.3:0/1380565961 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f158806ad60 0x7f158806d220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f15a4158c50 tx=0x7f159c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.295+0000 7f15aaa73700 1 --2- 172.21.15.3:0/1380565961 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x7f1588072f80 0x7f15880753a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.294+0000 7f2fd78d2700 1 -- 172.21.15.3:0/2621787563 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2fb0000fc0 con 0x7f2fd015db80 2024-05-08T21:21:58.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.296+0000 7f2fc5ffb700 1 -- 172.21.15.3:0/2621787563 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2fc800ac50 con 0x7f2fd015db80 2024-05-08T21:21:58.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.296+0000 7f890d6bc700 1 --2- 172.21.15.3:0/74003060 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x7f88ec072eb0 0x7f88ec0752d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.296+0000 7f70f533b700 1 -- 172.21.15.3:0/1931365513 shutdown_connections 2024-05-08T21:21:58.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.296+0000 7f159affd700 1 -- 172.21.15.3:0/1380565961 <== osd.4 v2:172.21.15.186:6810/47344 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f1588075a50 con 0x7f1588072f80 2024-05-08T21:21:58.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.296+0000 7f2b56ffd700 1 -- 172.21.15.3:0/430449891 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2b50012c40 con 0x7f2b600cda40 2024-05-08T21:21:58.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.296+0000 7f2b682dd700 1 -- 172.21.15.3:0/430449891 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2b6015d140 con 0x7f2b600cda40 2024-05-08T21:21:58.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.297+0000 7f2b56ffd700 1 --2- 172.21.15.3:0/430449891 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2b4406ac90 0x7f2b4406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.297+0000 7f2b56ffd700 1 -- 172.21.15.3:0/430449891 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b50089420 con 0x7f2b600cda40 2024-05-08T21:21:58.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.296+0000 7f2fc5ffb700 1 --2- 172.21.15.3:0/2621787563 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2fb406ac90 0x7f2fb406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.296+0000 7f2fc5ffb700 1 -- 172.21.15.3:0/2621787563 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fc80890b0 con 0x7f2fd015db80 2024-05-08T21:21:58.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.297+0000 7f2fc5ffb700 1 --2- 172.21.15.3:0/2621787563 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x7f2fb4072e30 0x7f2fb4075250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.297+0000 7f2fc5ffb700 1 -- 172.21.15.3:0/2621787563 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fb4075900 con 0x7f2fb4072e30 2024-05-08T21:21:58.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.297+0000 7f2fc5ffb700 1 -- 172.21.15.3:0/2621787563 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fc808b770 con 0x7f2fd015db80 2024-05-08T21:21:58.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.297+0000 7f2b56ffd700 1 --2- 172.21.15.3:0/430449891 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x7f2b44072e30 0x7f2b44075250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.297+0000 7f2b56ffd700 1 -- 172.21.15.3:0/430449891 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2b44075900 con 0x7f2b44072e30 2024-05-08T21:21:58.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.297+0000 7f2b65878700 1 --2- 172.21.15.3:0/430449891 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2b4406ac90 0x7f2b4406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.297+0000 7f2b6687a700 1 --2- 172.21.15.3:0/430449891 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x7f2b44072e30 0x7f2b44075250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.297+0000 7f2fd566e700 1 --2- 172.21.15.3:0/2621787563 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2fb406ac90 0x7f2fb406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.296+0000 7f70f533b700 1 -- 172.21.15.3:0/1931365513 wait complete. 2024-05-08T21:21:58.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.296+0000 7ff6d17fa700 1 -- 172.21.15.3:0/4201326527 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff6c8013b20 con 0x7ff6d40d1ae0 2024-05-08T21:21:58.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.297+0000 7ff6d17fa700 1 --2- 172.21.15.3:0/4201326527 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6b806ad60 0x7ff6b806d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.297+0000 7ff6d17fa700 1 -- 172.21.15.3:0/4201326527 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6c8089040 con 0x7ff6d40d1ae0 2024-05-08T21:21:58.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.298+0000 7f2fd566e700 1 --2- 172.21.15.3:0/2621787563 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2fb406ac90 0x7f2fb406d150 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc000c00 tx=0x7f2fcc008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.298+0000 7f890d6bc700 1 --2- 172.21.15.3:0/74003060 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x7f88ec072eb0 0x7f88ec0752d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.298+0000 7f2b6687a700 1 --2- 172.21.15.3:0/430449891 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x7f2b44072e30 0x7f2b44075250 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.298+0000 7f2b65878700 1 --2- 172.21.15.3:0/430449891 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2b4406ac90 0x7f2b4406d150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0090b0 tx=0x7f2b5c008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.299+0000 7f2b56ffd700 1 -- 172.21.15.3:0/430449891 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b50012ef0 con 0x7f2b600cda40 2024-05-08T21:21:58.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.299+0000 7f2b56ffd700 1 -- 172.21.15.3:0/430449891 <== osd.1 v2:172.21.15.3:6816/47119 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f2b44075900 con 0x7f2b44072e30 2024-05-08T21:21:58.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.298+0000 7f70f533b700 1 Processor -- start 2024-05-08T21:21:58.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.300+0000 7f89057fa700 1 -- 172.21.15.3:0/74003060 <== osd.0 v2:172.21.15.3:6824/47120 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f88ec075980 con 0x7f88ec072eb0 2024-05-08T21:21:58.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.299+0000 7f70f533b700 1 -- start start 2024-05-08T21:21:58.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.300+0000 7f70f533b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70f00cda40 0x7f70f0141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.300+0000 7f70f533b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f0142bc0 0x7f70f013c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.300+0000 7f70f533b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70f0142180 0x7f70f013fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.300+0000 7f70f533b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f01407a0 con 0x7f70f0142180 2024-05-08T21:21:58.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.300+0000 7f70f533b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f0140a80 con 0x7f70f0142bc0 2024-05-08T21:21:58.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.300+0000 7f70f533b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f0140d60 con 0x7f70f00cda40 2024-05-08T21:21:58.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.300+0000 7f70eeffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70f00cda40 0x7f70f0141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.300+0000 7f70eeffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70f00cda40 0x7f70f0141840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.3:55736/0 (socket says 172.21.15.3:55736) 2024-05-08T21:21:58.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.300+0000 7f70eeffd700 1 -- 172.21.15.3:0/462847136 learned_addr learned my addr 172.21.15.3:0/462847136 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.300+0000 7f70ee7fc700 1 --2- 172.21.15.3:0/462847136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f0142bc0 0x7f70f013c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.301+0000 7f70eeffd700 1 -- 172.21.15.3:0/462847136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f0142bc0 msgr2=0x7f70f013c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.301+0000 7f70eeffd700 1 --2- 172.21.15.3:0/462847136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f0142bc0 0x7f70f013c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.301+0000 7f70eeffd700 1 -- 172.21.15.3:0/462847136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70f0142180 msgr2=0x7f70f013fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.301+0000 7f70eeffd700 1 --2- 172.21.15.3:0/462847136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70f0142180 0x7f70f013fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.301+0000 7f70eeffd700 1 -- 172.21.15.3:0/462847136 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e800bd40 con 0x7f70f00cda40 2024-05-08T21:21:58.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.301+0000 7f70eeffd700 1 --2- 172.21.15.3:0/462847136 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70f00cda40 0x7f70f0141840 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f70e000e810 tx=0x7f70e000eb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.301+0000 7f70d77fe700 1 -- 172.21.15.3:0/462847136 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e0013610 con 0x7f70f00cda40 2024-05-08T21:21:58.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.301+0000 7f70d77fe700 1 -- 172.21.15.3:0/462847136 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e0013770 con 0x7f70f00cda40 2024-05-08T21:21:58.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.301+0000 7f70d77fe700 1 -- 172.21.15.3:0/462847136 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e0013b20 con 0x7f70f00cda40 2024-05-08T21:21:58.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.302+0000 7f70f533b700 1 -- 172.21.15.3:0/462847136 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f0141040 con 0x7f70f00cda40 2024-05-08T21:21:58.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.302+0000 7f70f533b700 1 -- 172.21.15.3:0/462847136 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f013c690 con 0x7f70f00cda40 2024-05-08T21:21:58.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.303+0000 7f70f533b700 1 -- 172.21.15.3:0/462847136 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f70f006e6f0 con 0x7f70f00cda40 2024-05-08T21:21:58.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.303+0000 7ff4dbd86700 1 Processor -- start 2024-05-08T21:21:58.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.300+0000 7ff6d8d35700 1 --2- 172.21.15.3:0/4201326527 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6b806ad60 0x7ff6b806d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.300+0000 7ff6daf99700 1 --2- 172.21.15.3:0/4201326527 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x7ff6b40015e0 0x7ff6b4003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.300+0000 7ff6daf99700 1 -- 172.21.15.3:0/4201326527 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6b4006bd0 con 0x7ff6b40015e0 2024-05-08T21:21:58.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.300+0000 7ff6d9536700 1 --2- 172.21.15.3:0/4201326527 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x7ff6b40015e0 0x7ff6b4003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.301+0000 7ff6d9536700 1 --2- 172.21.15.3:0/4201326527 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x7ff6b40015e0 0x7ff6b4003aa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.304+0000 7ff6d17fa700 1 -- 172.21.15.3:0/4201326527 <== osd.7 v2:172.21.15.186:6809/47341 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7ff6b4006bd0 con 0x7ff6b40015e0 2024-05-08T21:21:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.305+0000 7f70d77fe700 1 -- 172.21.15.3:0/462847136 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f70e0013c80 con 0x7f70f00cda40 2024-05-08T21:21:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.305+0000 7f2fd4e6d700 1 --2- 172.21.15.3:0/2621787563 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x7f2fb4072e30 0x7f2fb4075250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.305+0000 7f2fd4e6d700 1 --2- 172.21.15.3:0/2621787563 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x7f2fb4072e30 0x7f2fb4075250 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.305+0000 7f70d77fe700 1 --2- 172.21.15.3:0/462847136 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f70dc06ac90 0x7f70dc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.304+0000 7ff4dbd86700 1 -- start start 2024-05-08T21:21:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.305+0000 7ff4dbd86700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4d415a770 0x7ff4d40ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.305+0000 7ff4dbd86700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff4d40cea20 0x7ff4d40cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.305+0000 7ff4dbd86700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d40cf790 con 0x7ff4d40ccbd0 2024-05-08T21:21:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.305+0000 7ff4dbd86700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d40caa30 con 0x7ff4d40cea20 2024-05-08T21:21:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.305+0000 7ff4dbd86700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d40cfa70 con 0x7ff4d415a770 2024-05-08T21:21:58.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.305+0000 7f70d77fe700 1 -- 172.21.15.3:0/462847136 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f70e0089290 con 0x7f70f00cda40 2024-05-08T21:21:58.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.306+0000 7f70ee7fc700 1 --2- 172.21.15.3:0/462847136 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f70dc06ac90 0x7f70dc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.306+0000 7f70d77fe700 1 --2- 172.21.15.3:0/462847136 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x7f70dc072eb0 0x7f70dc0752d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.306+0000 7f70d77fe700 1 -- 172.21.15.3:0/462847136 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70dc075980 con 0x7f70dc072eb0 2024-05-08T21:21:58.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.306+0000 7f70d77fe700 1 -- 172.21.15.3:0/462847136 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70e0089640 con 0x7f70f00cda40 2024-05-08T21:21:58.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.306+0000 7f70ee7fc700 1 --2- 172.21.15.3:0/462847136 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f70dc06ac90 0x7f70dc06d150 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f70e8000f80 tx=0x7f70e8012000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.308+0000 7f2fc5ffb700 1 -- 172.21.15.3:0/2621787563 <== osd.2 v2:172.21.15.3:6808/47117 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f2fb4075900 con 0x7f2fb4072e30 2024-05-08T21:21:58.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.309+0000 7f70ef7fe700 1 --2- 172.21.15.3:0/462847136 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x7f70dc072eb0 0x7f70dc0752d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.309+0000 7f70ef7fe700 1 --2- 172.21.15.3:0/462847136 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x7f70dc072eb0 0x7f70dc0752d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.312+0000 7f70d77fe700 1 -- 172.21.15.3:0/462847136 <== osd.3 v2:172.21.15.3:6800/47118 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f70dc075980 con 0x7f70dc072eb0 2024-05-08T21:21:58.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.308+0000 7ff4d9b22700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4d415a770 0x7ff4d40ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.309+0000 7ff4d9321700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff4d40cea20 0x7ff4d40cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.309+0000 7ff4d9321700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff4d40cea20 0x7ff4d40cee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:38424/0 (socket says 172.21.15.3:38424) 2024-05-08T21:21:58.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.309+0000 7ff4d9321700 1 -- 172.21.15.3:0/590828265 learned_addr learned my addr 172.21.15.3:0/590828265 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.311+0000 7ff4d9321700 1 -- 172.21.15.3:0/590828265 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4d415a770 msgr2=0x7ff4d40ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.311+0000 7ff4d9321700 1 --2- 172.21.15.3:0/590828265 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4d415a770 0x7ff4d40ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.311+0000 7ff4d9321700 1 -- 172.21.15.3:0/590828265 >> v1:172.21.15.3:6789/0 conn(0x7ff4d40ccbd0 legacy=0x7ff4d40cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.311+0000 7ff4d9321700 1 -- 172.21.15.3:0/590828265 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d40cfd50 con 0x7ff4d40cea20 2024-05-08T21:21:58.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.311+0000 7ff4d9321700 1 --2- 172.21.15.3:0/590828265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4d40cea20 0x7ff4d40cee80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4010bb0 tx=0x7ff4c4010ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f42029c906ce32a server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.312+0000 7ff4d8b20700 1 -- 172.21.15.3:0/590828265 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4004030 con 0x7ff4d40cea20 2024-05-08T21:21:58.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.312+0000 7ff4d8b20700 1 -- 172.21.15.3:0/590828265 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c4004190 con 0x7ff4d40cea20 2024-05-08T21:21:58.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.312+0000 7ff4d8b20700 1 -- 172.21.15.3:0/590828265 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4004520 con 0x7ff4d40cea20 2024-05-08T21:21:58.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.316+0000 7ff4dbd86700 1 -- 172.21.15.3:0/590828265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4d40cea20 msgr2=0x7ff4d40cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.316+0000 7ff4dbd86700 1 --2- 172.21.15.3:0/590828265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4d40cea20 0x7ff4d40cee80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4010bb0 tx=0x7ff4c4010ec0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.316+0000 7ff4dbd86700 1 -- 172.21.15.3:0/590828265 shutdown_connections 2024-05-08T21:21:58.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.316+0000 7ff4dbd86700 1 --2- 172.21.15.3:0/590828265 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4d415a770 0x7ff4d40ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.316+0000 7ff4dbd86700 1 --2- 172.21.15.3:0/590828265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4d40cea20 0x7ff4d40cee80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.316+0000 7ff4dbd86700 1 -- 172.21.15.3:0/590828265 >> 172.21.15.3:0/590828265 conn(0x7ff4d40b1f80 msgr2=0x7ff4d40b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.316+0000 7ff4dbd86700 1 -- 172.21.15.3:0/590828265 shutdown_connections 2024-05-08T21:21:58.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.316+0000 7ff4dbd86700 1 -- 172.21.15.3:0/590828265 wait complete. 2024-05-08T21:21:58.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.319+0000 7ff4dbd86700 1 Processor -- start 2024-05-08T21:21:58.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.319+0000 7ff4dbd86700 1 -- start start 2024-05-08T21:21:58.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.320+0000 7ff4dbd86700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4d40ccbd0 0x7ff4d4158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.320+0000 7ff4dbd86700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4d40cea20 0x7ff4d414bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.320+0000 7ff4dbd86700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4d415a770 0x7ff4d41537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.320+0000 7ff4dbd86700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d414c120 con 0x7ff4d40cea20 2024-05-08T21:21:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.320+0000 7ff4dbd86700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d4159750 con 0x7ff4d40ccbd0 2024-05-08T21:21:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.320+0000 7ff4dbd86700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d4158bd0 con 0x7ff4d415a770 2024-05-08T21:21:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.323+0000 7ff4d9321700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4d40cea20 0x7ff4d414bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.323+0000 7ff4d9321700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4d40cea20 0x7ff4d414bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52386/0 (socket says 172.21.15.3:52386) 2024-05-08T21:21:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.323+0000 7ff4d9321700 1 -- 172.21.15.3:0/2107120098 learned_addr learned my addr 172.21.15.3:0/2107120098 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.323+0000 7ff4d9321700 1 -- 172.21.15.3:0/2107120098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4d415a770 msgr2=0x7ff4d41537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.323+0000 7ff4d9321700 1 --2- 172.21.15.3:0/2107120098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4d415a770 0x7ff4d41537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.323+0000 7ff4d9321700 1 -- 172.21.15.3:0/2107120098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4d40ccbd0 msgr2=0x7ff4d4158550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-08T21:21:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.323+0000 7ff4d9321700 1 --2- 172.21.15.3:0/2107120098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4d40ccbd0 0x7ff4d4158550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.323+0000 7ff4d9321700 1 -- 172.21.15.3:0/2107120098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c4010810 con 0x7ff4d40cea20 2024-05-08T21:21:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.324+0000 7ff4d9321700 1 --2- 172.21.15.3:0/2107120098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4d40cea20 0x7ff4d414bbe0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4010f00 tx=0x7ff4c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.325+0000 7f2b682dd700 1 -- 172.21.15.3:0/430449891 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2b60143830 con 0x7f2b44072e30 2024-05-08T21:21:58.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.324+0000 7ff4ca7fc700 1 -- 172.21.15.3:0/2107120098 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4003ab0 con 0x7ff4d40cea20 2024-05-08T21:21:58.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.324+0000 7ff4ca7fc700 1 -- 172.21.15.3:0/2107120098 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c4003c10 con 0x7ff4d40cea20 2024-05-08T21:21:58.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.324+0000 7ff4ca7fc700 1 -- 172.21.15.3:0/2107120098 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c400be80 con 0x7ff4d40cea20 2024-05-08T21:21:58.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.324+0000 7ff4dbd86700 1 -- 172.21.15.3:0/2107120098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d4158eb0 con 0x7ff4d40cea20 2024-05-08T21:21:58.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.324+0000 7ff4dbd86700 1 -- 172.21.15.3:0/2107120098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d4159470 con 0x7ff4d40cea20 2024-05-08T21:21:58.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.321+0000 7ff6d8d35700 1 --2- 172.21.15.3:0/4201326527 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6b806ad60 0x7ff6b806d220 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0007fd0 tx=0x7ff6c0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.328+0000 7f2b56ffd700 1 -- 172.21.15.3:0/430449891 <== osd.1 v2:172.21.15.3:6816/47119 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2b58002030 con 0x7f2b44072e30 2024-05-08T21:21:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.329+0000 7f2b682dd700 1 -- 172.21.15.3:0/430449891 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x7f2b44072e30 msgr2=0x7f2b44075250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.329+0000 7f2b682dd700 1 --2- 172.21.15.3:0/430449891 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x7f2b44072e30 0x7f2b44075250 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.329+0000 7f2b682dd700 1 -- 172.21.15.3:0/430449891 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2b4406ac90 msgr2=0x7f2b4406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.329+0000 7f2b682dd700 1 --2- 172.21.15.3:0/430449891 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2b4406ac90 0x7f2b4406d150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0090b0 tx=0x7f2b5c008030 comp rx=0 tx=0).stop 2024-05-08T21:21:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.329+0000 7f2b682dd700 1 -- 172.21.15.3:0/430449891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b600cda40 msgr2=0x7f2b601ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.329+0000 7f2b682dd700 1 --2- 172.21.15.3:0/430449891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b600cda40 0x7f2b601ffca0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2b5000d5c0 tx=0x7f2b5000d8d0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.329+0000 7f2b6687a700 1 -- 172.21.15.3:0/430449891 reap_dead start 2024-05-08T21:21:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.328+0000 7ff4ca7fc700 1 -- 172.21.15.3:0/2107120098 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff4c400e3f0 con 0x7ff4d40cea20 2024-05-08T21:21:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.329+0000 7ff4ca7fc700 1 --2- 172.21.15.3:0/2107120098 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff4b806ad90 0x7ff4b806d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.329+0000 7ff4ca7fc700 1 -- 172.21.15.3:0/2107120098 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4c408a570 con 0x7ff4d40cea20 2024-05-08T21:21:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.329+0000 7f15ac4d6700 1 -- 172.21.15.3:0/1380565961 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f15a406cc60 con 0x7f1588072f80 2024-05-08T21:21:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.330+0000 7f2b682dd700 1 -- 172.21.15.3:0/430449891 shutdown_connections 2024-05-08T21:21:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.330+0000 7f2b682dd700 1 -- 172.21.15.3:0/430449891 >> 172.21.15.3:0/430449891 conn(0x7f2b6006b970 msgr2=0x7f2b60159ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.330+0000 7f2b682dd700 1 -- 172.21.15.3:0/430449891 shutdown_connections 2024-05-08T21:21:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.330+0000 7f2b682dd700 1 -- 172.21.15.3:0/430449891 wait complete. 2024-05-08T21:21:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.330+0000 7ff4dbd86700 1 --2- 172.21.15.3:0/2107120098 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x7ff4bc0015e0 0x7ff4bc003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.330+0000 7ff4dbd86700 1 -- 172.21.15.3:0/2107120098 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4bc006bd0 con 0x7ff4bc0015e0 2024-05-08T21:21:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.330+0000 7fb4e13ae700 1 -- 172.21.15.3:0/4124529186 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb4dc098e60 con 0x7fb4c8072e30 2024-05-08T21:21:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.331+0000 7fb4c37fe700 1 -- 172.21.15.3:0/4124529186 <== osd.6 v2:172.21.15.186:6808/47351 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb4d4012420 con 0x7fb4c8072e30 2024-05-08T21:21:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.331+0000 7f159affd700 1 -- 172.21.15.3:0/1380565961 <== osd.4 v2:172.21.15.186:6810/47344 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f15a406cc60 con 0x7f1588072f80 2024-05-08T21:21:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.332+0000 7fb4e13ae700 1 -- 172.21.15.3:0/4124529186 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x7fb4c8072e30 msgr2=0x7fb4c8075250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.332+0000 7fb4e13ae700 1 --2- 172.21.15.3:0/4124529186 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x7fb4c8072e30 0x7fb4c8075250 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.332+0000 7fb4e13ae700 1 -- 172.21.15.3:0/4124529186 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4c806ac90 msgr2=0x7fb4c806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.332+0000 7fb4e13ae700 1 --2- 172.21.15.3:0/4124529186 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4c806ac90 0x7fb4c806d150 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc006fd0 tx=0x7fb4cc008040 comp rx=0 tx=0).stop 2024-05-08T21:21:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.332+0000 7fb4e13ae700 1 -- 172.21.15.3:0/4124529186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4dc2151b0 msgr2=0x7fb4dc0ad720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.332+0000 7fb4e13ae700 1 --2- 172.21.15.3:0/4124529186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4dc2151b0 0x7fb4dc0ad720 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc06e8d0 tx=0x7fb4d000b9d0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.332+0000 7fb4db7fe700 1 -- 172.21.15.3:0/4124529186 reap_dead start 2024-05-08T21:21:58.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.332+0000 7fb4e13ae700 1 -- 172.21.15.3:0/4124529186 shutdown_connections 2024-05-08T21:21:58.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.332+0000 7fb4e13ae700 1 -- 172.21.15.3:0/4124529186 >> 172.21.15.3:0/4124529186 conn(0x7fb4dc0b4690 msgr2=0x7fb4dc09af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.334+0000 7fb4e13ae700 1 -- 172.21.15.3:0/4124529186 shutdown_connections 2024-05-08T21:21:58.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.334+0000 7fb4e13ae700 1 -- 172.21.15.3:0/4124529186 wait complete. 2024-05-08T21:21:58.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.334+0000 7f15ac4d6700 1 -- 172.21.15.3:0/1380565961 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x7f1588072f80 msgr2=0x7f15880753a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.334+0000 7f15ac4d6700 1 --2- 172.21.15.3:0/1380565961 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x7f1588072f80 0x7f15880753a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.334+0000 7f15ac4d6700 1 -- 172.21.15.3:0/1380565961 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f158806ad60 msgr2=0x7f158806d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.334+0000 7f15ac4d6700 1 --2- 172.21.15.3:0/1380565961 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f158806ad60 0x7f158806d220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f15a4158c50 tx=0x7f159c009040 comp rx=0 tx=0).stop 2024-05-08T21:21:58.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.334+0000 7f15ac4d6700 1 -- 172.21.15.3:0/1380565961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15a406c080 msgr2=0x7f15a415bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.334+0000 7f15ac4d6700 1 --2- 172.21.15.3:0/1380565961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15a406c080 0x7f15a415bfe0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f159400b4a0 tx=0x7f159400b7b0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.334+0000 7f15aaa73700 1 -- 172.21.15.3:0/1380565961 reap_dead start 2024-05-08T21:21:58.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.334+0000 7f15ac4d6700 1 -- 172.21.15.3:0/1380565961 shutdown_connections 2024-05-08T21:21:58.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.334+0000 7f15ac4d6700 1 -- 172.21.15.3:0/1380565961 >> 172.21.15.3:0/1380565961 conn(0x7f15a4071370 msgr2=0x7f15a40b0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.336+0000 7ff4d9b22700 1 --2- 172.21.15.3:0/2107120098 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff4b806ad90 0x7ff4b806d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.336+0000 7f2fd78d2700 1 -- 172.21.15.3:0/2621787563 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2fb0002d40 con 0x7f2fb4072e30 2024-05-08T21:21:58.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.336+0000 7f890f11f700 1 -- 172.21.15.3:0/74003060 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f88e8002d40 con 0x7f88ec072eb0 2024-05-08T21:21:58.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.336+0000 7f15ac4d6700 1 -- 172.21.15.3:0/1380565961 shutdown_connections 2024-05-08T21:21:58.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.336+0000 7f15ac4d6700 1 -- 172.21.15.3:0/1380565961 wait complete. 2024-05-08T21:21:58.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.337+0000 7ff4da323700 1 --2- 172.21.15.3:0/2107120098 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x7ff4bc0015e0 0x7ff4bc003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.338+0000 7f89057fa700 1 -- 172.21.15.3:0/74003060 <== osd.0 v2:172.21.15.3:6824/47120 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f88e8002d40 con 0x7f88ec072eb0 2024-05-08T21:21:58.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.338+0000 7f890f11f700 1 -- 172.21.15.3:0/74003060 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x7f88ec072eb0 msgr2=0x7f88ec0752d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.338+0000 7f890f11f700 1 --2- 172.21.15.3:0/74003060 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x7f88ec072eb0 0x7f88ec0752d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.338+0000 7f890f11f700 1 -- 172.21.15.3:0/74003060 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f88ec06ac90 msgr2=0x7f88ec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.338+0000 7f890f11f700 1 --2- 172.21.15.3:0/74003060 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f88ec06ac90 0x7f88ec06d150 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f88f4019040 tx=0x7f88f4008040 comp rx=0 tx=0).stop 2024-05-08T21:21:58.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.338+0000 7f890f11f700 1 -- 172.21.15.3:0/74003060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8908142bc0 msgr2=0x7f8908157290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.338+0000 7f890f11f700 1 --2- 172.21.15.3:0/74003060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8908142bc0 0x7f8908157290 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f88fc00c3e0 tx=0x7f88fc00c6f0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.339+0000 7f890d6bc700 1 -- 172.21.15.3:0/74003060 reap_dead start 2024-05-08T21:21:58.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.339+0000 7f890f11f700 1 -- 172.21.15.3:0/74003060 shutdown_connections 2024-05-08T21:21:58.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.339+0000 7f890f11f700 1 -- 172.21.15.3:0/74003060 >> 172.21.15.3:0/74003060 conn(0x7f890806c080 msgr2=0x7f890806afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.339+0000 7ff4da323700 1 --2- 172.21.15.3:0/2107120098 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x7ff4bc0015e0 0x7ff4bc003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.339+0000 7ff4d9b22700 1 --2- 172.21.15.3:0/2107120098 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff4b806ad90 0x7ff4b806d250 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000b850 tx=0x7ff4d0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.339+0000 7f890f11f700 1 -- 172.21.15.3:0/74003060 shutdown_connections 2024-05-08T21:21:58.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.339+0000 7f890f11f700 1 -- 172.21.15.3:0/74003060 wait complete. 2024-05-08T21:21:58.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.340+0000 7ff4ca7fc700 1 -- 172.21.15.3:0/2107120098 <== osd.5 v2:172.21.15.186:6800/47340 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7ff4bc006bd0 con 0x7ff4bc0015e0 2024-05-08T21:21:58.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.342+0000 7ff6daf99700 1 -- 172.21.15.3:0/4201326527 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff6b4005c50 con 0x7ff6b40015e0 2024-05-08T21:21:58.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.343+0000 7ff6d17fa700 1 -- 172.21.15.3:0/4201326527 <== osd.7 v2:172.21.15.186:6809/47341 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff6b4005c50 con 0x7ff6b40015e0 2024-05-08T21:21:58.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.343+0000 7f2fc5ffb700 1 -- 172.21.15.3:0/2621787563 <== osd.2 v2:172.21.15.3:6808/47117 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2fb0002d40 con 0x7f2fb4072e30 2024-05-08T21:21:58.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.346+0000 7f2fd78d2700 1 -- 172.21.15.3:0/2621787563 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x7f2fb4072e30 msgr2=0x7f2fb4075250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.346+0000 7f2fd78d2700 1 --2- 172.21.15.3:0/2621787563 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x7f2fb4072e30 0x7f2fb4075250 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.346+0000 7f2fd78d2700 1 -- 172.21.15.3:0/2621787563 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2fb406ac90 msgr2=0x7f2fb406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.346+0000 7f2fd78d2700 1 --2- 172.21.15.3:0/2621787563 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2fb406ac90 0x7f2fb406d150 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc000c00 tx=0x7f2fcc008030 comp rx=0 tx=0).stop 2024-05-08T21:21:58.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.346+0000 7f2fd78d2700 1 -- 172.21.15.3:0/2621787563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd015db80 msgr2=0x7f2fd0160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.346+0000 7f2fd78d2700 1 --2- 172.21.15.3:0/2621787563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd015db80 0x7f2fd0160730 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800cb40 tx=0x7f2fc800cf00 comp rx=0 tx=0).stop 2024-05-08T21:21:58.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.347+0000 7f2fd5e6f700 1 -- 172.21.15.3:0/2621787563 reap_dead start 2024-05-08T21:21:58.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.347+0000 7f2fd78d2700 1 -- 172.21.15.3:0/2621787563 shutdown_connections 2024-05-08T21:21:58.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.347+0000 7f2fd78d2700 1 -- 172.21.15.3:0/2621787563 >> 172.21.15.3:0/2621787563 conn(0x7f2fd008afe0 msgr2=0x7f2fd015b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.348+0000 7f70f533b700 1 -- 172.21.15.3:0/462847136 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f70f00b3130 con 0x7f70dc072eb0 2024-05-08T21:21:58.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.350+0000 7f2fd78d2700 1 -- 172.21.15.3:0/2621787563 shutdown_connections 2024-05-08T21:21:58.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.350+0000 7f2fd78d2700 1 -- 172.21.15.3:0/2621787563 wait complete. 2024-05-08T21:21:58.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.352+0000 7f70d77fe700 1 -- 172.21.15.3:0/462847136 <== osd.3 v2:172.21.15.3:6800/47118 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f70e40092a0 con 0x7f70dc072eb0 2024-05-08T21:21:58.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.352+0000 7f70f533b700 1 -- 172.21.15.3:0/462847136 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x7f70dc072eb0 msgr2=0x7f70dc0752d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.352+0000 7f70f533b700 1 --2- 172.21.15.3:0/462847136 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x7f70dc072eb0 0x7f70dc0752d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.352+0000 7f70f533b700 1 -- 172.21.15.3:0/462847136 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f70dc06ac90 msgr2=0x7f70dc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.352+0000 7f70f533b700 1 --2- 172.21.15.3:0/462847136 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f70dc06ac90 0x7f70dc06d150 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f70e8000f80 tx=0x7f70e8012000 comp rx=0 tx=0).stop 2024-05-08T21:21:58.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.352+0000 7f70f533b700 1 -- 172.21.15.3:0/462847136 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70f00cda40 msgr2=0x7f70f0141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.352+0000 7f70f533b700 1 --2- 172.21.15.3:0/462847136 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70f00cda40 0x7f70f0141840 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f70e000e810 tx=0x7f70e000eb20 comp rx=0 tx=0).stop 2024-05-08T21:21:58.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.352+0000 7f70ef7fe700 1 -- 172.21.15.3:0/462847136 reap_dead start 2024-05-08T21:21:58.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.356+0000 7f70f533b700 1 -- 172.21.15.3:0/462847136 shutdown_connections 2024-05-08T21:21:58.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.356+0000 7f70f533b700 1 -- 172.21.15.3:0/462847136 >> 172.21.15.3:0/462847136 conn(0x7f70f006b970 msgr2=0x7f70f00666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.356+0000 7f70f533b700 1 -- 172.21.15.3:0/462847136 shutdown_connections 2024-05-08T21:21:58.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.356+0000 7f70f533b700 1 -- 172.21.15.3:0/462847136 wait complete. 2024-05-08T21:21:58.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.361+0000 7ff6daf99700 1 -- 172.21.15.3:0/4201326527 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x7ff6b40015e0 msgr2=0x7ff6b4003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.361+0000 7ff6daf99700 1 --2- 172.21.15.3:0/4201326527 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x7ff6b40015e0 0x7ff6b4003aa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.360+0000 7ff4dbd86700 1 -- 172.21.15.3:0/2107120098 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff4bc005cb0 con 0x7ff4bc0015e0 2024-05-08T21:21:58.370 INFO:teuthology.orchestra.run.smithi003.stdout:55834574855 2024-05-08T21:21:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.362+0000 7ff4ca7fc700 1 -- 172.21.15.3:0/2107120098 <== osd.5 v2:172.21.15.186:6800/47340 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff4bc005cb0 con 0x7ff4bc0015e0 2024-05-08T21:21:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.363+0000 7ff4dbd86700 1 -- 172.21.15.3:0/2107120098 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x7ff4bc0015e0 msgr2=0x7ff4bc003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-08T21:21:58.371 INFO:teuthology.orchestra.run.smithi003.stdout:55834574855 2024-05-08T21:21:58.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-08T21:21:58.373 INFO:teuthology.orchestra.run.smithi003.stdout:55834574857 2024-05-08T21:21:58.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-08T21:21:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.364+0000 7ff6daf99700 1 -- 172.21.15.3:0/4201326527 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6b806ad60 msgr2=0x7ff6b806d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.364+0000 7ff6daf99700 1 --2- 172.21.15.3:0/4201326527 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6b806ad60 0x7ff6b806d220 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0007fd0 tx=0x7ff6c0009040 comp rx=0 tx=0).stop 2024-05-08T21:21:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.364+0000 7ff6daf99700 1 -- 172.21.15.3:0/4201326527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6d40d1ae0 msgr2=0x7ff6d41da430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.364+0000 7ff6daf99700 1 --2- 172.21.15.3:0/4201326527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6d40d1ae0 0x7ff6d41da430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800e7e0 tx=0x7ff6c800eaf0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.363+0000 7ff4dbd86700 1 --2- 172.21.15.3:0/2107120098 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x7ff4bc0015e0 0x7ff4bc003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.363+0000 7ff4dbd86700 1 -- 172.21.15.3:0/2107120098 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff4b806ad90 msgr2=0x7ff4b806d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.364+0000 7ff6d9536700 1 -- 172.21.15.3:0/4201326527 reap_dead start 2024-05-08T21:21:58.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.363+0000 7ff4dbd86700 1 --2- 172.21.15.3:0/2107120098 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff4b806ad90 0x7ff4b806d250 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000b850 tx=0x7ff4d0008040 comp rx=0 tx=0).stop 2024-05-08T21:21:58.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.363+0000 7ff4dbd86700 1 -- 172.21.15.3:0/2107120098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4d40cea20 msgr2=0x7ff4d414bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.363+0000 7ff4dbd86700 1 --2- 172.21.15.3:0/2107120098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4d40cea20 0x7ff4d414bbe0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4010f00 tx=0x7ff4c4004750 comp rx=0 tx=0).stop 2024-05-08T21:21:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.363+0000 7ff4da323700 1 -- 172.21.15.3:0/2107120098 reap_dead start 2024-05-08T21:21:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.363+0000 7ff4dbd86700 1 -- 172.21.15.3:0/2107120098 shutdown_connections 2024-05-08T21:21:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.363+0000 7ff4dbd86700 1 -- 172.21.15.3:0/2107120098 >> 172.21.15.3:0/2107120098 conn(0x7ff4d40b1f80 msgr2=0x7ff4d40afef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.363+0000 7ff4dbd86700 1 -- 172.21.15.3:0/2107120098 shutdown_connections 2024-05-08T21:21:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.363+0000 7ff4dbd86700 1 -- 172.21.15.3:0/2107120098 wait complete. 2024-05-08T21:21:58.376 INFO:teuthology.orchestra.run.smithi003.stdout:55834574855 2024-05-08T21:21:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.364+0000 7ff6daf99700 1 -- 172.21.15.3:0/4201326527 shutdown_connections 2024-05-08T21:21:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.364+0000 7ff6daf99700 1 -- 172.21.15.3:0/4201326527 >> 172.21.15.3:0/4201326527 conn(0x7ff6d408afe0 msgr2=0x7ff6d41e4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-08T21:21:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.364+0000 7ff6daf99700 1 -- 172.21.15.3:0/4201326527 shutdown_connections 2024-05-08T21:21:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.364+0000 7ff6daf99700 1 -- 172.21.15.3:0/4201326527 wait complete. 2024-05-08T21:21:58.379 INFO:teuthology.orchestra.run.smithi003.stdout:55834574855 2024-05-08T21:21:58.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-08T21:21:58.380 INFO:teuthology.orchestra.run.smithi003.stdout:55834574856 2024-05-08T21:21:58.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-08T21:21:58.388 INFO:teuthology.orchestra.run.smithi003.stdout:55834574855 2024-05-08T21:21:58.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-08T21:21:58.389 INFO:teuthology.orchestra.run.smithi003.stdout:55834574855 2024-05-08T21:21:58.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-08T21:21:58.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.590+0000 7ff2abfbc700 1 Processor -- start 2024-05-08T21:21:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.590+0000 7ff2abfbc700 1 -- start start 2024-05-08T21:21:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.592+0000 7ff2abfbc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff2a4161a80 0x7ff2a40b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.592+0000 7ff2abfbc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2a40ce970 0x7ff2a4070f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.592+0000 7ff2abfbc700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a40b3cc0 con 0x7ff2a40714c0 2024-05-08T21:21:58.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.592+0000 7ff2abfbc700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a4071930 con 0x7ff2a4161a80 2024-05-08T21:21:58.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.592+0000 7ff2abfbc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40b3fa0 con 0x7ff2a40ce970 2024-05-08T21:21:58.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.593+0000 7ff2aa559700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff2a40714c0 0x7ff2a41656a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:44320/0 (socket says 172.21.15.3:44320) 2024-05-08T21:21:58.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.593+0000 7ff2aa559700 1 -- 172.21.15.3:0/2775020795 learned_addr learned my addr 172.21.15.3:0/2775020795 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:58.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.598+0000 7ff2a8d56700 1 -- 172.21.15.3:0/2775020795 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793119745 0 0) 0x7ff2a40b3cc0 con 0x7ff2a40714c0 2024-05-08T21:21:58.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.598+0000 7ff2a8d56700 1 -- 172.21.15.3:0/2775020795 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2900035b0 con 0x7ff2a40714c0 2024-05-08T21:21:58.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.599+0000 7ff2a9d58700 1 --2- 172.21.15.3:0/2775020795 >> v2:172.21.15.3:3300/0 conn(0x7ff2a4161a80 0x7ff2a40b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.599+0000 7ff2a8d56700 1 -- 172.21.15.3:0/2775020795 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3889512340 0 0) 0x7ff2900035b0 con 0x7ff2a40714c0 2024-05-08T21:21:58.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.599+0000 7ff2a8d56700 1 -- 172.21.15.3:0/2775020795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2a40ce970 msgr2=0x7ff2a4070f80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-08T21:21:58.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.599+0000 7ff2a8d56700 1 --2- 172.21.15.3:0/2775020795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2a40ce970 0x7ff2a4070f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.599+0000 7ff2a8d56700 1 -- 172.21.15.3:0/2775020795 >> v2:172.21.15.3:3300/0 conn(0x7ff2a4161a80 msgr2=0x7ff2a40b30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.599+0000 7ff2a8d56700 1 --2- 172.21.15.3:0/2775020795 >> v2:172.21.15.3:3300/0 conn(0x7ff2a4161a80 0x7ff2a40b30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.599+0000 7ff2a8d56700 1 -- 172.21.15.3:0/2775020795 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a40b4280 con 0x7ff2a40714c0 2024-05-08T21:21:58.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.599+0000 7ff2a8d56700 1 -- 172.21.15.3:0/2775020795 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7ff29c0019f0 con 0x7ff2a40714c0 2024-05-08T21:21:58.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.599+0000 7ff2a8d56700 1 -- 172.21.15.3:0/2775020795 >> v1:172.21.15.186:6789/0 conn(0x7ff2a40714c0 legacy=0x7ff2a41656a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.599+0000 7ff2a8d56700 1 --2- 172.21.15.3:0/2775020795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff290003a00 0x7ff290003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.599+0000 7ff2a8d56700 1 --2- 172.21.15.3:0/2775020795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff290004850 0x7ff290004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.599+0000 7ff2a8d56700 1 --2- 172.21.15.3:0/2775020795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff290005210 0x7ff29000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.599+0000 7ff2a8d56700 1 -- 172.21.15.3:0/2775020795 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a4071930 con 0x7ff290005210 2024-05-08T21:21:58.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.599+0000 7ff2a8d56700 1 -- 172.21.15.3:0/2775020795 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40b3fa0 con 0x7ff290004850 2024-05-08T21:21:58.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.599+0000 7ff2a8d56700 1 -- 172.21.15.3:0/2775020795 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29000c110 con 0x7ff290003a00 2024-05-08T21:21:58.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.599+0000 7ff2a9557700 1 --2- 172.21.15.3:0/2775020795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff290004850 0x7ff290004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.600+0000 7ff2a9d58700 1 --2- 172.21.15.3:0/2775020795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff290003a00 0x7ff290003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.600+0000 7ff2a9557700 1 -- 172.21.15.3:0/2775020795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff290003a00 msgr2=0x7ff290003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.600+0000 7ff2a9557700 1 --2- 172.21.15.3:0/2775020795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff290003a00 0x7ff290003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.600+0000 7ff2a9557700 1 -- 172.21.15.3:0/2775020795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff290005210 msgr2=0x7ff29000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.600+0000 7ff2a9557700 1 --2- 172.21.15.3:0/2775020795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff290005210 0x7ff29000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.600+0000 7ff2a9557700 1 -- 172.21.15.3:0/2775020795 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a40b4280 con 0x7ff290004850 2024-05-08T21:21:58.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.600+0000 7ff2a9d58700 1 --2- 172.21.15.3:0/2775020795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff290003a00 0x7ff290003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:21:58.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.600+0000 7ff2a9557700 1 --2- 172.21.15.3:0/2775020795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff290004850 0x7ff290004cd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4067bc0 tx=0x7ff290005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc3c9baced325be0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.601+0000 7ff2a8d56700 1 -- 172.21.15.3:0/2775020795 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a00092b0 con 0x7ff290004850 2024-05-08T21:21:58.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.601+0000 7ff2a8d56700 1 -- 172.21.15.3:0/2775020795 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a0011640 con 0x7ff290004850 2024-05-08T21:21:58.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.601+0000 7ff2a8d56700 1 -- 172.21.15.3:0/2775020795 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a00119b0 con 0x7ff290004850 2024-05-08T21:21:58.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.601+0000 7f90f5ce5700 1 Processor -- start 2024-05-08T21:21:58.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.602+0000 7ff2abfbc700 1 -- 172.21.15.3:0/2775020795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff290004850 msgr2=0x7ff290004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.602+0000 7ff2abfbc700 1 --2- 172.21.15.3:0/2775020795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff290004850 0x7ff290004cd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4067bc0 tx=0x7ff290005880 comp rx=0 tx=0).stop 2024-05-08T21:21:58.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.601+0000 7f90f5ce5700 1 -- start start 2024-05-08T21:21:58.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.602+0000 7ff2aa559700 1 -- 172.21.15.3:0/2775020795 reap_dead start 2024-05-08T21:21:58.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.602+0000 7ff2abfbc700 1 -- 172.21.15.3:0/2775020795 shutdown_connections 2024-05-08T21:21:58.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.602+0000 7ff2abfbc700 1 -- 172.21.15.3:0/2775020795 >> 172.21.15.3:0/2775020795 conn(0x7ff2a406af90 msgr2=0x7ff2a40b1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.602+0000 7f90f5ce5700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f90f015f900 0x7f90f006ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.603+0000 7f90f5ce5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f90f01628e0 0x7f90f0071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.603+0000 7f90f5ce5700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f00711d0 con 0x7f90f006b2c0 2024-05-08T21:21:58.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.603+0000 7f90f5ce5700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f006b750 con 0x7f90f015f900 2024-05-08T21:21:58.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.603+0000 7f90f5ce5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f0070f30 con 0x7f90f01628e0 2024-05-08T21:21:58.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.603+0000 7f90eeffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f90f006b2c0 0x7f90f01621d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:33072/0 (socket says 172.21.15.3:33072) 2024-05-08T21:21:58.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.603+0000 7f90eeffd700 1 -- 172.21.15.3:0/2782772518 learned_addr learned my addr 172.21.15.3:0/2782772518 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:58.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.603+0000 7f90effff700 1 --2- 172.21.15.3:0/2782772518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f90f01628e0 0x7f90f0071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.604+0000 7f90ef7fe700 1 --2- 172.21.15.3:0/2782772518 >> v2:172.21.15.186:3300/0 conn(0x7f90f015f900 0x7f90f006ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.604+0000 7f90effff700 1 -- 172.21.15.3:0/2782772518 >> v2:172.21.15.186:3300/0 conn(0x7f90f015f900 msgr2=0x7f90f006ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.604+0000 7f90effff700 1 --2- 172.21.15.3:0/2782772518 >> v2:172.21.15.186:3300/0 conn(0x7f90f015f900 0x7f90f006ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.604+0000 7f90effff700 1 -- 172.21.15.3:0/2782772518 >> v1:172.21.15.3:6789/0 conn(0x7f90f006b2c0 legacy=0x7f90f01621d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.604+0000 7f90effff700 1 -- 172.21.15.3:0/2782772518 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f0162e30 con 0x7f90f01628e0 2024-05-08T21:21:58.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.604+0000 7f90effff700 1 --2- 172.21.15.3:0/2782772518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f90f01628e0 0x7f90f0071c70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f90e0009840 tx=0x7f90e0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7169b3415f4a1203 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.603+0000 7ff2abfbc700 1 -- 172.21.15.3:0/2775020795 shutdown_connections 2024-05-08T21:21:58.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.603+0000 7ff2abfbc700 1 -- 172.21.15.3:0/2775020795 wait complete. 2024-05-08T21:21:58.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.605+0000 7f90ee7fc700 1 -- 172.21.15.3:0/2782772518 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e0004230 con 0x7f90f01628e0 2024-05-08T21:21:58.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.605+0000 7ff2abfbc700 1 Processor -- start 2024-05-08T21:21:58.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.605+0000 7f90ee7fc700 1 -- 172.21.15.3:0/2782772518 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e0004390 con 0x7f90f01628e0 2024-05-08T21:21:58.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.605+0000 7f90ee7fc700 1 -- 172.21.15.3:0/2782772518 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e00036a0 con 0x7f90f01628e0 2024-05-08T21:21:58.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.601+0000 7fe1e0130700 1 Processor -- start 2024-05-08T21:21:58.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.602+0000 7fe1e0130700 1 -- start start 2024-05-08T21:21:58.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.603+0000 7fe1e0130700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1d8098a80 0x7fe1d806ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.603+0000 7fe1e0130700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe1d806e4f0 0x7fe1d80b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.603+0000 7fe1e0130700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d80b1790 con 0x7fe1d80b1340 2024-05-08T21:21:58.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.603+0000 7fe1e0130700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d806d4b0 con 0x7fe1d806e4f0 2024-05-08T21:21:58.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.603+0000 7fe1e0130700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d806d790 con 0x7fe1d8098a80 2024-05-08T21:21:58.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.603+0000 7fe1ddecc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1d8098a80 0x7fe1d806ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.604+0000 7fe1de6cd700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe1d80b1340 0x7fe1d816ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:44334/0 (socket says 172.21.15.3:44334) 2024-05-08T21:21:58.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.604+0000 7fe1de6cd700 1 -- 172.21.15.3:0/2955630830 learned_addr learned my addr 172.21.15.3:0/2955630830 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:58.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.606+0000 7f90f5ce5700 1 -- 172.21.15.3:0/2782772518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f90f01628e0 msgr2=0x7f90f0071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.606+0000 7f90f5ce5700 1 --2- 172.21.15.3:0/2782772518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f90f01628e0 0x7f90f0071c70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f90e0009840 tx=0x7f90e0009b50 comp rx=0 tx=0).stop 2024-05-08T21:21:58.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.605+0000 7ff2abfbc700 1 -- start start 2024-05-08T21:21:58.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.606+0000 7ff2abfbc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2a40cdaa0 0x7ff2a40cde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.606+0000 7ff2abfbc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2a4163e30 0x7ff2a41642b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.606+0000 7ff2abfbc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2a41451e0 0x7ff2a4145660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.606+0000 7ff2abfbc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40cc1c0 con 0x7ff2a4163e30 2024-05-08T21:21:58.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.606+0000 7ff2abfbc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40ccd40 con 0x7ff2a41451e0 2024-05-08T21:21:58.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.606+0000 7ff2abfbc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40cd1d0 con 0x7ff2a40cdaa0 2024-05-08T21:21:58.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.606+0000 7f90f5ce5700 1 -- 172.21.15.3:0/2782772518 shutdown_connections 2024-05-08T21:21:58.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.606+0000 7f90f5ce5700 1 --2- 172.21.15.3:0/2782772518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f90f01628e0 0x7f90f0071c70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.606+0000 7f90f5ce5700 1 --2- 172.21.15.3:0/2782772518 >> v2:172.21.15.186:3300/0 conn(0x7f90f015f900 0x7f90f006ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.606+0000 7f90f5ce5700 1 -- 172.21.15.3:0/2782772518 >> 172.21.15.3:0/2782772518 conn(0x7f90f00b1290 msgr2=0x7f90f00b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.606+0000 7ff2a9557700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2a4163e30 0x7ff2a41642b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.606+0000 7ff2a9557700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2a4163e30 0x7ff2a41642b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52436/0 (socket says 172.21.15.3:52436) 2024-05-08T21:21:58.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.606+0000 7ff2a9557700 1 -- 172.21.15.3:0/2894462281 learned_addr learned my addr 172.21.15.3:0/2894462281 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.606+0000 7ff2a9d58700 1 --2- 172.21.15.3:0/2894462281 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2a40cdaa0 0x7ff2a40cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.607+0000 7ff2aa559700 1 --2- 172.21.15.3:0/2894462281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2a41451e0 0x7ff2a4145660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.607+0000 7ff2a9557700 1 -- 172.21.15.3:0/2894462281 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2a40cdaa0 msgr2=0x7ff2a40cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.607+0000 7ff2a9557700 1 --2- 172.21.15.3:0/2894462281 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2a40cdaa0 0x7ff2a40cde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.607+0000 7ff2a9557700 1 -- 172.21.15.3:0/2894462281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2a41451e0 msgr2=0x7ff2a4145660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.607+0000 7ff2a9557700 1 --2- 172.21.15.3:0/2894462281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2a41451e0 0x7ff2a4145660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.607+0000 7ff2a9557700 1 -- 172.21.15.3:0/2894462281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a40ce3c0 con 0x7ff2a4163e30 2024-05-08T21:21:58.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.604+0000 7fe1dd6cb700 1 --2- 172.21.15.3:0/2955630830 >> v2:172.21.15.3:3300/0 conn(0x7fe1d806e4f0 0x7fe1d80b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.607+0000 7ff2a9557700 1 --2- 172.21.15.3:0/2894462281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2a4163e30 0x7ff2a41642b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0013040 tx=0x7ff2a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.608+0000 7ff29a7fc700 1 -- 172.21.15.3:0/2894462281 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a0025630 con 0x7ff2a4163e30 2024-05-08T21:21:58.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.608+0000 7ff29a7fc700 1 -- 172.21.15.3:0/2894462281 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a0020070 con 0x7ff2a4163e30 2024-05-08T21:21:58.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.608+0000 7ff29a7fc700 1 -- 172.21.15.3:0/2894462281 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a00259f0 con 0x7ff2a4163e30 2024-05-08T21:21:58.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.608+0000 7fe1ddecc700 1 -- 172.21.15.3:0/2955630830 >> v2:172.21.15.3:3300/0 conn(0x7fe1d806e4f0 msgr2=0x7fe1d80b0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.608+0000 7fe1ddecc700 1 --2- 172.21.15.3:0/2955630830 >> v2:172.21.15.3:3300/0 conn(0x7fe1d806e4f0 0x7fe1d80b0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.608+0000 7fe1ddecc700 1 -- 172.21.15.3:0/2955630830 >> v1:172.21.15.186:6789/0 conn(0x7fe1d80b1340 legacy=0x7fe1d816ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.608+0000 7fe1ddecc700 1 -- 172.21.15.3:0/2955630830 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d806da70 con 0x7fe1d8098a80 2024-05-08T21:21:58.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.608+0000 7fe1ddecc700 1 --2- 172.21.15.3:0/2955630830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1d8098a80 0x7fe1d806ce20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4007360 tx=0x7fe1d4007670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fad0dcc2e15b6cc7 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.608+0000 7ff2abfbc700 1 -- 172.21.15.3:0/2894462281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a40cc410 con 0x7ff2a4163e30 2024-05-08T21:21:58.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.608+0000 7ff2abfbc700 1 -- 172.21.15.3:0/2894462281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a40cc9d0 con 0x7ff2a4163e30 2024-05-08T21:21:58.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.610+0000 7ff2abfbc700 1 -- 172.21.15.3:0/2894462281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a40ae7b0 con 0x7ff2a4163e30 2024-05-08T21:21:58.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.610+0000 7fe1dceca700 1 -- 172.21.15.3:0/2955630830 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d400a580 con 0x7fe1d8098a80 2024-05-08T21:21:58.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.610+0000 7fe1dceca700 1 -- 172.21.15.3:0/2955630830 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d400a6e0 con 0x7fe1d8098a80 2024-05-08T21:21:58.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.610+0000 7fe1dceca700 1 -- 172.21.15.3:0/2955630830 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4004230 con 0x7fe1d8098a80 2024-05-08T21:21:58.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.611+0000 7fe1dd6cb700 1 --2- 172.21.15.3:0/2955630830 >> v2:172.21.15.3:3300/0 conn(0x7fe1d806e4f0 0x7fe1d80b0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:21:58.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.611+0000 7f90f5ce5700 1 -- 172.21.15.3:0/2782772518 shutdown_connections 2024-05-08T21:21:58.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.613+0000 7fe1e0130700 1 -- 172.21.15.3:0/2955630830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1d8098a80 msgr2=0x7fe1d806ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.613+0000 7fe1e0130700 1 --2- 172.21.15.3:0/2955630830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1d8098a80 0x7fe1d806ce20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4007360 tx=0x7fe1d4007670 comp rx=0 tx=0).stop 2024-05-08T21:21:58.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.614+0000 7fe1e0130700 1 -- 172.21.15.3:0/2955630830 shutdown_connections 2024-05-08T21:21:58.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.614+0000 7fe1e0130700 1 --2- 172.21.15.3:0/2955630830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1d8098a80 0x7fe1d806ce20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.614+0000 7fe1e0130700 1 --2- 172.21.15.3:0/2955630830 >> v2:172.21.15.3:3300/0 conn(0x7fe1d806e4f0 0x7fe1d80b0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.614+0000 7fe1e0130700 1 -- 172.21.15.3:0/2955630830 >> 172.21.15.3:0/2955630830 conn(0x7fe1d80b4690 msgr2=0x7fe1d80b4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.613+0000 7f90f5ce5700 1 -- 172.21.15.3:0/2782772518 wait complete. 2024-05-08T21:21:58.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.615+0000 7f90f5ce5700 1 Processor -- start 2024-05-08T21:21:58.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.614+0000 7fe1e0130700 1 -- 172.21.15.3:0/2955630830 shutdown_connections 2024-05-08T21:21:58.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.614+0000 7fe1e0130700 1 -- 172.21.15.3:0/2955630830 wait complete. 2024-05-08T21:21:58.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.616+0000 7f90f5ce5700 1 -- start start 2024-05-08T21:21:58.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.617+0000 7f90f5ce5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90f006b2c0 0x7f90f00ac150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.617+0000 7f90f5ce5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90f015f900 0x7f90f00adbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.617+0000 7f90f5ce5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f90f01628e0 0x7f90f009cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.617+0000 7f90f5ce5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f014e1f0 con 0x7f90f006b2c0 2024-05-08T21:21:58.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.617+0000 7f90f5ce5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00ac690 con 0x7f90f015f900 2024-05-08T21:21:58.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.617+0000 7f90f5ce5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f014d570 con 0x7f90f01628e0 2024-05-08T21:21:58.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.617+0000 7f90eeffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90f015f900 0x7f90f00adbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.617+0000 7f90eeffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90f015f900 0x7f90f00adbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:38484/0 (socket says 172.21.15.3:38484) 2024-05-08T21:21:58.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.617+0000 7f90eeffd700 1 -- 172.21.15.3:0/3848415122 learned_addr learned my addr 172.21.15.3:0/3848415122 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.617+0000 7f90effff700 1 --2- 172.21.15.3:0/3848415122 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f90f01628e0 0x7f90f009cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.618+0000 7f90eeffd700 1 -- 172.21.15.3:0/3848415122 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f90f01628e0 msgr2=0x7f90f009cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.618+0000 7f90eeffd700 1 --2- 172.21.15.3:0/3848415122 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f90f01628e0 0x7f90f009cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.618+0000 7f90eeffd700 1 -- 172.21.15.3:0/3848415122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90f006b2c0 msgr2=0x7f90f00ac150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-08T21:21:58.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.618+0000 7f90eeffd700 1 --2- 172.21.15.3:0/3848415122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90f006b2c0 0x7f90f00ac150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.618+0000 7f90eeffd700 1 -- 172.21.15.3:0/3848415122 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e00094a0 con 0x7f90f015f900 2024-05-08T21:21:58.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.614+0000 7ff29a7fc700 1 -- 172.21.15.3:0/2894462281 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff2a0022030 con 0x7ff2a4163e30 2024-05-08T21:21:58.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.618+0000 7f90eeffd700 1 --2- 172.21.15.3:0/3848415122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90f015f900 0x7f90f00adbd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f90e8002080 tx=0x7f90e800a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.618+0000 7f90d7fff700 1 -- 172.21.15.3:0/3848415122 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e800ae00 con 0x7f90f015f900 2024-05-08T21:21:58.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.618+0000 7f90d7fff700 1 -- 172.21.15.3:0/3848415122 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e80070d0 con 0x7f90f015f900 2024-05-08T21:21:58.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.618+0000 7f90d7fff700 1 -- 172.21.15.3:0/3848415122 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e8007480 con 0x7f90f015f900 2024-05-08T21:21:58.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.617+0000 7fe1e0130700 1 Processor -- start 2024-05-08T21:21:58.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.617+0000 7fe1e0130700 1 -- start start 2024-05-08T21:21:58.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.619+0000 7fe1e0130700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1d806e4f0 0x7fe1d80af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.619+0000 7fe1e0130700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1d8098a80 0x7fe1d8214890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.619+0000 7fe1e0130700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1d80b1340 0x7fe1d8099760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.619+0000 7fe1e0130700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d8215810 con 0x7fe1d806e4f0 2024-05-08T21:21:58.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.619+0000 7fe1e0130700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d80ac760 con 0x7fe1d8098a80 2024-05-08T21:21:58.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.619+0000 7fe1e0130700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d8215250 con 0x7fe1d80b1340 2024-05-08T21:21:58.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.619+0000 7fe1ddecc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1d806e4f0 0x7fe1d80af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.619+0000 7fe1ddecc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1d806e4f0 0x7fe1d80af080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52444/0 (socket says 172.21.15.3:52444) 2024-05-08T21:21:58.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.619+0000 7fe1ddecc700 1 -- 172.21.15.3:0/3321901447 learned_addr learned my addr 172.21.15.3:0/3321901447 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.619+0000 7fe1de6cd700 1 --2- 172.21.15.3:0/3321901447 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1d80b1340 0x7fe1d8099760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.620+0000 7fe1ddecc700 1 -- 172.21.15.3:0/3321901447 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1d80b1340 msgr2=0x7fe1d8099760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.620+0000 7fe1ddecc700 1 --2- 172.21.15.3:0/3321901447 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1d80b1340 0x7fe1d8099760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.620+0000 7fe1ddecc700 1 -- 172.21.15.3:0/3321901447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1d8098a80 msgr2=0x7fe1d8214890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-08T21:21:58.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.620+0000 7fe1ddecc700 1 --2- 172.21.15.3:0/3321901447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1d8098a80 0x7fe1d8214890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.620+0000 7fe1ddecc700 1 -- 172.21.15.3:0/3321901447 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d4007040 con 0x7fe1d806e4f0 2024-05-08T21:21:58.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.620+0000 7fe1de6cd700 1 --2- 172.21.15.3:0/3321901447 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1d80b1340 0x7fe1d8099760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:21:58.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.620+0000 7fe1ddecc700 1 --2- 172.21.15.3:0/3321901447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1d806e4f0 0x7fe1d80af080 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4000c00 tx=0x7fe1d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.617+0000 7ff29a7fc700 1 --2- 172.21.15.3:0/2894462281 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff29006c720 0x7ff29006ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.617+0000 7ff29a7fc700 1 -- 172.21.15.3:0/2894462281 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2a008a7d0 con 0x7ff2a4163e30 2024-05-08T21:21:58.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.617+0000 7ff2a9d58700 1 --2- 172.21.15.3:0/2894462281 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff29006c720 0x7ff29006ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.620+0000 7f90f5ce5700 1 -- 172.21.15.3:0/3848415122 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f014d850 con 0x7f90f015f900 2024-05-08T21:21:58.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.620+0000 7f90f5ce5700 1 -- 172.21.15.3:0/3848415122 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f014de10 con 0x7f90f015f900 2024-05-08T21:21:58.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.620+0000 7ff29a7fc700 1 -- 172.21.15.3:0/2894462281 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff2a0055b40 con 0x7ff2a4163e30 2024-05-08T21:21:58.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.621+0000 7fe1ce7fc700 1 -- 172.21.15.3:0/3321901447 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4003c60 con 0x7fe1d806e4f0 2024-05-08T21:21:58.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.621+0000 7fe1ce7fc700 1 -- 172.21.15.3:0/3321901447 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d4003dc0 con 0x7fe1d806e4f0 2024-05-08T21:21:58.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.621+0000 7fe1ce7fc700 1 -- 172.21.15.3:0/3321901447 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4012680 con 0x7fe1d806e4f0 2024-05-08T21:21:58.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.622+0000 7fe1e0130700 1 -- 172.21.15.3:0/3321901447 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d8215530 con 0x7fe1d806e4f0 2024-05-08T21:21:58.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.622+0000 7fe1e0130700 1 -- 172.21.15.3:0/3321901447 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d80af5c0 con 0x7fe1d806e4f0 2024-05-08T21:21:58.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.622+0000 7ff2a9d58700 1 --2- 172.21.15.3:0/2894462281 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff29006c720 0x7ff29006ebe0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff294006b20 tx=0x7ff294008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.624+0000 7fe1e0130700 1 -- 172.21.15.3:0/3321901447 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d80678f0 con 0x7fe1d806e4f0 2024-05-08T21:21:58.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.625+0000 7fe1ce7fc700 1 -- 172.21.15.3:0/3321901447 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe1d40127e0 con 0x7fe1d806e4f0 2024-05-08T21:21:58.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.626+0000 7fe1ce7fc700 1 --2- 172.21.15.3:0/3321901447 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe1bc06ac90 0x7fe1bc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.626+0000 7fe1ce7fc700 1 -- 172.21.15.3:0/3321901447 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1d408aae0 con 0x7fe1d806e4f0 2024-05-08T21:21:58.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.627+0000 7fe1dd6cb700 1 --2- 172.21.15.3:0/3321901447 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe1bc06ac90 0x7fe1bc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.628+0000 7fe1dd6cb700 1 --2- 172.21.15.3:0/3321901447 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe1bc06ac90 0x7fe1bc06d150 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8006e70 tx=0x7fe1c8006c00 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.631+0000 7f069f477700 1 Processor -- start 2024-05-08T21:21:58.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.631+0000 7f069f477700 1 -- start start 2024-05-08T21:21:58.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.632+0000 7f069f477700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0698165820 0x7f0698165c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.632+0000 7f069f477700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0698161ee0 0x7f0698166140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.632+0000 7f069f477700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f06980b22d0 0x7f06980b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.632+0000 7f069f477700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0698166680 con 0x7f0698161ee0 2024-05-08T21:21:58.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.632+0000 7f069f477700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06980b4a90 con 0x7f06980b22d0 2024-05-08T21:21:58.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.632+0000 7f069f477700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06980b4ce0 con 0x7f0698165820 2024-05-08T21:21:58.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.633+0000 7f069d213700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0698165820 0x7f0698165c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.633+0000 7fe1ce7fc700 1 -- 172.21.15.3:0/3321901447 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe1d4055dd0 con 0x7fe1d806e4f0 2024-05-08T21:21:58.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.633+0000 7f069d213700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0698165820 0x7f0698165c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.3:55832/0 (socket says 172.21.15.3:55832) 2024-05-08T21:21:58.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.633+0000 7f069d213700 1 -- 172.21.15.3:0/1682676108 learned_addr learned my addr 172.21.15.3:0/1682676108 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.633+0000 7f069da14700 1 --2- 172.21.15.3:0/1682676108 >> v2:172.21.15.186:3300/0 conn(0x7f06980b22d0 0x7f06980b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.633+0000 7f069d213700 1 -- 172.21.15.3:0/1682676108 >> v2:172.21.15.186:3300/0 conn(0x7f06980b22d0 msgr2=0x7f06980b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.633+0000 7f069d213700 1 --2- 172.21.15.3:0/1682676108 >> v2:172.21.15.186:3300/0 conn(0x7f06980b22d0 0x7f06980b35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.633+0000 7f069d213700 1 -- 172.21.15.3:0/1682676108 >> v2:172.21.15.3:3300/0 conn(0x7f0698161ee0 msgr2=0x7f0698166140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.633+0000 7f069d213700 1 --2- 172.21.15.3:0/1682676108 >> v2:172.21.15.3:3300/0 conn(0x7f0698161ee0 0x7f0698166140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.633+0000 7f069d213700 1 -- 172.21.15.3:0/1682676108 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06980d2510 con 0x7f0698165820 2024-05-08T21:21:58.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.634+0000 7f069d213700 1 --2- 172.21.15.3:0/1682676108 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0698165820 0x7f0698165c00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f06980d11a0 tx=0x7f0688009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c278a869f51e107 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.635+0000 7f068ffff700 1 -- 172.21.15.3:0/1682676108 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068800b870 con 0x7f0698165820 2024-05-08T21:21:58.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.635+0000 7f068ffff700 1 -- 172.21.15.3:0/1682676108 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f068800b9d0 con 0x7f0698165820 2024-05-08T21:21:58.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.635+0000 7f068ffff700 1 -- 172.21.15.3:0/1682676108 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068800bd60 con 0x7f0698165820 2024-05-08T21:21:58.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.636+0000 7f069f477700 1 -- 172.21.15.3:0/1682676108 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0698165820 msgr2=0x7f0698165c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.636+0000 7f069f477700 1 --2- 172.21.15.3:0/1682676108 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0698165820 0x7f0698165c00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f06980d11a0 tx=0x7f0688009bd0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.636+0000 7f069f477700 1 -- 172.21.15.3:0/1682676108 shutdown_connections 2024-05-08T21:21:58.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.636+0000 7f069f477700 1 --2- 172.21.15.3:0/1682676108 >> v2:172.21.15.186:3300/0 conn(0x7f06980b22d0 0x7f06980b35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.636+0000 7f069f477700 1 --2- 172.21.15.3:0/1682676108 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0698165820 0x7f0698165c00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.636+0000 7f069f477700 1 --2- 172.21.15.3:0/1682676108 >> v2:172.21.15.3:3300/0 conn(0x7f0698161ee0 0x7f0698166140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.636+0000 7f069f477700 1 -- 172.21.15.3:0/1682676108 >> 172.21.15.3:0/1682676108 conn(0x7f06980d3620 msgr2=0x7f06980d3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.637+0000 7f069f477700 1 -- 172.21.15.3:0/1682676108 shutdown_connections 2024-05-08T21:21:58.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.637+0000 7f069f477700 1 -- 172.21.15.3:0/1682676108 wait complete. 2024-05-08T21:21:58.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.639+0000 7f069f477700 1 Processor -- start 2024-05-08T21:21:58.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.639+0000 7f069f477700 1 -- start start 2024-05-08T21:21:58.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.640+0000 7f069f477700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f06980b22d0 0x7f06980c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.640+0000 7f069f477700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0698161ee0 0x7f06980c8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.640+0000 7f069f477700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0698165820 0x7f069806ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.640+0000 7f069f477700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0698154430 con 0x7f0698165820 2024-05-08T21:21:58.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.640+0000 7f069f477700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0698162530 con 0x7f0698161ee0 2024-05-08T21:21:58.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.640+0000 7f069f477700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06980c8aa0 con 0x7f06980b22d0 2024-05-08T21:21:58.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.641+0000 7f069d213700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f06980b22d0 0x7f06980c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.641+0000 7f069d213700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f06980b22d0 0x7f06980c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.3:55848/0 (socket says 172.21.15.3:55848) 2024-05-08T21:21:58.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.641+0000 7f069d213700 1 -- 172.21.15.3:0/2484101751 learned_addr learned my addr 172.21.15.3:0/2484101751 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.641+0000 7f069ca12700 1 --2- 172.21.15.3:0/2484101751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0698161ee0 0x7f06980c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.641+0000 7f069da14700 1 --2- 172.21.15.3:0/2484101751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0698165820 0x7f069806ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.641+0000 7f069d213700 1 -- 172.21.15.3:0/2484101751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0698161ee0 msgr2=0x7f06980c8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.641+0000 7f069d213700 1 --2- 172.21.15.3:0/2484101751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0698161ee0 0x7f06980c8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.641+0000 7f069d213700 1 -- 172.21.15.3:0/2484101751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0698165820 msgr2=0x7f069806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.641+0000 7f069d213700 1 --2- 172.21.15.3:0/2484101751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0698165820 0x7f069806ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.641+0000 7f069d213700 1 -- 172.21.15.3:0/2484101751 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0688009550 con 0x7f06980b22d0 2024-05-08T21:21:58.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.641+0000 7f069da14700 1 --2- 172.21.15.3:0/2484101751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0698165820 0x7f069806ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:21:58.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.642+0000 7f069d213700 1 --2- 172.21.15.3:0/2484101751 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f06980b22d0 0x7f06980c7380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0688014040 tx=0x7f0688004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.635+0000 7f90d7fff700 1 -- 172.21.15.3:0/3848415122 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f90e80075e0 con 0x7f90f015f900 2024-05-08T21:21:58.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.636+0000 7f90d7fff700 1 --2- 172.21.15.3:0/3848415122 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f90dc06ace0 0x7f90dc06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.636+0000 7f90d7fff700 1 -- 172.21.15.3:0/3848415122 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90e8089310 con 0x7f90f015f900 2024-05-08T21:21:58.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.643+0000 7f068dffb700 1 -- 172.21.15.3:0/2484101751 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688004310 con 0x7f06980b22d0 2024-05-08T21:21:58.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.643+0000 7f068dffb700 1 -- 172.21.15.3:0/2484101751 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f068801f070 con 0x7f06980b22d0 2024-05-08T21:21:58.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.643+0000 7f068dffb700 1 -- 172.21.15.3:0/2484101751 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06880036a0 con 0x7f06980b22d0 2024-05-08T21:21:58.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.645+0000 7f90ef7fe700 1 --2- 172.21.15.3:0/3848415122 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f90dc06ace0 0x7f90dc06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.645+0000 7f90ef7fe700 1 --2- 172.21.15.3:0/3848415122 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f90dc06ace0 0x7f90dc06d1a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f90e4006ba0 tx=0x7f90e4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.643+0000 7f069f477700 1 -- 172.21.15.3:0/2484101751 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06980c8d40 con 0x7f06980b22d0 2024-05-08T21:21:58.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.643+0000 7f069f477700 1 -- 172.21.15.3:0/2484101751 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06980c9300 con 0x7f06980b22d0 2024-05-08T21:21:58.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.646+0000 7f069f477700 1 -- 172.21.15.3:0/2484101751 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06780052f0 con 0x7f06980b22d0 2024-05-08T21:21:58.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.646+0000 7f90f5ce5700 1 -- 172.21.15.3:0/3848415122 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d00052f0 con 0x7f90f015f900 2024-05-08T21:21:58.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.648+0000 7f068dffb700 1 -- 172.21.15.3:0/2484101751 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0688021030 con 0x7f06980b22d0 2024-05-08T21:21:58.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.649+0000 7f068dffb700 1 --2- 172.21.15.3:0/2484101751 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f067c06ac90 0x7f067c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.649+0000 7f068dffb700 1 -- 172.21.15.3:0/2484101751 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f068808a950 con 0x7f06980b22d0 2024-05-08T21:21:58.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.649+0000 7f069ca12700 1 --2- 172.21.15.3:0/2484101751 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f067c06ac90 0x7f067c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.650+0000 7f069ca12700 1 --2- 172.21.15.3:0/2484101751 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f067c06ac90 0x7f067c06d150 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0694006fd0 tx=0x7f0694008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.652+0000 7f068dffb700 1 -- 172.21.15.3:0/2484101751 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0688055ca0 con 0x7f06980b22d0 2024-05-08T21:21:58.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.657+0000 7f90d7fff700 1 -- 172.21.15.3:0/3848415122 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f90e8018080 con 0x7f90f015f900 2024-05-08T21:21:58.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.665+0000 7fccbb1c9700 1 Processor -- start 2024-05-08T21:21:58.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.665+0000 7fccbb1c9700 1 -- start start 2024-05-08T21:21:58.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.667+0000 7fccbb1c9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccb415ba50 0x7fccb40b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.667+0000 7fccbb1c9700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fccb409a160 0x7fccb40cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.667+0000 7fccbb1c9700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb40cfca0 con 0x7fccb409a7f0 2024-05-08T21:21:58.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.667+0000 7fccbb1c9700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb40cff80 con 0x7fccb409a160 2024-05-08T21:21:58.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.667+0000 7fccbb1c9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb40d0260 con 0x7fccb415ba50 2024-05-08T21:21:58.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.667+0000 7fccb9766700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fccb409a160 0x7fccb40cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.667+0000 7fccb9766700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fccb409a160 0x7fccb40cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52472/0 (socket says 172.21.15.3:52472) 2024-05-08T21:21:58.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.667+0000 7fccb9766700 1 -- 172.21.15.3:0/3422700793 learned_addr learned my addr 172.21.15.3:0/3422700793 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.668+0000 7fccb8f65700 1 --2- 172.21.15.3:0/3422700793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccb415ba50 0x7fccb40b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.668+0000 7fccb9766700 1 -- 172.21.15.3:0/3422700793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccb415ba50 msgr2=0x7fccb40b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.668+0000 7fccb9766700 1 --2- 172.21.15.3:0/3422700793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccb415ba50 0x7fccb40b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.668+0000 7fccb9766700 1 -- 172.21.15.3:0/3422700793 >> v1:172.21.15.186:6789/0 conn(0x7fccb409a7f0 legacy=0x7fccb4099a50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-08T21:21:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.668+0000 7fccb9766700 1 -- 172.21.15.3:0/3422700793 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb40cbfb0 con 0x7fccb409a160 2024-05-08T21:21:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.669+0000 7fccb9766700 1 --2- 172.21.15.3:0/3422700793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccb409a160 0x7fccb40cc500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fccac009e50 tx=0x7fccac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=646bf4afc56c387b server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.669+0000 7fccb37fe700 1 -- 172.21.15.3:0/3422700793 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac00e070 con 0x7fccb409a160 2024-05-08T21:21:58.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.669+0000 7fccb37fe700 1 -- 172.21.15.3:0/3422700793 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccac003a20 con 0x7fccb409a160 2024-05-08T21:21:58.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.669+0000 7fccb37fe700 1 -- 172.21.15.3:0/3422700793 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac003dd0 con 0x7fccb409a160 2024-05-08T21:21:58.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.670+0000 7fccbb1c9700 1 -- 172.21.15.3:0/3422700793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccb409a160 msgr2=0x7fccb40cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.670+0000 7fccbb1c9700 1 --2- 172.21.15.3:0/3422700793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccb409a160 0x7fccb40cc500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fccac009e50 tx=0x7fccac00b330 comp rx=0 tx=0).stop 2024-05-08T21:21:58.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.673+0000 7fccbb1c9700 1 -- 172.21.15.3:0/3422700793 shutdown_connections 2024-05-08T21:21:58.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.673+0000 7fccbb1c9700 1 --2- 172.21.15.3:0/3422700793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccb409a160 0x7fccb40cc500 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.673+0000 7fccbb1c9700 1 --2- 172.21.15.3:0/3422700793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccb415ba50 0x7fccb40b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.673+0000 7fccbb1c9700 1 -- 172.21.15.3:0/3422700793 >> 172.21.15.3:0/3422700793 conn(0x7fccb40b2f00 msgr2=0x7fccb40b3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.673+0000 7fccbb1c9700 1 -- 172.21.15.3:0/3422700793 shutdown_connections 2024-05-08T21:21:58.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.673+0000 7fccbb1c9700 1 -- 172.21.15.3:0/3422700793 wait complete. 2024-05-08T21:21:58.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.675+0000 7fccbb1c9700 1 Processor -- start 2024-05-08T21:21:58.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.675+0000 7fccbb1c9700 1 -- start start 2024-05-08T21:21:58.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.676+0000 7fccbb1c9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb409a160 0x7fccb40ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.676+0000 7fccbb1c9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccb415ba50 0x7fccb4203030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.676+0000 7fccbb1c9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccb40c5270 0x7fccb40c5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.676+0000 7fccbb1c9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb40c5c40 con 0x7fccb40c5270 2024-05-08T21:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.676+0000 7fccbb1c9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb40caef0 con 0x7fccb409a160 2024-05-08T21:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.676+0000 7fccbb1c9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb40ca520 con 0x7fccb415ba50 2024-05-08T21:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.676+0000 7fccb8f65700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb409a160 0x7fccb40ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.676+0000 7fccb3fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccb415ba50 0x7fccb4203030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.676+0000 7fccb8f65700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb409a160 0x7fccb40ca9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:38510/0 (socket says 172.21.15.3:38510) 2024-05-08T21:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.676+0000 7fccb8f65700 1 -- 172.21.15.3:0/344246319 learned_addr learned my addr 172.21.15.3:0/344246319 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.676+0000 7fccb9766700 1 --2- 172.21.15.3:0/344246319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccb40c5270 0x7fccb40c5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.677+0000 7fccb9766700 1 -- 172.21.15.3:0/344246319 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccb415ba50 msgr2=0x7fccb4203030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.677+0000 7fccb9766700 1 --2- 172.21.15.3:0/344246319 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccb415ba50 0x7fccb4203030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.677+0000 7fccb9766700 1 -- 172.21.15.3:0/344246319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb409a160 msgr2=0x7fccb40ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.677+0000 7fccb9766700 1 --2- 172.21.15.3:0/344246319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb409a160 0x7fccb40ca9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.677+0000 7fccb9766700 1 -- 172.21.15.3:0/344246319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac009b30 con 0x7fccb40c5270 2024-05-08T21:21:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.677+0000 7fccb9766700 1 --2- 172.21.15.3:0/344246319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccb40c5270 0x7fccb40c5700 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fccb40cef90 tx=0x7fccac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.677+0000 7fccb17fa700 1 -- 172.21.15.3:0/344246319 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac004440 con 0x7fccb40c5270 2024-05-08T21:21:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.678+0000 7fccb17fa700 1 -- 172.21.15.3:0/344246319 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccac0045c0 con 0x7fccb40c5270 2024-05-08T21:21:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.678+0000 7fccbb1c9700 1 -- 172.21.15.3:0/344246319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb4203570 con 0x7fccb40c5270 2024-05-08T21:21:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.678+0000 7fccbb1c9700 1 -- 172.21.15.3:0/344246319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb40c9bf0 con 0x7fccb40c5270 2024-05-08T21:21:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.678+0000 7fccb17fa700 1 -- 172.21.15.3:0/344246319 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac004970 con 0x7fccb40c5270 2024-05-08T21:21:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.679+0000 7fccbb1c9700 1 -- 172.21.15.3:0/344246319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb406e9a0 con 0x7fccb40c5270 2024-05-08T21:21:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.679+0000 7fccb17fa700 1 -- 172.21.15.3:0/344246319 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fccac019070 con 0x7fccb40c5270 2024-05-08T21:21:58.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.680+0000 7fccb17fa700 1 --2- 172.21.15.3:0/344246319 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcc9c06ae20 0x7fcc9c06d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.680+0000 7fccb17fa700 1 -- 172.21.15.3:0/344246319 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccac08a110 con 0x7fccb40c5270 2024-05-08T21:21:58.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.683+0000 7fccb8f65700 1 --2- 172.21.15.3:0/344246319 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcc9c06ae20 0x7fcc9c06d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.685+0000 7fbe7bbf2700 1 Processor -- start 2024-05-08T21:21:58.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.686+0000 7fbe7bbf2700 1 -- start start 2024-05-08T21:21:58.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.687+0000 7fbe7bbf2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe7415a800 0x7fbe740cba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.687+0000 7fbe7bbf2700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbe7406ed30 0x7fbe740ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.687+0000 7fbe7bbf2700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7415abe0 con 0x7fbe740cbf90 2024-05-08T21:21:58.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.687+0000 7fbe7bbf2700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe7406e8a0 con 0x7fbe7406ed30 2024-05-08T21:21:58.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.687+0000 7fbe7bbf2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7406dc90 con 0x7fbe7415a800 2024-05-08T21:21:58.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.688+0000 7fccb17fa700 1 -- 172.21.15.3:0/344246319 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fccac055380 con 0x7fccb40c5270 2024-05-08T21:21:58.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.689+0000 7fbe7998e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe7415a800 0x7fbe740cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.689+0000 7fbe7998e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe7415a800 0x7fbe740cba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.3:55866/0 (socket says 172.21.15.3:55866) 2024-05-08T21:21:58.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.689+0000 7fbe7998e700 1 -- 172.21.15.3:0/710228644 learned_addr learned my addr 172.21.15.3:0/710228644 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.690+0000 7fbe7918d700 1 --2- 172.21.15.3:0/710228644 >> v2:172.21.15.186:3300/0 conn(0x7fbe7406ed30 0x7fbe740ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.690+0000 7fbe7998e700 1 -- 172.21.15.3:0/710228644 >> v2:172.21.15.186:3300/0 conn(0x7fbe7406ed30 msgr2=0x7fbe740ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.690+0000 7fbe7998e700 1 --2- 172.21.15.3:0/710228644 >> v2:172.21.15.186:3300/0 conn(0x7fbe7406ed30 0x7fbe740ccc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.690+0000 7fbe7998e700 1 -- 172.21.15.3:0/710228644 >> v1:172.21.15.186:6789/0 conn(0x7fbe740cbf90 legacy=0x7fbe7415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.690+0000 7fbe7998e700 1 -- 172.21.15.3:0/710228644 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7406df70 con 0x7fbe7415a800 2024-05-08T21:21:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.691+0000 7fbe7998e700 1 --2- 172.21.15.3:0/710228644 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe7415a800 0x7fbe740cba50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbe6400bb90 tx=0x7fbe6400bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=793e28eedaebe6b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.691+0000 7fbe7898c700 1 -- 172.21.15.3:0/710228644 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6400dd10 con 0x7fbe7415a800 2024-05-08T21:21:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.691+0000 7fbe7898c700 1 -- 172.21.15.3:0/710228644 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6400de70 con 0x7fbe7415a800 2024-05-08T21:21:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.691+0000 7fbe7898c700 1 -- 172.21.15.3:0/710228644 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe64004230 con 0x7fbe7415a800 2024-05-08T21:21:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.692+0000 7fbe7bbf2700 1 -- 172.21.15.3:0/710228644 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe7415a800 msgr2=0x7fbe740cba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.692+0000 7fbe7bbf2700 1 --2- 172.21.15.3:0/710228644 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe7415a800 0x7fbe740cba50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbe6400bb90 tx=0x7fbe6400bea0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.692+0000 7fbe7bbf2700 1 -- 172.21.15.3:0/710228644 shutdown_connections 2024-05-08T21:21:58.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.692+0000 7fbe7bbf2700 1 --2- 172.21.15.3:0/710228644 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe7415a800 0x7fbe740cba50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.692+0000 7fbe7bbf2700 1 --2- 172.21.15.3:0/710228644 >> v2:172.21.15.186:3300/0 conn(0x7fbe7406ed30 0x7fbe740ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.692+0000 7fbe7bbf2700 1 -- 172.21.15.3:0/710228644 >> 172.21.15.3:0/710228644 conn(0x7fbe74099a50 msgr2=0x7fbe740d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.693+0000 7fbe7bbf2700 1 -- 172.21.15.3:0/710228644 shutdown_connections 2024-05-08T21:21:58.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.693+0000 7fbe7bbf2700 1 -- 172.21.15.3:0/710228644 wait complete. 2024-05-08T21:21:58.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.695+0000 7fbe7bbf2700 1 Processor -- start 2024-05-08T21:21:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.696+0000 7fbe7bbf2700 1 -- start start 2024-05-08T21:21:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.697+0000 7fbe7bbf2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe7406ed30 0x7fbe740c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.697+0000 7fbe7bbf2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe740cbf90 0x7fbe740b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.697+0000 7fbe7bbf2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe7415a800 0x7fbe740b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.697+0000 7fbe7bbf2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe740c5ea0 con 0x7fbe7406ed30 2024-05-08T21:21:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.697+0000 7fbe7bbf2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe740c99e0 con 0x7fbe7415a800 2024-05-08T21:21:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.697+0000 7fbe7bbf2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe740c5600 con 0x7fbe740cbf90 2024-05-08T21:21:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.693+0000 7fccb8f65700 1 --2- 172.21.15.3:0/344246319 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcc9c06ae20 0x7fcc9c06d2e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcca4009c00 tx=0x7fcca4009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.699+0000 7fbe7918d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe740cbf90 0x7fbe740b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.699+0000 7fbe7998e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe7406ed30 0x7fbe740c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.699+0000 7fbe7998e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe7406ed30 0x7fbe740c9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52482/0 (socket says 172.21.15.3:52482) 2024-05-08T21:21:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.699+0000 7fbe7998e700 1 -- 172.21.15.3:0/1725045796 learned_addr learned my addr 172.21.15.3:0/1725045796 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.699+0000 7fbe7a18f700 1 --2- 172.21.15.3:0/1725045796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe7415a800 0x7fbe740b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.699+0000 7fbe7998e700 1 -- 172.21.15.3:0/1725045796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe740cbf90 msgr2=0x7fbe740b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.699+0000 7fbe7998e700 1 --2- 172.21.15.3:0/1725045796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe740cbf90 0x7fbe740b5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.699+0000 7fbe7998e700 1 -- 172.21.15.3:0/1725045796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe7415a800 msgr2=0x7fbe740b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.699+0000 7fbe7998e700 1 --2- 172.21.15.3:0/1725045796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe7415a800 0x7fbe740b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.699+0000 7fbe7998e700 1 -- 172.21.15.3:0/1725045796 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6400b7f0 con 0x7fbe7406ed30 2024-05-08T21:21:58.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.700+0000 7fbe7998e700 1 --2- 172.21.15.3:0/1725045796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe7406ed30 0x7fbe740c9ec0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbe7406ddd0 tx=0x7fbe64004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.700+0000 7fbe6a7fc700 1 -- 172.21.15.3:0/1725045796 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6401e070 con 0x7fbe7406ed30 2024-05-08T21:21:58.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.700+0000 7fbe6a7fc700 1 -- 172.21.15.3:0/1725045796 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe64003b10 con 0x7fbe7406ed30 2024-05-08T21:21:58.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.700+0000 7fbe6a7fc700 1 -- 172.21.15.3:0/1725045796 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe64010470 con 0x7fbe7406ed30 2024-05-08T21:21:58.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.700+0000 7fbe7bbf2700 1 -- 172.21.15.3:0/1725045796 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe740c58e0 con 0x7fbe7406ed30 2024-05-08T21:21:58.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.701+0000 7fbe7bbf2700 1 -- 172.21.15.3:0/1725045796 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe740ca400 con 0x7fbe7406ed30 2024-05-08T21:21:58.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.702+0000 7fbe7bbf2700 1 -- 172.21.15.3:0/1725045796 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe740c9ba0 con 0x7fbe7406ed30 2024-05-08T21:21:58.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.703+0000 7f6c9ee41700 1 Processor -- start 2024-05-08T21:21:58.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.703+0000 7f6c9ee41700 1 -- start start 2024-05-08T21:21:58.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.704+0000 7f6c9ee41700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6c9815f900 0x7f6c9806ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.704+0000 7f6c9ee41700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c981628e0 0x7f6c98071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.704+0000 7f6c9ee41700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c980711d0 con 0x7f6c9806b2c0 2024-05-08T21:21:58.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.704+0000 7f6c9ee41700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9806b750 con 0x7f6c9815f900 2024-05-08T21:21:58.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.704+0000 7f6c9ee41700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c98070f30 con 0x7f6c981628e0 2024-05-08T21:21:58.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.705+0000 7f6c97fff700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6c9806b2c0 0x7f6c981621d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:44376/0 (socket says 172.21.15.3:44376) 2024-05-08T21:21:58.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.705+0000 7f6c97fff700 1 -- 172.21.15.3:0/2952841225 learned_addr learned my addr 172.21.15.3:0/2952841225 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:58.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c977fe700 1 -- 172.21.15.3:0/2952841225 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145414435 0 0) 0x7f6c980711d0 con 0x7f6c9806b2c0 2024-05-08T21:21:58.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c977fe700 1 -- 172.21.15.3:0/2952841225 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c840035b0 con 0x7f6c9806b2c0 2024-05-08T21:21:58.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c977fe700 1 -- 172.21.15.3:0/2952841225 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2898750185 0 0) 0x7f6c840035b0 con 0x7f6c9806b2c0 2024-05-08T21:21:58.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c977fe700 1 -- 172.21.15.3:0/2952841225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c981628e0 msgr2=0x7f6c98071c70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-08T21:21:58.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c977fe700 1 --2- 172.21.15.3:0/2952841225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c981628e0 0x7f6c98071c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c977fe700 1 -- 172.21.15.3:0/2952841225 >> v2:172.21.15.3:3300/0 conn(0x7f6c9815f900 msgr2=0x7f6c9806ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c977fe700 1 --2- 172.21.15.3:0/2952841225 >> v2:172.21.15.3:3300/0 conn(0x7f6c9815f900 0x7f6c9806ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c977fe700 1 -- 172.21.15.3:0/2952841225 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c98162e30 con 0x7f6c9806b2c0 2024-05-08T21:21:58.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c977fe700 1 -- 172.21.15.3:0/2952841225 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f6c900019f0 con 0x7f6c9806b2c0 2024-05-08T21:21:58.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c977fe700 1 -- 172.21.15.3:0/2952841225 >> v1:172.21.15.186:6789/0 conn(0x7f6c9806b2c0 legacy=0x7f6c981621d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c977fe700 1 --2- 172.21.15.3:0/2952841225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c84003a30 0x7f6c84003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c977fe700 1 --2- 172.21.15.3:0/2952841225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c84004850 0x7f6c84004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c977fe700 1 --2- 172.21.15.3:0/2952841225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c84005330 0x7f6c8400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c977fe700 1 -- 172.21.15.3:0/2952841225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9806b750 con 0x7f6c84004850 2024-05-08T21:21:58.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c977fe700 1 -- 172.21.15.3:0/2952841225 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c98070f30 con 0x7f6c84003a30 2024-05-08T21:21:58.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c977fe700 1 -- 172.21.15.3:0/2952841225 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8400c230 con 0x7f6c84005330 2024-05-08T21:21:58.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f6c9cbdd700 1 --2- 172.21.15.3:0/2952841225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c84003a30 0x7f6c84003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.708+0000 7f6c9cbdd700 1 -- 172.21.15.3:0/2952841225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c84005330 msgr2=0x7f6c8400ba10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-08T21:21:58.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.708+0000 7f6c9cbdd700 1 --2- 172.21.15.3:0/2952841225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c84005330 0x7f6c8400ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.708+0000 7f6c9cbdd700 1 -- 172.21.15.3:0/2952841225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c84004850 msgr2=0x7f6c84004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.708+0000 7f6c97fff700 1 --2- 172.21.15.3:0/2952841225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c84004850 0x7f6c84004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.704+0000 7f1ed2344700 1 Processor -- start 2024-05-08T21:21:58.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.705+0000 7f1ed2344700 1 -- start start 2024-05-08T21:21:58.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.706+0000 7f1ed2344700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ecc15c380 0x7f1ecc0b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.706+0000 7f1ed2344700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1ecc0b2960 0x7f1ecc0b1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.706+0000 7f1ed2344700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ecc06ce20 con 0x7f1ecc163440 2024-05-08T21:21:58.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.706+0000 7f1ed2344700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ecc06da10 con 0x7f1ecc0b2960 2024-05-08T21:21:58.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.706+0000 7f1ed2344700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc1656b0 con 0x7f1ecc15c380 2024-05-08T21:21:58.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.706+0000 7f1ecbfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ecc15c380 0x7f1ecc0b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.706+0000 7f1ecb7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1ecc163440 0x7f1ecc0b2250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:33086/0 (socket says 172.21.15.3:33086) 2024-05-08T21:21:58.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f1ecb7fe700 1 -- 172.21.15.3:0/3762076453 learned_addr learned my addr 172.21.15.3:0/3762076453 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:21:58.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f1ed08e1700 1 --2- 172.21.15.3:0/3762076453 >> v2:172.21.15.186:3300/0 conn(0x7f1ecc0b2960 0x7f1ecc0b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.708+0000 7f6c9cbdd700 1 --2- 172.21.15.3:0/2952841225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c84004850 0x7f6c84004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.708+0000 7f6c9cbdd700 1 -- 172.21.15.3:0/2952841225 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c98162e30 con 0x7f6c84003a30 2024-05-08T21:21:58.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.708+0000 7f6c97fff700 1 --2- 172.21.15.3:0/2952841225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c84004850 0x7f6c84004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:21:58.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.708+0000 7f6c9cbdd700 1 --2- 172.21.15.3:0/2952841225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c84003a30 0x7f6c84003e50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009910 tx=0x7f6c84005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecf724a077327d19 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.709+0000 7f6c977fe700 1 -- 172.21.15.3:0/2952841225 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80011620 con 0x7f6c84003a30 2024-05-08T21:21:58.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.709+0000 7f6c977fe700 1 -- 172.21.15.3:0/2952841225 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c80011780 con 0x7f6c84003a30 2024-05-08T21:21:58.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.709+0000 7f6c977fe700 1 -- 172.21.15.3:0/2952841225 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80011b10 con 0x7f6c84003a30 2024-05-08T21:21:58.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.710+0000 7f6c9ee41700 1 -- 172.21.15.3:0/2952841225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c84003a30 msgr2=0x7f6c84003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.710+0000 7f6c9ee41700 1 --2- 172.21.15.3:0/2952841225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c84003a30 0x7f6c84003e50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009910 tx=0x7f6c84005880 comp rx=0 tx=0).stop 2024-05-08T21:21:58.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f1ecbfff700 1 -- 172.21.15.3:0/3762076453 >> v2:172.21.15.186:3300/0 conn(0x7f1ecc0b2960 msgr2=0x7f1ecc0b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:21:58.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f1ecbfff700 1 --2- 172.21.15.3:0/3762076453 >> v2:172.21.15.186:3300/0 conn(0x7f1ecc0b2960 0x7f1ecc0b1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f1ecbfff700 1 -- 172.21.15.3:0/3762076453 >> v1:172.21.15.3:6789/0 conn(0x7f1ecc163440 legacy=0x7f1ecc0b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f1ecbfff700 1 -- 172.21.15.3:0/3762076453 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ecc15c760 con 0x7f1ecc15c380 2024-05-08T21:21:58.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.707+0000 7f1ecbfff700 1 --2- 172.21.15.3:0/3762076453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ecc15c380 0x7f1ecc0b0e00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0009840 tx=0x7f1ec0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=807bce147e405ed server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.710+0000 7f1ecaffd700 1 -- 172.21.15.3:0/3762076453 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0011830 con 0x7f1ecc15c380 2024-05-08T21:21:58.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.710+0000 7f1ecaffd700 1 -- 172.21.15.3:0/3762076453 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec0011990 con 0x7f1ecc15c380 2024-05-08T21:21:58.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.710+0000 7f1ecaffd700 1 -- 172.21.15.3:0/3762076453 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0011d20 con 0x7f1ecc15c380 2024-05-08T21:21:58.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.711+0000 7f1ed2344700 1 -- 172.21.15.3:0/3762076453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ecc15c380 msgr2=0x7f1ecc0b0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.711+0000 7f1ed2344700 1 --2- 172.21.15.3:0/3762076453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ecc15c380 0x7f1ecc0b0e00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0009840 tx=0x7f1ec0009b50 comp rx=0 tx=0).stop 2024-05-08T21:21:58.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.712+0000 7f1ed2344700 1 -- 172.21.15.3:0/3762076453 shutdown_connections 2024-05-08T21:21:58.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.712+0000 7f1ed2344700 1 --2- 172.21.15.3:0/3762076453 >> v2:172.21.15.186:3300/0 conn(0x7f1ecc0b2960 0x7f1ecc0b1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.712+0000 7f1ed2344700 1 --2- 172.21.15.3:0/3762076453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ecc15c380 0x7f1ecc0b0e00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.712+0000 7f1ed2344700 1 -- 172.21.15.3:0/3762076453 >> 172.21.15.3:0/3762076453 conn(0x7f1ecc0b4690 msgr2=0x7f1ecc0b4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.712+0000 7f6c9d3de700 1 -- 172.21.15.3:0/2952841225 reap_dead start 2024-05-08T21:21:58.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.712+0000 7f6c9ee41700 1 -- 172.21.15.3:0/2952841225 shutdown_connections 2024-05-08T21:21:58.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.712+0000 7f6c9ee41700 1 -- 172.21.15.3:0/2952841225 >> 172.21.15.3:0/2952841225 conn(0x7f6c980b1290 msgr2=0x7f6c980b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:58.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.712+0000 7f6c9ee41700 1 -- 172.21.15.3:0/2952841225 shutdown_connections 2024-05-08T21:21:58.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.714+0000 7fbe6a7fc700 1 -- 172.21.15.3:0/1725045796 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbe6402b3f0 con 0x7fbe7406ed30 2024-05-08T21:21:58.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.714+0000 7fbe6a7fc700 1 --2- 172.21.15.3:0/1725045796 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fbe5806ad60 0x7fbe5806d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.715+0000 7fbe6a7fc700 1 -- 172.21.15.3:0/1725045796 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe6408a550 con 0x7fbe7406ed30 2024-05-08T21:21:58.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.715+0000 7fbe6a7fc700 1 -- 172.21.15.3:0/1725045796 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fbe6408a9d0 con 0x7fbe7406ed30 2024-05-08T21:21:58.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.715+0000 7fbe7918d700 1 --2- 172.21.15.3:0/1725045796 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fbe5806ad60 0x7fbe5806d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.712+0000 7f1ed2344700 1 -- 172.21.15.3:0/3762076453 shutdown_connections 2024-05-08T21:21:58.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.716+0000 7fbe7918d700 1 --2- 172.21.15.3:0/1725045796 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fbe5806ad60 0x7fbe5806d220 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbe740c72a0 tx=0x7fbe70008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.716+0000 7f1ed2344700 1 -- 172.21.15.3:0/3762076453 wait complete. 2024-05-08T21:21:58.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.712+0000 7f6c9ee41700 1 -- 172.21.15.3:0/2952841225 wait complete. 2024-05-08T21:21:58.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.714+0000 7f6c9ee41700 1 Processor -- start 2024-05-08T21:21:58.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.715+0000 7f6c9ee41700 1 -- start start 2024-05-08T21:21:58.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.716+0000 7f6c9ee41700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9809a8c0 0x7f6c98088450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.716+0000 7f6c9ee41700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c9809aca0 0x7f6c981416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.716+0000 7f6c9ee41700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c98141c10 0x7f6c9809ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.716+0000 7f6c9ee41700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c980ad4d0 con 0x7f6c9809a8c0 2024-05-08T21:21:58.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.716+0000 7f6c9ee41700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9809b100 con 0x7f6c9809aca0 2024-05-08T21:21:58.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.716+0000 7f6c9ee41700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c98088a30 con 0x7f6c98141c10 2024-05-08T21:21:58.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.716+0000 7f6c9d3de700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c98141c10 0x7f6c9809ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.717+0000 7f6c97fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c9809aca0 0x7f6c981416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.717+0000 7f6c97fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c9809aca0 0x7f6c981416d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:38540/0 (socket says 172.21.15.3:38540) 2024-05-08T21:21:58.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.717+0000 7f6c97fff700 1 -- 172.21.15.3:0/4166302195 learned_addr learned my addr 172.21.15.3:0/4166302195 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.718+0000 7f6c97fff700 1 -- 172.21.15.3:0/4166302195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c98141c10 msgr2=0x7f6c9809ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.718+0000 7f6c97fff700 1 --2- 172.21.15.3:0/4166302195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c98141c10 0x7f6c9809ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.718+0000 7f6c97fff700 1 -- 172.21.15.3:0/4166302195 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9809a8c0 msgr2=0x7f6c98088450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.718+0000 7f6c97fff700 1 --2- 172.21.15.3:0/4166302195 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9809a8c0 0x7f6c98088450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.718+0000 7f6c97fff700 1 -- 172.21.15.3:0/4166302195 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c98088d10 con 0x7f6c9809aca0 2024-05-08T21:21:58.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.718+0000 7f6c97fff700 1 --2- 172.21.15.3:0/4166302195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c9809aca0 0x7f6c981416d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000b1a0 tx=0x7f6c9000b4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.718+0000 7f6c957fa700 1 -- 172.21.15.3:0/4166302195 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c900125d0 con 0x7f6c9809aca0 2024-05-08T21:21:58.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.718+0000 7f6c957fa700 1 -- 172.21.15.3:0/4166302195 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c90012730 con 0x7f6c9809aca0 2024-05-08T21:21:58.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.718+0000 7f6c957fa700 1 -- 172.21.15.3:0/4166302195 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90012ac0 con 0x7f6c9809aca0 2024-05-08T21:21:58.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.718+0000 7f6c9ee41700 1 -- 172.21.15.3:0/4166302195 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c98088ff0 con 0x7f6c9809aca0 2024-05-08T21:21:58.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.718+0000 7f6c9ee41700 1 -- 172.21.15.3:0/4166302195 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c980892d0 con 0x7f6c9809aca0 2024-05-08T21:21:58.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.719+0000 7f1ed2344700 1 Processor -- start 2024-05-08T21:21:58.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.722+0000 7f6c957fa700 1 -- 172.21.15.3:0/4166302195 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6c90012c20 con 0x7f6c9809aca0 2024-05-08T21:21:58.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.722+0000 7f6c957fa700 1 --2- 172.21.15.3:0/4166302195 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6c7806ad60 0x7f6c7806d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.722+0000 7f6c957fa700 1 -- 172.21.15.3:0/4166302195 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c90089a90 con 0x7f6c9809aca0 2024-05-08T21:21:58.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.721+0000 7f1ed2344700 1 -- start start 2024-05-08T21:21:58.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.722+0000 7f1ed2344700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ecc0b2960 0x7f1ecc1f7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.722+0000 7f1ed2344700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ecc15c380 0x7f1ecc157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.722+0000 7f1ed2344700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ecc163440 0x7f1ecc159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.722+0000 7f1ed2344700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc0ac150 con 0x7f1ecc15c380 2024-05-08T21:21:58.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.722+0000 7f1ed2344700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc15e690 con 0x7f1ecc0b2960 2024-05-08T21:21:58.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.722+0000 7f1ed2344700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc1f7e60 con 0x7f1ecc163440 2024-05-08T21:21:58.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.722+0000 7f1ecbfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ecc0b2960 0x7f1ecc1f7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.723+0000 7f1ed08e1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ecc163440 0x7f1ecc159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.723+0000 7f1ed08e1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ecc163440 0x7f1ecc159180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.3:55900/0 (socket says 172.21.15.3:55900) 2024-05-08T21:21:58.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.723+0000 7f1ed08e1700 1 -- 172.21.15.3:0/792403197 learned_addr learned my addr 172.21.15.3:0/792403197 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:21:58.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.723+0000 7f6c9cbdd700 1 --2- 172.21.15.3:0/4166302195 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6c7806ad60 0x7f6c7806d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.723+0000 7f6c9cbdd700 1 --2- 172.21.15.3:0/4166302195 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6c7806ad60 0x7f6c7806d220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6c80000c00 tx=0x7f6c80010000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.723+0000 7f1ecbfff700 1 -- 172.21.15.3:0/792403197 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ecc163440 msgr2=0x7f1ecc159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.723+0000 7f1ecbfff700 1 --2- 172.21.15.3:0/792403197 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ecc163440 0x7f1ecc159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.723+0000 7f1ecbfff700 1 -- 172.21.15.3:0/792403197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ecc15c380 msgr2=0x7f1ecc157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:58.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.723+0000 7f1ecbfff700 1 --2- 172.21.15.3:0/792403197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ecc15c380 0x7f1ecc157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:58.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.723+0000 7f1ecbfff700 1 -- 172.21.15.3:0/792403197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec00094a0 con 0x7f1ecc0b2960 2024-05-08T21:21:58.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.724+0000 7f6c9ee41700 1 -- 172.21.15.3:0/4166302195 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c7c0052f0 con 0x7f6c9809aca0 2024-05-08T21:21:58.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.724+0000 7f1ecbfff700 1 --2- 172.21.15.3:0/792403197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ecc0b2960 0x7f1ecc1f7920 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0016000 tx=0x7f1ec0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.729+0000 7f1ec8ff9700 1 -- 172.21.15.3:0/792403197 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0004590 con 0x7f1ecc0b2960 2024-05-08T21:21:58.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.729+0000 7f1ec8ff9700 1 -- 172.21.15.3:0/792403197 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec00036a0 con 0x7f1ecc0b2960 2024-05-08T21:21:58.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.729+0000 7f1ec8ff9700 1 -- 172.21.15.3:0/792403197 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0003a30 con 0x7f1ecc0b2960 2024-05-08T21:21:58.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.729+0000 7f1ed2344700 1 -- 172.21.15.3:0/792403197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ecc0ac330 con 0x7f1ecc0b2960 2024-05-08T21:21:58.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.729+0000 7f6c957fa700 1 -- 172.21.15.3:0/4166302195 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6c90019020 con 0x7f6c9809aca0 2024-05-08T21:21:58.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.730+0000 7f1ed2344700 1 -- 172.21.15.3:0/792403197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ecc0ac8f0 con 0x7f1ecc0b2960 2024-05-08T21:21:58.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.732+0000 7f1ec8ff9700 1 -- 172.21.15.3:0/792403197 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1ec0003b90 con 0x7f1ecc0b2960 2024-05-08T21:21:58.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.732+0000 7f1ec8ff9700 1 --2- 172.21.15.3:0/792403197 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1eb806ac90 0x7f1eb806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:21:58.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.733+0000 7f1ec8ff9700 1 -- 172.21.15.3:0/792403197 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ec0003e40 con 0x7f1ecc0b2960 2024-05-08T21:21:58.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.733+0000 7f1ecb7fe700 1 --2- 172.21.15.3:0/792403197 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1eb806ac90 0x7f1eb806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:21:58.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.733+0000 7f1ecb7fe700 1 --2- 172.21.15.3:0/792403197 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1eb806ac90 0x7f1eb806d150 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc0015d0 tx=0x7f1ebc007ad0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:21:58.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.740+0000 7f1ed2344700 1 -- 172.21.15.3:0/792403197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ecc0678f0 con 0x7f1ecc0b2960 2024-05-08T21:21:58.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:58.746+0000 7f1ec8ff9700 1 -- 172.21.15.3:0/792403197 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f1ec008f060 con 0x7f1ecc0b2960 2024-05-08T21:21:59.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.073+0000 7fe1e0130700 1 -- 172.21.15.3:0/3321901447 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fe1d80678f0 con 0x7fe1d806e4f0 2024-05-08T21:21:59.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.074+0000 7fe1ce7fc700 1 -- 172.21.15.3:0/3321901447 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe1d40593f0 con 0x7fe1d806e4f0 2024-05-08T21:21:59.077 INFO:teuthology.orchestra.run.smithi003.stdout:55834574854 2024-05-08T21:21:59.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.076+0000 7ff2abfbc700 1 -- 172.21.15.3:0/2894462281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff2a4068000 con 0x7ff2a4163e30 2024-05-08T21:21:59.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.077+0000 7ff29a7fc700 1 -- 172.21.15.3:0/2894462281 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff2a0059160 con 0x7ff2a4163e30 2024-05-08T21:21:59.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.077+0000 7fe1e0130700 1 -- 172.21.15.3:0/3321901447 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe1bc06ac90 msgr2=0x7fe1bc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:59.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.078+0000 7fe1e0130700 1 --2- 172.21.15.3:0/3321901447 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe1bc06ac90 0x7fe1bc06d150 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8006e70 tx=0x7fe1c8006c00 comp rx=0 tx=0).stop 2024-05-08T21:21:59.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.078+0000 7fe1e0130700 1 -- 172.21.15.3:0/3321901447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1d806e4f0 msgr2=0x7fe1d80af080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:59.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.078+0000 7fe1e0130700 1 --2- 172.21.15.3:0/3321901447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1d806e4f0 0x7fe1d80af080 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4000c00 tx=0x7fe1d4004750 comp rx=0 tx=0).stop 2024-05-08T21:21:59.080 INFO:teuthology.orchestra.run.smithi003.stdout:55834574856 2024-05-08T21:21:59.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.078+0000 7fe1e0130700 1 -- 172.21.15.3:0/3321901447 shutdown_connections 2024-05-08T21:21:59.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.078+0000 7fe1e0130700 1 --2- 172.21.15.3:0/3321901447 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1d80b1340 0x7fe1d8099760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.078+0000 7fe1e0130700 1 --2- 172.21.15.3:0/3321901447 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe1bc06ac90 0x7fe1bc06d150 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.078+0000 7fe1e0130700 1 --2- 172.21.15.3:0/3321901447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1d806e4f0 0x7fe1d80af080 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.078+0000 7fe1e0130700 1 --2- 172.21.15.3:0/3321901447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1d8098a80 0x7fe1d8214890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.078+0000 7fe1e0130700 1 -- 172.21.15.3:0/3321901447 >> 172.21.15.3:0/3321901447 conn(0x7fe1d80b4690 msgr2=0x7fe1d8096ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:59.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.078+0000 7fe1e0130700 1 -- 172.21.15.3:0/3321901447 shutdown_connections 2024-05-08T21:21:59.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.079+0000 7fe1e0130700 1 -- 172.21.15.3:0/3321901447 wait complete. 2024-05-08T21:21:59.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.082+0000 7ff2abfbc700 1 -- 172.21.15.3:0/2894462281 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff29006c720 msgr2=0x7ff29006ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:59.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.082+0000 7ff2abfbc700 1 --2- 172.21.15.3:0/2894462281 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff29006c720 0x7ff29006ebe0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff294006b20 tx=0x7ff294008040 comp rx=0 tx=0).stop 2024-05-08T21:21:59.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.082+0000 7ff2abfbc700 1 -- 172.21.15.3:0/2894462281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2a4163e30 msgr2=0x7ff2a41642b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:59.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.082+0000 7ff2abfbc700 1 --2- 172.21.15.3:0/2894462281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2a4163e30 0x7ff2a41642b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0013040 tx=0x7ff2a0004750 comp rx=0 tx=0).stop 2024-05-08T21:21:59.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.083+0000 7ff2abfbc700 1 -- 172.21.15.3:0/2894462281 shutdown_connections 2024-05-08T21:21:59.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.083+0000 7ff2abfbc700 1 --2- 172.21.15.3:0/2894462281 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2a40cdaa0 0x7ff2a40cde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.083+0000 7ff2abfbc700 1 --2- 172.21.15.3:0/2894462281 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff29006c720 0x7ff29006ebe0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.083+0000 7ff2abfbc700 1 --2- 172.21.15.3:0/2894462281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2a41451e0 0x7ff2a4145660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.083+0000 7ff2abfbc700 1 --2- 172.21.15.3:0/2894462281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2a4163e30 0x7ff2a41642b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.083+0000 7ff2abfbc700 1 -- 172.21.15.3:0/2894462281 >> 172.21.15.3:0/2894462281 conn(0x7ff2a406af90 msgr2=0x7ff2a40af5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:59.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.085+0000 7ff2abfbc700 1 -- 172.21.15.3:0/2894462281 shutdown_connections 2024-05-08T21:21:59.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.085+0000 7ff2abfbc700 1 -- 172.21.15.3:0/2894462281 wait complete. 2024-05-08T21:21:59.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.096+0000 7f069f477700 1 -- 172.21.15.3:0/2484101751 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f0678005870 con 0x7f06980b22d0 2024-05-08T21:21:59.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.098+0000 7f068dffb700 1 -- 172.21.15.3:0/2484101751 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0688015090 con 0x7f06980b22d0 2024-05-08T21:21:59.103 INFO:teuthology.orchestra.run.smithi003.stdout:55834574854 2024-05-08T21:21:59.103 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.6 2024-05-08T21:21:59.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.103+0000 7f069f477700 1 -- 172.21.15.3:0/2484101751 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f067c06ac90 msgr2=0x7f067c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:59.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.103+0000 7f069f477700 1 --2- 172.21.15.3:0/2484101751 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f067c06ac90 0x7f067c06d150 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0694006fd0 tx=0x7f0694008040 comp rx=0 tx=0).stop 2024-05-08T21:21:59.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.103+0000 7f069f477700 1 -- 172.21.15.3:0/2484101751 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f06980b22d0 msgr2=0x7f06980c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:59.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.103+0000 7f069f477700 1 --2- 172.21.15.3:0/2484101751 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f06980b22d0 0x7f06980c7380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0688014040 tx=0x7f0688004750 comp rx=0 tx=0).stop 2024-05-08T21:21:59.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.104+0000 7f069f477700 1 -- 172.21.15.3:0/2484101751 shutdown_connections 2024-05-08T21:21:59.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.104+0000 7f069f477700 1 --2- 172.21.15.3:0/2484101751 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f06980b22d0 0x7f06980c7380 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.104+0000 7f069f477700 1 --2- 172.21.15.3:0/2484101751 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f067c06ac90 0x7f067c06d150 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.104+0000 7f069f477700 1 --2- 172.21.15.3:0/2484101751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0698165820 0x7f069806ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.104+0000 7f069f477700 1 --2- 172.21.15.3:0/2484101751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0698161ee0 0x7f06980c8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.104+0000 7f069f477700 1 -- 172.21.15.3:0/2484101751 >> 172.21.15.3:0/2484101751 conn(0x7f06980d3620 msgr2=0x7f069806bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:59.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.106+0000 7f069f477700 1 -- 172.21.15.3:0/2484101751 shutdown_connections 2024-05-08T21:21:59.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.106+0000 7f069f477700 1 -- 172.21.15.3:0/2484101751 wait complete. 2024-05-08T21:21:59.110 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.0 2024-05-08T21:21:59.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.114+0000 7f90f5ce5700 1 -- 172.21.15.3:0/3848415122 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f90d0005870 con 0x7f90f015f900 2024-05-08T21:21:59.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.126+0000 7f90d7fff700 1 -- 172.21.15.3:0/3848415122 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f90e80580f0 con 0x7f90f015f900 2024-05-08T21:21:59.127 INFO:teuthology.orchestra.run.smithi003.stdout:55834574855 2024-05-08T21:21:59.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2024-05-08T21:21:59.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.129+0000 7f90f5ce5700 1 -- 172.21.15.3:0/3848415122 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f90dc06ace0 msgr2=0x7f90dc06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:59.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.129+0000 7f90f5ce5700 1 --2- 172.21.15.3:0/3848415122 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f90dc06ace0 0x7f90dc06d1a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f90e4006ba0 tx=0x7f90e4008040 comp rx=0 tx=0).stop 2024-05-08T21:21:59.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.129+0000 7f90f5ce5700 1 -- 172.21.15.3:0/3848415122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90f015f900 msgr2=0x7f90f00adbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:59.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.129+0000 7f90f5ce5700 1 --2- 172.21.15.3:0/3848415122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90f015f900 0x7f90f00adbd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f90e8002080 tx=0x7f90e800a340 comp rx=0 tx=0).stop 2024-05-08T21:21:59.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.131+0000 7f90f5ce5700 1 -- 172.21.15.3:0/3848415122 shutdown_connections 2024-05-08T21:21:59.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.131+0000 7f90f5ce5700 1 --2- 172.21.15.3:0/3848415122 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f90f01628e0 0x7f90f009cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.131+0000 7f90f5ce5700 1 --2- 172.21.15.3:0/3848415122 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f90dc06ace0 0x7f90dc06d1a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.131+0000 7f90f5ce5700 1 --2- 172.21.15.3:0/3848415122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90f006b2c0 0x7f90f00ac150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.131+0000 7f90f5ce5700 1 --2- 172.21.15.3:0/3848415122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90f015f900 0x7f90f00adbd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.131+0000 7f90f5ce5700 1 -- 172.21.15.3:0/3848415122 >> 172.21.15.3:0/3848415122 conn(0x7f90f00b1290 msgr2=0x7f90f00b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:59.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.131+0000 7f90f5ce5700 1 -- 172.21.15.3:0/3848415122 shutdown_connections 2024-05-08T21:21:59.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.131+0000 7f90f5ce5700 1 -- 172.21.15.3:0/3848415122 wait complete. 2024-05-08T21:21:59.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.132+0000 7fccbb1c9700 1 -- 172.21.15.3:0/344246319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fccb40678f0 con 0x7fccb40c5270 2024-05-08T21:21:59.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.132+0000 7fccb17fa700 1 -- 172.21.15.3:0/344246319 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fccac0589a0 con 0x7fccb40c5270 2024-05-08T21:21:59.134 INFO:teuthology.orchestra.run.smithi003.stdout:55834574856 2024-05-08T21:21:59.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.134+0000 7fccbb1c9700 1 -- 172.21.15.3:0/344246319 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcc9c06ae20 msgr2=0x7fcc9c06d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:59.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.134+0000 7fccbb1c9700 1 --2- 172.21.15.3:0/344246319 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcc9c06ae20 0x7fcc9c06d2e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcca4009c00 tx=0x7fcca4009210 comp rx=0 tx=0).stop 2024-05-08T21:21:59.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.134+0000 7fccbb1c9700 1 -- 172.21.15.3:0/344246319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccb40c5270 msgr2=0x7fccb40c5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.134+0000 7fccbb1c9700 1 --2- 172.21.15.3:0/344246319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccb40c5270 0x7fccb40c5700 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fccb40cef90 tx=0x7fccac004ad0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.134+0000 7fccbb1c9700 1 -- 172.21.15.3:0/344246319 shutdown_connections 2024-05-08T21:21:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.134+0000 7fccbb1c9700 1 --2- 172.21.15.3:0/344246319 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcc9c06ae20 0x7fcc9c06d2e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.134+0000 7fccbb1c9700 1 --2- 172.21.15.3:0/344246319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccb40c5270 0x7fccb40c5700 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.134+0000 7fccbb1c9700 1 --2- 172.21.15.3:0/344246319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb409a160 0x7fccb40ca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.134+0000 7fccbb1c9700 1 --2- 172.21.15.3:0/344246319 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccb415ba50 0x7fccb4203030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.134+0000 7fccbb1c9700 1 -- 172.21.15.3:0/344246319 >> 172.21.15.3:0/344246319 conn(0x7fccb40b2f00 msgr2=0x7fccb40ce5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.135+0000 7fccbb1c9700 1 -- 172.21.15.3:0/344246319 shutdown_connections 2024-05-08T21:21:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.135+0000 7fccbb1c9700 1 -- 172.21.15.3:0/344246319 wait complete. 2024-05-08T21:21:59.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.148+0000 7fbe7bbf2700 1 -- 172.21.15.3:0/1725045796 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fbe7415abe0 con 0x7fbe7406ed30 2024-05-08T21:21:59.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.148+0000 7fbe6a7fc700 1 -- 172.21.15.3:0/1725045796 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbe7415abe0 con 0x7fbe7406ed30 2024-05-08T21:21:59.150 INFO:teuthology.orchestra.run.smithi003.stdout:55834574855 2024-05-08T21:21:59.152 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.3 2024-05-08T21:21:59.152 DEBUG:teuthology.parallel:result is None 2024-05-08T21:21:59.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.152+0000 7fbe7bbf2700 1 -- 172.21.15.3:0/1725045796 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fbe5806ad60 msgr2=0x7fbe5806d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:59.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.152+0000 7fbe7bbf2700 1 --2- 172.21.15.3:0/1725045796 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fbe5806ad60 0x7fbe5806d220 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbe740c72a0 tx=0x7fbe70008040 comp rx=0 tx=0).stop 2024-05-08T21:21:59.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.153+0000 7fbe7bbf2700 1 -- 172.21.15.3:0/1725045796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe7406ed30 msgr2=0x7fbe740c9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:59.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.153+0000 7fbe7bbf2700 1 --2- 172.21.15.3:0/1725045796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe7406ed30 0x7fbe740c9ec0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbe7406ddd0 tx=0x7fbe64004750 comp rx=0 tx=0).stop 2024-05-08T21:21:59.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.153+0000 7fbe7bbf2700 1 -- 172.21.15.3:0/1725045796 shutdown_connections 2024-05-08T21:21:59.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.153+0000 7fbe7bbf2700 1 --2- 172.21.15.3:0/1725045796 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fbe5806ad60 0x7fbe5806d220 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.153+0000 7fbe7bbf2700 1 --2- 172.21.15.3:0/1725045796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe7415a800 0x7fbe740b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.153+0000 7fbe7bbf2700 1 --2- 172.21.15.3:0/1725045796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe7406ed30 0x7fbe740c9ec0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.153+0000 7fbe7bbf2700 1 --2- 172.21.15.3:0/1725045796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe740cbf90 0x7fbe740b5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.153+0000 7fbe7bbf2700 1 -- 172.21.15.3:0/1725045796 >> 172.21.15.3:0/1725045796 conn(0x7fbe74099a50 msgr2=0x7fbe740ad710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:59.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.153+0000 7fbe7bbf2700 1 -- 172.21.15.3:0/1725045796 shutdown_connections 2024-05-08T21:21:59.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.153+0000 7fbe7bbf2700 1 -- 172.21.15.3:0/1725045796 wait complete. 2024-05-08T21:21:59.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.156+0000 7f6c9ee41700 1 -- 172.21.15.3:0/4166302195 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6c7c0055a0 con 0x7f6c9809aca0 2024-05-08T21:21:59.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.157+0000 7f6c957fa700 1 -- 172.21.15.3:0/4166302195 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6c90054c50 con 0x7f6c9809aca0 2024-05-08T21:21:59.158 INFO:teuthology.orchestra.run.smithi003.stdout:55834574855 2024-05-08T21:21:59.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.157+0000 7f1ed2344700 1 -- 172.21.15.3:0/792403197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1ecc0b19a0 con 0x7f1ecc0b2960 2024-05-08T21:21:59.160 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2024-05-08T21:21:59.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.158+0000 7f1ec8ff9700 1 -- 172.21.15.3:0/792403197 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1ec00551e0 con 0x7f1ecc0b2960 2024-05-08T21:21:59.160 INFO:teuthology.orchestra.run.smithi003.stdout:55834574855 2024-05-08T21:21:59.160 DEBUG:teuthology.parallel:result is None 2024-05-08T21:21:59.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.160+0000 7f6c9ee41700 1 -- 172.21.15.3:0/4166302195 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6c7806ad60 msgr2=0x7f6c7806d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:59.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.160+0000 7f6c9ee41700 1 --2- 172.21.15.3:0/4166302195 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6c7806ad60 0x7f6c7806d220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6c80000c00 tx=0x7f6c80010000 comp rx=0 tx=0).stop 2024-05-08T21:21:59.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.160+0000 7f6c9ee41700 1 -- 172.21.15.3:0/4166302195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c9809aca0 msgr2=0x7f6c981416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:59.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.160+0000 7f6c9ee41700 1 --2- 172.21.15.3:0/4166302195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c9809aca0 0x7f6c981416d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000b1a0 tx=0x7f6c9000b4b0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.160+0000 7f6c9ee41700 1 -- 172.21.15.3:0/4166302195 shutdown_connections 2024-05-08T21:21:59.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.160+0000 7f6c9ee41700 1 --2- 172.21.15.3:0/4166302195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c98141c10 0x7f6c9809ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.160+0000 7f6c9ee41700 1 --2- 172.21.15.3:0/4166302195 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6c7806ad60 0x7f6c7806d220 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.160+0000 7f6c9ee41700 1 --2- 172.21.15.3:0/4166302195 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9809a8c0 0x7f6c98088450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.160+0000 7f6c9ee41700 1 --2- 172.21.15.3:0/4166302195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c9809aca0 0x7f6c981416d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.160+0000 7f6c9ee41700 1 -- 172.21.15.3:0/4166302195 >> 172.21.15.3:0/4166302195 conn(0x7f6c980b1290 msgr2=0x7f6c98164cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:59.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.160+0000 7f6c9ee41700 1 -- 172.21.15.3:0/4166302195 shutdown_connections 2024-05-08T21:21:59.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.160+0000 7f6c9ee41700 1 -- 172.21.15.3:0/4166302195 wait complete. 2024-05-08T21:21:59.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.160+0000 7f1ed2344700 1 -- 172.21.15.3:0/792403197 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1eb806ac90 msgr2=0x7f1eb806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:59.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.160+0000 7f1ed2344700 1 --2- 172.21.15.3:0/792403197 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1eb806ac90 0x7f1eb806d150 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc0015d0 tx=0x7f1ebc007ad0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.160+0000 7f1ed2344700 1 -- 172.21.15.3:0/792403197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ecc0b2960 msgr2=0x7f1ecc1f7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:21:59.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.160+0000 7f1ed2344700 1 --2- 172.21.15.3:0/792403197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ecc0b2960 0x7f1ecc1f7920 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0016000 tx=0x7f1ec0004750 comp rx=0 tx=0).stop 2024-05-08T21:21:59.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.161+0000 7f1ed2344700 1 -- 172.21.15.3:0/792403197 shutdown_connections 2024-05-08T21:21:59.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.161+0000 7f1ed2344700 1 --2- 172.21.15.3:0/792403197 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ecc163440 0x7f1ecc159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.161+0000 7f1ed2344700 1 --2- 172.21.15.3:0/792403197 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1eb806ac90 0x7f1eb806d150 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.161+0000 7f1ed2344700 1 --2- 172.21.15.3:0/792403197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ecc0b2960 0x7f1ecc1f7920 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.161+0000 7f1ed2344700 1 --2- 172.21.15.3:0/792403197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ecc15c380 0x7f1ecc157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:21:59.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.161+0000 7f1ed2344700 1 -- 172.21.15.3:0/792403197 >> 172.21.15.3:0/792403197 conn(0x7f1ecc0b4690 msgr2=0x7f1ecc15ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:21:59.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.161+0000 7f1ed2344700 1 -- 172.21.15.3:0/792403197 shutdown_connections 2024-05-08T21:21:59.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:21:59.161+0000 7f1ed2344700 1 -- 172.21.15.3:0/792403197 wait complete. 2024-05-08T21:21:59.172 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2024-05-08T21:21:59.172 DEBUG:teuthology.parallel:result is None 2024-05-08T21:21:59.178 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2024-05-08T21:21:59.179 DEBUG:teuthology.parallel:result is None 2024-05-08T21:21:59.179 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2024-05-08T21:21:59.179 DEBUG:teuthology.parallel:result is None 2024-05-08T21:22:00.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-08T21:22:00.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-08T21:22:00.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-08T21:22:00.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.256+0000 7f64dee35700 1 Processor -- start 2024-05-08T21:22:00.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.256+0000 7f64dee35700 1 -- start start 2024-05-08T21:22:00.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64dee35700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64d815a770 0x7f64d80ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64dee35700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f64d80cea20 0x7f64d80cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64dee35700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f64d80ccbd0 0x7f64d80cd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64dee35700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d80caa30 con 0x7f64d80cea20 2024-05-08T21:22:00.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64dee35700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d80cf790 con 0x7f64d80ccbd0 2024-05-08T21:22:00.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64dee35700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d80cfa70 con 0x7f64d815a770 2024-05-08T21:22:00.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64d7fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f64d80cea20 0x7f64d80cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64d7fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f64d80cea20 0x7f64d80cee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52514/0 (socket says 172.21.15.3:52514) 2024-05-08T21:22:00.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64d7fff700 1 -- 172.21.15.3:0/274292247 learned_addr learned my addr 172.21.15.3:0/274292247 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:00.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64dd3d2700 1 --2- 172.21.15.3:0/274292247 >> v2:172.21.15.186:3300/0 conn(0x7f64d80ccbd0 0x7f64d80cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64dcbd1700 1 --2- 172.21.15.3:0/274292247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64d815a770 0x7f64d80ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64d7fff700 1 -- 172.21.15.3:0/274292247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64d815a770 msgr2=0x7f64d80ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:00.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64d7fff700 1 --2- 172.21.15.3:0/274292247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64d815a770 0x7f64d80ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64d7fff700 1 -- 172.21.15.3:0/274292247 >> v2:172.21.15.186:3300/0 conn(0x7f64d80ccbd0 msgr2=0x7f64d80cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:00.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64d7fff700 1 --2- 172.21.15.3:0/274292247 >> v2:172.21.15.186:3300/0 conn(0x7f64d80ccbd0 0x7f64d80cd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64d7fff700 1 -- 172.21.15.3:0/274292247 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d80cfd50 con 0x7f64d80cea20 2024-05-08T21:22:00.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.258+0000 7f64d7fff700 1 --2- 172.21.15.3:0/274292247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d80cea20 0x7f64d80cee80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f64d8072cc0 tx=0x7f64c4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72de013bcecfcd6a server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:00.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.259+0000 7f64d77fe700 1 -- 172.21.15.3:0/274292247 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4011830 con 0x7f64d80cea20 2024-05-08T21:22:00.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.259+0000 7f64d77fe700 1 -- 172.21.15.3:0/274292247 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c4011990 con 0x7f64d80cea20 2024-05-08T21:22:00.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.259+0000 7f64d77fe700 1 -- 172.21.15.3:0/274292247 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4011d20 con 0x7f64d80cea20 2024-05-08T21:22:00.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.259+0000 7f64dee35700 1 -- 172.21.15.3:0/274292247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d80cea20 msgr2=0x7f64d80cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.259+0000 7f64dee35700 1 --2- 172.21.15.3:0/274292247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d80cea20 0x7f64d80cee80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f64d8072cc0 tx=0x7f64c4009bd0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.259+0000 7f64dee35700 1 -- 172.21.15.3:0/274292247 shutdown_connections 2024-05-08T21:22:00.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.259+0000 7f64dee35700 1 --2- 172.21.15.3:0/274292247 >> v2:172.21.15.186:3300/0 conn(0x7f64d80ccbd0 0x7f64d80cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.259+0000 7f64dee35700 1 --2- 172.21.15.3:0/274292247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64d815a770 0x7f64d80ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.259+0000 7f64dee35700 1 --2- 172.21.15.3:0/274292247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d80cea20 0x7f64d80cee80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.259+0000 7f64dee35700 1 -- 172.21.15.3:0/274292247 >> 172.21.15.3:0/274292247 conn(0x7f64d80b1f80 msgr2=0x7f64d80b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:00.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.259+0000 7f64dee35700 1 -- 172.21.15.3:0/274292247 shutdown_connections 2024-05-08T21:22:00.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.260+0000 7f64dee35700 1 -- 172.21.15.3:0/274292247 wait complete. 2024-05-08T21:22:00.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.261+0000 7f64dee35700 1 Processor -- start 2024-05-08T21:22:00.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.261+0000 7f64dee35700 1 -- start start 2024-05-08T21:22:00.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64dee35700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64d80ccbd0 0x7f64d813c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64dee35700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d80cea20 0x7f64d8144d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64dee35700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d815a770 0x7f64d8145280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64dee35700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d80c5290 con 0x7f64d80cea20 2024-05-08T21:22:00.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64dee35700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d80c5530 con 0x7f64d815a770 2024-05-08T21:22:00.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64dee35700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d80c5810 con 0x7f64d80ccbd0 2024-05-08T21:22:00.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64d7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d80cea20 0x7f64d8144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64d7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d80cea20 0x7f64d8144d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52528/0 (socket says 172.21.15.3:52528) 2024-05-08T21:22:00.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64d7fff700 1 -- 172.21.15.3:0/4111205661 learned_addr learned my addr 172.21.15.3:0/4111205661 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:00.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64dcbd1700 1 --2- 172.21.15.3:0/4111205661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64d80ccbd0 0x7f64d813c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64dd3d2700 1 --2- 172.21.15.3:0/4111205661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d815a770 0x7f64d8145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64d7fff700 1 -- 172.21.15.3:0/4111205661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64d80ccbd0 msgr2=0x7f64d813c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64d7fff700 1 --2- 172.21.15.3:0/4111205661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64d80ccbd0 0x7f64d813c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64d7fff700 1 -- 172.21.15.3:0/4111205661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d815a770 msgr2=0x7f64d8145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64d7fff700 1 --2- 172.21.15.3:0/4111205661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d815a770 0x7f64d8145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64d7fff700 1 -- 172.21.15.3:0/4111205661 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c4009550 con 0x7f64d80cea20 2024-05-08T21:22:00.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.262+0000 7f64dd3d2700 1 --2- 172.21.15.3:0/4111205661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d815a770 0x7f64d8145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:00.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.263+0000 7f64d7fff700 1 --2- 172.21.15.3:0/4111205661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d80cea20 0x7f64d8144d40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009fd0 tx=0x7f64c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:00.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.263+0000 7f64d57fa700 1 -- 172.21.15.3:0/4111205661 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c40036a0 con 0x7f64d80cea20 2024-05-08T21:22:00.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.263+0000 7f64d57fa700 1 -- 172.21.15.3:0/4111205661 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c401f070 con 0x7f64d80cea20 2024-05-08T21:22:00.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.263+0000 7f64dee35700 1 -- 172.21.15.3:0/4111205661 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d80c5af0 con 0x7f64d80cea20 2024-05-08T21:22:00.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.263+0000 7f64d57fa700 1 -- 172.21.15.3:0/4111205661 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4003a40 con 0x7f64d80cea20 2024-05-08T21:22:00.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.263+0000 7f64dee35700 1 -- 172.21.15.3:0/4111205661 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d813c7b0 con 0x7f64d80cea20 2024-05-08T21:22:00.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.264+0000 7f64dee35700 1 -- 172.21.15.3:0/4111205661 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d80c6010 con 0x7f64d80cea20 2024-05-08T21:22:00.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.265+0000 7f64d57fa700 1 -- 172.21.15.3:0/4111205661 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f64c4003ba0 con 0x7f64d80cea20 2024-05-08T21:22:00.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.266+0000 7f64d57fa700 1 --2- 172.21.15.3:0/4111205661 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f64bc06ac90 0x7f64bc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.266+0000 7f64d57fa700 1 -- 172.21.15.3:0/4111205661 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f64c4021070 con 0x7f64d80cea20 2024-05-08T21:22:00.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.266+0000 7f64dcbd1700 1 --2- 172.21.15.3:0/4111205661 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f64bc06ac90 0x7f64bc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.266+0000 7f64dcbd1700 1 --2- 172.21.15.3:0/4111205661 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f64bc06ac90 0x7f64bc06d150 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009ad0 tx=0x7f64cc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:00.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.274+0000 7f64d57fa700 1 -- 172.21.15.3:0/4111205661 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f64d80c6010 con 0x7f64d80cea20 2024-05-08T21:22:00.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.291+0000 7ff0ef567700 1 Processor -- start 2024-05-08T21:22:00.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.292+0000 7ff0ef567700 1 -- start start 2024-05-08T21:22:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.293+0000 7ff0ef567700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff0e815a770 0x7ff0e80ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.293+0000 7ff0ef567700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e80cea20 0x7ff0e80cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.293+0000 7ff0ef567700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e80cf790 con 0x7ff0e80ccbd0 2024-05-08T21:22:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.293+0000 7ff0ef567700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e80caa30 con 0x7ff0e815a770 2024-05-08T21:22:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.293+0000 7ff0ef567700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e80cfa70 con 0x7ff0e80cea20 2024-05-08T21:22:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.293+0000 7ff0ed303700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff0e815a770 0x7ff0e80ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.293+0000 7ff0ed303700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff0e815a770 0x7ff0e80ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52538/0 (socket says 172.21.15.3:52538) 2024-05-08T21:22:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.293+0000 7ff0edb04700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff0e80ccbd0 0x7ff0e80cd060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:33088/0 (socket says 172.21.15.3:33088) 2024-05-08T21:22:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.293+0000 7ff0ed303700 1 -- 172.21.15.3:0/2464954711 learned_addr learned my addr 172.21.15.3:0/2464954711 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.293+0000 7ff0ecb02700 1 --2- 172.21.15.3:0/2464954711 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e80cea20 0x7ff0e80cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.293+0000 7ff0ed303700 1 -- 172.21.15.3:0/2464954711 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e80cea20 msgr2=0x7ff0e80cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.293+0000 7ff0ed303700 1 --2- 172.21.15.3:0/2464954711 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e80cea20 0x7ff0e80cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.293+0000 7ff0ed303700 1 -- 172.21.15.3:0/2464954711 >> v1:172.21.15.3:6789/0 conn(0x7ff0e80ccbd0 legacy=0x7ff0e80cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.293+0000 7ff0dffff700 1 -- 172.21.15.3:0/2464954711 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942611022 0 0) 0x7ff0e80cf790 con 0x7ff0e80ccbd0 2024-05-08T21:22:00.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.293+0000 7ff0ed303700 1 -- 172.21.15.3:0/2464954711 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e80cfd50 con 0x7ff0e815a770 2024-05-08T21:22:00.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.294+0000 7ff0ed303700 1 --2- 172.21.15.3:0/2464954711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0e815a770 0x7ff0e80ce4e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff0e40098f0 tx=0x7ff0e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf2700acd6d66e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:00.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.294+0000 7ff0dffff700 1 -- 172.21.15.3:0/2464954711 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e400f8a0 con 0x7ff0e815a770 2024-05-08T21:22:00.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.294+0000 7ff0dffff700 1 -- 172.21.15.3:0/2464954711 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e400fa00 con 0x7ff0e815a770 2024-05-08T21:22:00.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.294+0000 7ff0dffff700 1 -- 172.21.15.3:0/2464954711 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e400fd90 con 0x7ff0e815a770 2024-05-08T21:22:00.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.295+0000 7ff0ef567700 1 -- 172.21.15.3:0/2464954711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0e815a770 msgr2=0x7ff0e80ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.295+0000 7ff0ef567700 1 --2- 172.21.15.3:0/2464954711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0e815a770 0x7ff0e80ce4e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff0e40098f0 tx=0x7ff0e4009c00 comp rx=0 tx=0).stop 2024-05-08T21:22:00.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.295+0000 7ff0ef567700 1 -- 172.21.15.3:0/2464954711 shutdown_connections 2024-05-08T21:22:00.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.295+0000 7ff0ef567700 1 --2- 172.21.15.3:0/2464954711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0e815a770 0x7ff0e80ce4e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.295+0000 7ff0ef567700 1 --2- 172.21.15.3:0/2464954711 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e80cea20 0x7ff0e80cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.295+0000 7ff0ef567700 1 -- 172.21.15.3:0/2464954711 >> 172.21.15.3:0/2464954711 conn(0x7ff0e80b1f80 msgr2=0x7ff0e80b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:00.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.295+0000 7ff0ef567700 1 -- 172.21.15.3:0/2464954711 shutdown_connections 2024-05-08T21:22:00.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.295+0000 7ff0ef567700 1 -- 172.21.15.3:0/2464954711 wait complete. 2024-05-08T21:22:00.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.296+0000 7f61528ec700 1 Processor -- start 2024-05-08T21:22:00.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.296+0000 7f61528ec700 1 -- start start 2024-05-08T21:22:00.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.296+0000 7ff0ef567700 1 Processor -- start 2024-05-08T21:22:00.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.296+0000 7ff0ef567700 1 -- start start 2024-05-08T21:22:00.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7f61528ec700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f614c15a800 0x7f614c0cba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7f61528ec700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f614c0cd380 0x7f614c0ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7f61528ec700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614c15abe0 con 0x7f614c06ed30 2024-05-08T21:22:00.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7f61528ec700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f614c06e8a0 con 0x7f614c0cd380 2024-05-08T21:22:00.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7f61528ec700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614c06dc90 con 0x7f614c15a800 2024-05-08T21:22:00.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7f6150e89700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f614c0cd380 0x7f614c0ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7f614b7fe700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f614c06ed30 0x7f614c0ccc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:44388/0 (socket says 172.21.15.3:44388) 2024-05-08T21:22:00.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7f614b7fe700 1 -- 172.21.15.3:0/629246535 learned_addr learned my addr 172.21.15.3:0/629246535 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:00.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7f614bfff700 1 --2- 172.21.15.3:0/629246535 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f614c15a800 0x7f614c0cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7ff0ef567700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e80ccbd0 0x7ff0e8157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7ff0ef567700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0e80cea20 0x7ff0e80c7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7ff0ef567700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0e815a770 0x7ff0e80c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7ff0ef567700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e8141d40 con 0x7ff0e80cea20 2024-05-08T21:22:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7ff0ef567700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e81583c0 con 0x7ff0e815a770 2024-05-08T21:22:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7ff0ef567700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e81586a0 con 0x7ff0e80ccbd0 2024-05-08T21:22:00.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7ff0ecb02700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0e80cea20 0x7ff0e80c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7ff0ecb02700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0e80cea20 0x7ff0e80c7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52568/0 (socket says 172.21.15.3:52568) 2024-05-08T21:22:00.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7ff0ecb02700 1 -- 172.21.15.3:0/670192631 learned_addr learned my addr 172.21.15.3:0/670192631 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:00.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7ff0edb04700 1 --2- 172.21.15.3:0/670192631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0e815a770 0x7ff0e80c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7f614affd700 1 -- 172.21.15.3:0/629246535 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135108785 0 0) 0x7f6140001110 con 0x7f614c06ed30 2024-05-08T21:22:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.297+0000 7f614affd700 1 -- 172.21.15.3:0/629246535 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61340079c0 con 0x7f614c06ed30 2024-05-08T21:22:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7f6150e89700 1 -- 172.21.15.3:0/629246535 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f614c15a800 msgr2=0x7f614c0cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7f6150e89700 1 --2- 172.21.15.3:0/629246535 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f614c15a800 0x7f614c0cba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7f6150e89700 1 -- 172.21.15.3:0/629246535 >> v1:172.21.15.186:6789/0 conn(0x7f614c06ed30 legacy=0x7f614c0ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7f6150e89700 1 -- 172.21.15.3:0/629246535 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c06df70 con 0x7f614c0cd380 2024-05-08T21:22:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7f614bfff700 1 --2- 172.21.15.3:0/629246535 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f614c15a800 0x7f614c0cba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7ff0ecb02700 1 -- 172.21.15.3:0/670192631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e80ccbd0 msgr2=0x7ff0e8157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7ff0ecb02700 1 --2- 172.21.15.3:0/670192631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e80ccbd0 0x7ff0e8157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7ff0ecb02700 1 -- 172.21.15.3:0/670192631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0e815a770 msgr2=0x7ff0e80c7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7ff0ecb02700 1 --2- 172.21.15.3:0/670192631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0e815a770 0x7ff0e80c7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7ff0ecb02700 1 -- 172.21.15.3:0/670192631 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e4009550 con 0x7ff0e80cea20 2024-05-08T21:22:00.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7f6150e89700 1 --2- 172.21.15.3:0/629246535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614c0cd380 0x7f614c0ce230 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6144009d90 tx=0x7f614400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50b993664c1326f server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:00.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7ff0ecb02700 1 --2- 172.21.15.3:0/670192631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0e80cea20 0x7ff0e80c7420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800e9b0 tx=0x7ff0d800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:00.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7f614affd700 1 -- 172.21.15.3:0/629246535 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614400e070 con 0x7f614c0cd380 2024-05-08T21:22:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7f614affd700 1 -- 172.21.15.3:0/629246535 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614400bb20 con 0x7f614c0cd380 2024-05-08T21:22:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7f614affd700 1 -- 172.21.15.3:0/629246535 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6144012430 con 0x7f614c0cd380 2024-05-08T21:22:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7ff0ddffb700 1 -- 172.21.15.3:0/670192631 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d800c850 con 0x7ff0e80cea20 2024-05-08T21:22:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7ff0ddffb700 1 -- 172.21.15.3:0/670192631 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d800c9b0 con 0x7ff0e80cea20 2024-05-08T21:22:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7ff0ef567700 1 -- 172.21.15.3:0/670192631 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e8158980 con 0x7ff0e80cea20 2024-05-08T21:22:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7ff0ddffb700 1 -- 172.21.15.3:0/670192631 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d800cd20 con 0x7ff0e80cea20 2024-05-08T21:22:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.298+0000 7ff0ef567700 1 -- 172.21.15.3:0/670192631 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e8158f40 con 0x7ff0e80cea20 2024-05-08T21:22:00.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.299+0000 7f61528ec700 1 -- 172.21.15.3:0/629246535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614c0cd380 msgr2=0x7f614c0ce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.299+0000 7f61528ec700 1 --2- 172.21.15.3:0/629246535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614c0cd380 0x7f614c0ce230 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6144009d90 tx=0x7f614400b330 comp rx=0 tx=0).stop 2024-05-08T21:22:00.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.299+0000 7f61528ec700 1 -- 172.21.15.3:0/629246535 shutdown_connections 2024-05-08T21:22:00.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.299+0000 7f61528ec700 1 --2- 172.21.15.3:0/629246535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614c0cd380 0x7f614c0ce230 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.299+0000 7f61528ec700 1 --2- 172.21.15.3:0/629246535 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f614c15a800 0x7f614c0cba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.299+0000 7f61528ec700 1 -- 172.21.15.3:0/629246535 >> 172.21.15.3:0/629246535 conn(0x7f614c099a50 msgr2=0x7f614c0d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:00.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.299+0000 7f61528ec700 1 -- 172.21.15.3:0/629246535 shutdown_connections 2024-05-08T21:22:00.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.299+0000 7f61528ec700 1 -- 172.21.15.3:0/629246535 wait complete. 2024-05-08T21:22:00.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.300+0000 7ff0ef567700 1 -- 172.21.15.3:0/670192631 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e806f870 con 0x7ff0e80cea20 2024-05-08T21:22:00.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.301+0000 7ff0ddffb700 1 -- 172.21.15.3:0/670192631 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff0d8013470 con 0x7ff0e80cea20 2024-05-08T21:22:00.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.301+0000 7f61528ec700 1 Processor -- start 2024-05-08T21:22:00.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.301+0000 7f61528ec700 1 -- start start 2024-05-08T21:22:00.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.301+0000 7ff0ddffb700 1 --2- 172.21.15.3:0/670192631 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff0cc06ac90 0x7ff0cc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.301+0000 7ff0ddffb700 1 -- 172.21.15.3:0/670192631 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0d8089490 con 0x7ff0e80cea20 2024-05-08T21:22:00.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.301+0000 7ff0ed303700 1 --2- 172.21.15.3:0/670192631 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff0cc06ac90 0x7ff0cc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7f61528ec700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614c06ed30 0x7f614c1575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7f61528ec700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f614c0cd380 0x7f614c0c3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7f61528ec700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f614c15a800 0x7f614c0c4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7f61528ec700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c158a50 con 0x7f614c06ed30 2024-05-08T21:22:00.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7f61528ec700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c0c36c0 con 0x7f614c0cd380 2024-05-08T21:22:00.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7f61528ec700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614c157dd0 con 0x7f614c15a800 2024-05-08T21:22:00.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7f614bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614c06ed30 0x7f614c1575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7f614bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614c06ed30 0x7f614c1575a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52574/0 (socket says 172.21.15.3:52574) 2024-05-08T21:22:00.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7ff0ed303700 1 --2- 172.21.15.3:0/670192631 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff0cc06ac90 0x7ff0cc06d150 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4000c00 tx=0x7ff0e4012000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:00.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7f614bfff700 1 -- 172.21.15.3:0/2138709168 learned_addr learned my addr 172.21.15.3:0/2138709168 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:00.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7f6150e89700 1 --2- 172.21.15.3:0/2138709168 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f614c15a800 0x7f614c0c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7f614b7fe700 1 --2- 172.21.15.3:0/2138709168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f614c0cd380 0x7f614c0c3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7f614bfff700 1 -- 172.21.15.3:0/2138709168 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f614c15a800 msgr2=0x7f614c0c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7f614bfff700 1 --2- 172.21.15.3:0/2138709168 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f614c15a800 0x7f614c0c4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7f614bfff700 1 -- 172.21.15.3:0/2138709168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f614c0cd380 msgr2=0x7f614c0c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7f614bfff700 1 --2- 172.21.15.3:0/2138709168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f614c0cd380 0x7f614c0c3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.302+0000 7f614bfff700 1 -- 172.21.15.3:0/2138709168 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613c0094a0 con 0x7f614c06ed30 2024-05-08T21:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.303+0000 7f614bfff700 1 --2- 172.21.15.3:0/2138709168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614c06ed30 0x7f614c1575a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f613c00fb60 tx=0x7f613c00fe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.306+0000 7f6148ff9700 1 -- 172.21.15.3:0/2138709168 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c00c070 con 0x7f614c06ed30 2024-05-08T21:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.306+0000 7f6148ff9700 1 -- 172.21.15.3:0/2138709168 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613c00da60 con 0x7f614c06ed30 2024-05-08T21:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.306+0000 7f61528ec700 1 -- 172.21.15.3:0/2138709168 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6144009a70 con 0x7f614c06ed30 2024-05-08T21:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.306+0000 7f6148ff9700 1 -- 172.21.15.3:0/2138709168 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c00ddf0 con 0x7f614c06ed30 2024-05-08T21:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.306+0000 7f61528ec700 1 -- 172.21.15.3:0/2138709168 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614c158390 con 0x7f614c06ed30 2024-05-08T21:22:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.307+0000 7ff0ddffb700 1 -- 172.21.15.3:0/670192631 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff0d8054560 con 0x7ff0e80cea20 2024-05-08T21:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.307+0000 7f61528ec700 1 -- 172.21.15.3:0/2138709168 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614c158cc0 con 0x7f614c06ed30 2024-05-08T21:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.308+0000 7f6148ff9700 1 -- 172.21.15.3:0/2138709168 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f613c01a480 con 0x7f614c06ed30 2024-05-08T21:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.309+0000 7f6148ff9700 1 --2- 172.21.15.3:0/2138709168 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f613806ac90 0x7f613806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.309+0000 7f6148ff9700 1 -- 172.21.15.3:0/2138709168 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f613c015070 con 0x7f614c06ed30 2024-05-08T21:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.309+0000 7f614b7fe700 1 --2- 172.21.15.3:0/2138709168 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f613806ac90 0x7f613806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.309+0000 7f614b7fe700 1 --2- 172.21.15.3:0/2138709168 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f613806ac90 0x7f613806d150 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6140007fd0 tx=0x7f6140007c40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:00.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.318+0000 7f6148ff9700 1 -- 172.21.15.3:0/2138709168 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f613c0549d0 con 0x7f614c06ed30 2024-05-08T21:22:00.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.501+0000 7f64dee35700 1 -- 172.21.15.3:0/4111205661 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f64d80c6010 con 0x7f64d80cea20 2024-05-08T21:22:00.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.502+0000 7f64d57fa700 1 -- 172.21.15.3:0/4111205661 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f64d80c6010 con 0x7f64d80cea20 2024-05-08T21:22:00.504 INFO:teuthology.orchestra.run.smithi003.stdout:55834574855 2024-05-08T21:22:00.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.504+0000 7f64dee35700 1 -- 172.21.15.3:0/4111205661 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f64bc06ac90 msgr2=0x7f64bc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.504+0000 7f64dee35700 1 --2- 172.21.15.3:0/4111205661 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f64bc06ac90 0x7f64bc06d150 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009ad0 tx=0x7f64cc008040 comp rx=0 tx=0).stop 2024-05-08T21:22:00.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.504+0000 7f64dee35700 1 -- 172.21.15.3:0/4111205661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d80cea20 msgr2=0x7f64d8144d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.504+0000 7f64dee35700 1 --2- 172.21.15.3:0/4111205661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d80cea20 0x7f64d8144d40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009fd0 tx=0x7f64c4004750 comp rx=0 tx=0).stop 2024-05-08T21:22:00.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.504+0000 7f64dee35700 1 -- 172.21.15.3:0/4111205661 shutdown_connections 2024-05-08T21:22:00.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.504+0000 7f64dee35700 1 --2- 172.21.15.3:0/4111205661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64d80ccbd0 0x7f64d813c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.504+0000 7f64dee35700 1 --2- 172.21.15.3:0/4111205661 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f64bc06ac90 0x7f64bc06d150 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.504+0000 7f64dee35700 1 --2- 172.21.15.3:0/4111205661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d815a770 0x7f64d8145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.504+0000 7f64dee35700 1 --2- 172.21.15.3:0/4111205661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d80cea20 0x7f64d8144d40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.504+0000 7f64dee35700 1 -- 172.21.15.3:0/4111205661 >> 172.21.15.3:0/4111205661 conn(0x7f64d80b1f80 msgr2=0x7f64d806e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:00.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.504+0000 7f64dee35700 1 -- 172.21.15.3:0/4111205661 shutdown_connections 2024-05-08T21:22:00.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.504+0000 7f64dee35700 1 -- 172.21.15.3:0/4111205661 wait complete. 2024-05-08T21:22:00.520 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2024-05-08T21:22:00.520 DEBUG:teuthology.parallel:result is None 2024-05-08T21:22:00.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.532+0000 7ff0ef567700 1 -- 172.21.15.3:0/670192631 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ff0e8068000 con 0x7ff0e80cea20 2024-05-08T21:22:00.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.533+0000 7ff0ddffb700 1 -- 172.21.15.3:0/670192631 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff0d8057b80 con 0x7ff0e80cea20 2024-05-08T21:22:00.534 INFO:teuthology.orchestra.run.smithi003.stdout:55834574855 2024-05-08T21:22:00.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.535+0000 7ff0ef567700 1 -- 172.21.15.3:0/670192631 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff0cc06ac90 msgr2=0x7ff0cc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.535+0000 7ff0ef567700 1 --2- 172.21.15.3:0/670192631 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff0cc06ac90 0x7ff0cc06d150 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4000c00 tx=0x7ff0e4012000 comp rx=0 tx=0).stop 2024-05-08T21:22:00.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.535+0000 7ff0ef567700 1 -- 172.21.15.3:0/670192631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0e80cea20 msgr2=0x7ff0e80c7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.535+0000 7ff0ef567700 1 --2- 172.21.15.3:0/670192631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0e80cea20 0x7ff0e80c7420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800e9b0 tx=0x7ff0d800ecc0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.535+0000 7ff0ef567700 1 -- 172.21.15.3:0/670192631 shutdown_connections 2024-05-08T21:22:00.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.535+0000 7ff0ef567700 1 --2- 172.21.15.3:0/670192631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e80ccbd0 0x7ff0e8157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.535+0000 7ff0ef567700 1 --2- 172.21.15.3:0/670192631 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff0cc06ac90 0x7ff0cc06d150 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.535+0000 7ff0ef567700 1 --2- 172.21.15.3:0/670192631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0e815a770 0x7ff0e80c7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.535+0000 7ff0ef567700 1 --2- 172.21.15.3:0/670192631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0e80cea20 0x7ff0e80c7420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.535+0000 7ff0ef567700 1 -- 172.21.15.3:0/670192631 >> 172.21.15.3:0/670192631 conn(0x7ff0e80b1f80 msgr2=0x7ff0e80afdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:00.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.535+0000 7ff0ef567700 1 -- 172.21.15.3:0/670192631 shutdown_connections 2024-05-08T21:22:00.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.535+0000 7ff0ef567700 1 -- 172.21.15.3:0/670192631 wait complete. 2024-05-08T21:22:00.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.539+0000 7f61528ec700 1 -- 172.21.15.3:0/2138709168 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f614c068000 con 0x7f614c06ed30 2024-05-08T21:22:00.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.539+0000 7f6148ff9700 1 -- 172.21.15.3:0/2138709168 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f613c057ff0 con 0x7f614c06ed30 2024-05-08T21:22:00.541 INFO:teuthology.orchestra.run.smithi003.stdout:55834574857 2024-05-08T21:22:00.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.542+0000 7f61528ec700 1 -- 172.21.15.3:0/2138709168 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f613806ac90 msgr2=0x7f613806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.542+0000 7f61528ec700 1 --2- 172.21.15.3:0/2138709168 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f613806ac90 0x7f613806d150 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6140007fd0 tx=0x7f6140007c40 comp rx=0 tx=0).stop 2024-05-08T21:22:00.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.542+0000 7f61528ec700 1 -- 172.21.15.3:0/2138709168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614c06ed30 msgr2=0x7f614c1575a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.542+0000 7f61528ec700 1 --2- 172.21.15.3:0/2138709168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614c06ed30 0x7f614c1575a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f613c00fb60 tx=0x7f613c00fe70 comp rx=0 tx=0).stop 2024-05-08T21:22:00.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.542+0000 7f61528ec700 1 -- 172.21.15.3:0/2138709168 shutdown_connections 2024-05-08T21:22:00.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.542+0000 7f61528ec700 1 --2- 172.21.15.3:0/2138709168 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f614c15a800 0x7f614c0c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.542+0000 7f61528ec700 1 --2- 172.21.15.3:0/2138709168 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f613806ac90 0x7f613806d150 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.542+0000 7f61528ec700 1 --2- 172.21.15.3:0/2138709168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614c06ed30 0x7f614c1575a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.542+0000 7f61528ec700 1 --2- 172.21.15.3:0/2138709168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f614c0cd380 0x7f614c0c3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.542+0000 7f61528ec700 1 -- 172.21.15.3:0/2138709168 >> 172.21.15.3:0/2138709168 conn(0x7f614c099a50 msgr2=0x7f614c0ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:00.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.542+0000 7f61528ec700 1 -- 172.21.15.3:0/2138709168 shutdown_connections 2024-05-08T21:22:00.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.542+0000 7f61528ec700 1 -- 172.21.15.3:0/2138709168 wait complete. 2024-05-08T21:22:00.552 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2024-05-08T21:22:00.552 DEBUG:teuthology.parallel:result is None 2024-05-08T21:22:00.558 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.0 2024-05-08T21:22:00.558 DEBUG:teuthology.parallel:result is None 2024-05-08T21:22:00.558 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-08T21:22:00.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T21:22:00.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.736+0000 7f8c904d2700 1 Processor -- start 2024-05-08T21:22:00.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.736+0000 7f8c904d2700 1 -- start start 2024-05-08T21:22:00.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.737+0000 7f8c904d2700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8c880cd4e0 0x7f8c8806f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.737+0000 7f8c904d2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8c880cecf0 0x7f8c8815c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.737+0000 7f8c904d2700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8815c280 con 0x7f8c8806f130 2024-05-08T21:22:00.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.737+0000 7f8c904d2700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c8806e780 con 0x7f8c880cecf0 2024-05-08T21:22:00.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.737+0000 7f8c904d2700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c8806ea60 con 0x7f8c880cd4e0 2024-05-08T21:22:00.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.737+0000 7f8c8ea6f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8c880cecf0 0x7f8c8815c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.737+0000 7f8c8da6d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f8c8806f130 0x7f8c880ce5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:33090/0 (socket says 172.21.15.3:33090) 2024-05-08T21:22:00.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.737+0000 7f8c8da6d700 1 -- 172.21.15.3:0/878112658 learned_addr learned my addr 172.21.15.3:0/878112658 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:00.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.738+0000 7f8c8e26e700 1 --2- 172.21.15.3:0/878112658 >> v2:172.21.15.186:3300/0 conn(0x7f8c880cd4e0 0x7f8c8806f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.738+0000 7f8c8ea6f700 1 -- 172.21.15.3:0/878112658 >> v2:172.21.15.186:3300/0 conn(0x7f8c880cd4e0 msgr2=0x7f8c8806f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:00.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.738+0000 7f8c8ea6f700 1 --2- 172.21.15.3:0/878112658 >> v2:172.21.15.186:3300/0 conn(0x7f8c880cd4e0 0x7f8c8806f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.738+0000 7f8c8ea6f700 1 -- 172.21.15.3:0/878112658 >> v1:172.21.15.3:6789/0 conn(0x7f8c8806f130 legacy=0x7f8c880ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.738+0000 7f8c8d26c700 1 -- 172.21.15.3:0/878112658 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492989890 0 0) 0x7f8c8815c280 con 0x7f8c8806f130 2024-05-08T21:22:00.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.738+0000 7f8c8ea6f700 1 -- 172.21.15.3:0/878112658 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8806ed40 con 0x7f8c880cecf0 2024-05-08T21:22:00.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.738+0000 7f8c8ea6f700 1 --2- 172.21.15.3:0/878112658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c880cecf0 0x7f8c8815c710 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8c88072ef0 tx=0x7f8c8000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0392f3ab3d7320 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:00.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.738+0000 7f8c8d26c700 1 -- 172.21.15.3:0/878112658 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8000e070 con 0x7f8c880cecf0 2024-05-08T21:22:00.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.738+0000 7f8c8d26c700 1 -- 172.21.15.3:0/878112658 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8000bbd0 con 0x7f8c880cecf0 2024-05-08T21:22:00.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.738+0000 7f8c8d26c700 1 -- 172.21.15.3:0/878112658 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c80012430 con 0x7f8c880cecf0 2024-05-08T21:22:00.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.739+0000 7f8c904d2700 1 -- 172.21.15.3:0/878112658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c880cecf0 msgr2=0x7f8c8815c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.739+0000 7f8c904d2700 1 --2- 172.21.15.3:0/878112658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c880cecf0 0x7f8c8815c710 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8c88072ef0 tx=0x7f8c8000b330 comp rx=0 tx=0).stop 2024-05-08T21:22:00.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.739+0000 7f8c904d2700 1 -- 172.21.15.3:0/878112658 shutdown_connections 2024-05-08T21:22:00.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.739+0000 7f8c904d2700 1 --2- 172.21.15.3:0/878112658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c880cecf0 0x7f8c8815c710 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.739+0000 7f8c904d2700 1 --2- 172.21.15.3:0/878112658 >> v2:172.21.15.186:3300/0 conn(0x7f8c880cd4e0 0x7f8c8806f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.739+0000 7f8c904d2700 1 -- 172.21.15.3:0/878112658 >> 172.21.15.3:0/878112658 conn(0x7f8c880d0320 msgr2=0x7f8c880d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:00.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.739+0000 7f8c904d2700 1 -- 172.21.15.3:0/878112658 shutdown_connections 2024-05-08T21:22:00.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.739+0000 7f8c904d2700 1 -- 172.21.15.3:0/878112658 wait complete. 2024-05-08T21:22:00.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.740+0000 7f8c904d2700 1 Processor -- start 2024-05-08T21:22:00.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.740+0000 7f8c904d2700 1 -- start start 2024-05-08T21:22:00.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c904d2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c8806f130 0x7f8c880c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c904d2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c880cd4e0 0x7f8c880b3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c904d2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c880cecf0 0x7f8c881518b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c904d2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880c7930 con 0x7f8c880cecf0 2024-05-08T21:22:00.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c904d2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880c5c70 con 0x7f8c880cd4e0 2024-05-08T21:22:00.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c904d2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880c5360 con 0x7f8c8806f130 2024-05-08T21:22:00.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c8ea6f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c880cecf0 0x7f8c881518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c8ea6f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c880cecf0 0x7f8c881518b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52588/0 (socket says 172.21.15.3:52588) 2024-05-08T21:22:00.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c8ea6f700 1 -- 172.21.15.3:0/1202523163 learned_addr learned my addr 172.21.15.3:0/1202523163 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:00.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c8da6d700 1 --2- 172.21.15.3:0/1202523163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c880cd4e0 0x7f8c880b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c8e26e700 1 --2- 172.21.15.3:0/1202523163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c8806f130 0x7f8c880c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c8ea6f700 1 -- 172.21.15.3:0/1202523163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c8806f130 msgr2=0x7f8c880c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c8ea6f700 1 --2- 172.21.15.3:0/1202523163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c8806f130 0x7f8c880c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c8ea6f700 1 -- 172.21.15.3:0/1202523163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c880cd4e0 msgr2=0x7f8c880b3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c8ea6f700 1 --2- 172.21.15.3:0/1202523163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c880cd4e0 0x7f8c880b3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c8ea6f700 1 -- 172.21.15.3:0/1202523163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c80009a70 con 0x7f8c880cecf0 2024-05-08T21:22:00.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.741+0000 7f8c8ea6f700 1 --2- 172.21.15.3:0/1202523163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c880cecf0 0x7f8c881518b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8c80013000 tx=0x7f8c80004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:00.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.742+0000 7f8c7effd700 1 -- 172.21.15.3:0/1202523163 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c80012430 con 0x7f8c880cecf0 2024-05-08T21:22:00.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.742+0000 7f8c7effd700 1 -- 172.21.15.3:0/1202523163 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c80020070 con 0x7f8c880cecf0 2024-05-08T21:22:00.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.742+0000 7f8c7effd700 1 -- 172.21.15.3:0/1202523163 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c80004280 con 0x7f8c880cecf0 2024-05-08T21:22:00.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.742+0000 7f8c904d2700 1 -- 172.21.15.3:0/1202523163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c880b3eb0 con 0x7f8c880cecf0 2024-05-08T21:22:00.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.742+0000 7f8c904d2700 1 -- 172.21.15.3:0/1202523163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c880c5900 con 0x7f8c880cecf0 2024-05-08T21:22:00.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.743+0000 7f8c904d2700 1 -- 172.21.15.3:0/1202523163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c88068000 con 0x7f8c880cecf0 2024-05-08T21:22:00.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.743+0000 7f8c7effd700 1 -- 172.21.15.3:0/1202523163 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8c800043e0 con 0x7f8c880cecf0 2024-05-08T21:22:00.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.743+0000 7f8c7effd700 1 --2- 172.21.15.3:0/1202523163 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f8c6c06ac90 0x7f8c6c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:00.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.744+0000 7f8c7effd700 1 -- 172.21.15.3:0/1202523163 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c80022070 con 0x7f8c880cecf0 2024-05-08T21:22:00.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.744+0000 7f8c8e26e700 1 --2- 172.21.15.3:0/1202523163 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f8c6c06ac90 0x7f8c6c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:00.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.744+0000 7f8c8e26e700 1 --2- 172.21.15.3:0/1202523163 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f8c6c06ac90 0x7f8c6c06d150 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8c84009950 tx=0x7f8c84008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:00.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.748+0000 7f8c7effd700 1 -- 172.21.15.3:0/1202523163 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f8c80056410 con 0x7f8c880cecf0 2024-05-08T21:22:00.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.962+0000 7f8c904d2700 1 -- 172.21.15.3:0/1202523163 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c880d0e40 con 0x7f8c6c06ac90 2024-05-08T21:22:00.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.970+0000 7f8c7effd700 1 -- 172.21.15.3:0/1202523163 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264535 (secure 0 0 0) 0x7f8c840095e0 con 0x7f8c6c06ac90 2024-05-08T21:22:00.975 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:00.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.974+0000 7f8c904d2700 1 -- 172.21.15.3:0/1202523163 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f8c6c06ac90 msgr2=0x7f8c6c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.974+0000 7f8c904d2700 1 --2- 172.21.15.3:0/1202523163 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f8c6c06ac90 0x7f8c6c06d150 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8c84009950 tx=0x7f8c84008040 comp rx=0 tx=0).stop 2024-05-08T21:22:00.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.974+0000 7f8c904d2700 1 -- 172.21.15.3:0/1202523163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c880cecf0 msgr2=0x7f8c881518b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:00.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.974+0000 7f8c904d2700 1 --2- 172.21.15.3:0/1202523163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c880cecf0 0x7f8c881518b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8c80013000 tx=0x7f8c80004ad0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.974+0000 7f8c904d2700 1 -- 172.21.15.3:0/1202523163 shutdown_connections 2024-05-08T21:22:00.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.974+0000 7f8c904d2700 1 --2- 172.21.15.3:0/1202523163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c8806f130 0x7f8c880c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.974+0000 7f8c904d2700 1 --2- 172.21.15.3:0/1202523163 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f8c6c06ac90 0x7f8c6c06d150 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.974+0000 7f8c904d2700 1 --2- 172.21.15.3:0/1202523163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c880cecf0 0x7f8c881518b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.974+0000 7f8c904d2700 1 --2- 172.21.15.3:0/1202523163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c880cd4e0 0x7f8c880b3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:00.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.974+0000 7f8c904d2700 1 -- 172.21.15.3:0/1202523163 >> 172.21.15.3:0/1202523163 conn(0x7f8c880d0320 msgr2=0x7f8c880cc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:00.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.974+0000 7f8c904d2700 1 -- 172.21.15.3:0/1202523163 shutdown_connections 2024-05-08T21:22:00.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:00.974+0000 7f8c904d2700 1 -- 172.21.15.3:0/1202523163 wait complete. 2024-05-08T21:22:00.978 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-08T21:22:00.989 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-05-08T21:22:00.533048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218984,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755736,"statfs":{"total":773094113280,"available":772869873664,"internally_reserved":0,"allocated":2891776,"data_stored":1752644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.465691"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604264+0000","last_change":"2024-05-08T21:21:52.093899+0000","last_active":"2024-05-08T21:21:55.604264+0000","last_peered":"2024-05-08T21:21:55.604264+0000","last_clean":"2024-05-08T21:21:55.604264+0000","last_became_active":"2024-05-08T21:21:52.093609+0000","last_became_peered":"2024-05-08T21:21:52.093609+0000","last_unstale":"2024-05-08T21:21:55.604264+0000","last_undegraded":"2024-05-08T21:21:55.604264+0000","last_fullsized":"2024-05-08T21:21:55.604264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:22:03.106073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073419+0000","last_change":"2024-05-08T21:21:52.074817+0000","last_active":"2024-05-08T21:21:53.073419+0000","last_peered":"2024-05-08T21:21:53.073419+0000","last_clean":"2024-05-08T21:21:53.073419+0000","last_became_active":"2024-05-08T21:21:52.074544+0000","last_became_peered":"2024-05-08T21:21:52.074544+0000","last_unstale":"2024-05-08T21:21:53.073419+0000","last_undegraded":"2024-05-08T21:21:53.073419+0000","last_fullsized":"2024-05-08T21:21:53.073419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:22:38.788235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603137+0000","last_change":"2024-05-08T21:21:52.082843+0000","last_active":"2024-05-08T21:21:55.603137+0000","last_peered":"2024-05-08T21:21:55.603137+0000","last_clean":"2024-05-08T21:21:55.603137+0000","last_became_active":"2024-05-08T21:21:52.080398+0000","last_became_peered":"2024-05-08T21:21:52.080398+0000","last_unstale":"2024-05-08T21:21:55.603137+0000","last_undegraded":"2024-05-08T21:21:55.603137+0000","last_fullsized":"2024-05-08T21:21:55.603137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:36:48.359422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603875+0000","last_change":"2024-05-08T21:21:52.086712+0000","last_active":"2024-05-08T21:21:55.603875+0000","last_peered":"2024-05-08T21:21:55.603875+0000","last_clean":"2024-05-08T21:21:55.603875+0000","last_became_active":"2024-05-08T21:21:52.086445+0000","last_became_peered":"2024-05-08T21:21:52.086445+0000","last_unstale":"2024-05-08T21:21:55.603875+0000","last_undegraded":"2024-05-08T21:21:55.603875+0000","last_fullsized":"2024-05-08T21:21:55.603875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:22:38.788235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603913+0000","last_change":"2024-05-08T21:21:52.085542+0000","last_active":"2024-05-08T21:21:55.603913+0000","last_peered":"2024-05-08T21:21:55.603913+0000","last_clean":"2024-05-08T21:21:55.603913+0000","last_became_active":"2024-05-08T21:21:52.085253+0000","last_became_peered":"2024-05-08T21:21:52.085253+0000","last_unstale":"2024-05-08T21:21:55.603913+0000","last_undegraded":"2024-05-08T21:21:55.603913+0000","last_fullsized":"2024-05-08T21:21:55.603913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:54:03.930846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602797+0000","last_change":"2024-05-08T21:21:52.090302+0000","last_active":"2024-05-08T21:21:55.602797+0000","last_peered":"2024-05-08T21:21:55.602797+0000","last_clean":"2024-05-08T21:21:55.602797+0000","last_became_active":"2024-05-08T21:21:52.088764+0000","last_became_peered":"2024-05-08T21:21:52.088764+0000","last_unstale":"2024-05-08T21:21:55.602797+0000","last_undegraded":"2024-05-08T21:21:55.602797+0000","last_fullsized":"2024-05-08T21:21:55.602797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:09:59.496154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603367+0000","last_change":"2024-05-08T21:21:52.095147+0000","last_active":"2024-05-08T21:21:55.603367+0000","last_peered":"2024-05-08T21:21:55.603367+0000","last_clean":"2024-05-08T21:21:55.603367+0000","last_became_active":"2024-05-08T21:21:52.094954+0000","last_became_peered":"2024-05-08T21:21:52.094954+0000","last_unstale":"2024-05-08T21:21:55.603367+0000","last_undegraded":"2024-05-08T21:21:55.603367+0000","last_fullsized":"2024-05-08T21:21:55.603367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:09:59.496154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076360+0000","last_change":"2024-05-08T21:21:52.083974+0000","last_active":"2024-05-08T21:21:53.076360+0000","last_peered":"2024-05-08T21:21:53.076360+0000","last_clean":"2024-05-08T21:21:53.076360+0000","last_became_active":"2024-05-08T21:21:52.083702+0000","last_became_peered":"2024-05-08T21:21:52.083702+0000","last_unstale":"2024-05-08T21:21:53.076360+0000","last_undegraded":"2024-05-08T21:21:53.076360+0000","last_fullsized":"2024-05-08T21:21:53.076360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:54:03.930846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604376+0000","last_change":"2024-05-08T21:21:52.087758+0000","last_active":"2024-05-08T21:21:55.604376+0000","last_peered":"2024-05-08T21:21:55.604376+0000","last_clean":"2024-05-08T21:21:55.604376+0000","last_became_active":"2024-05-08T21:21:52.087489+0000","last_became_peered":"2024-05-08T21:21:52.087489+0000","last_unstale":"2024-05-08T21:21:55.604376+0000","last_undegraded":"2024-05-08T21:21:55.604376+0000","last_fullsized":"2024-05-08T21:21:55.604376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:08.050201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603328+0000","last_change":"2024-05-08T21:21:52.092265+0000","last_active":"2024-05-08T21:21:55.603328+0000","last_peered":"2024-05-08T21:21:55.603328+0000","last_clean":"2024-05-08T21:21:55.603328+0000","last_became_active":"2024-05-08T21:21:52.091813+0000","last_became_peered":"2024-05-08T21:21:52.091813+0000","last_unstale":"2024-05-08T21:21:55.603328+0000","last_undegraded":"2024-05-08T21:21:55.603328+0000","last_fullsized":"2024-05-08T21:21:55.603328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:04:56.104718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602830+0000","last_change":"2024-05-08T21:21:52.087586+0000","last_active":"2024-05-08T21:21:55.602830+0000","last_peered":"2024-05-08T21:21:55.602830+0000","last_clean":"2024-05-08T21:21:55.602830+0000","last_became_active":"2024-05-08T21:21:52.087302+0000","last_became_peered":"2024-05-08T21:21:52.087302+0000","last_unstale":"2024-05-08T21:21:55.602830+0000","last_undegraded":"2024-05-08T21:21:55.602830+0000","last_fullsized":"2024-05-08T21:21:55.602830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:41:45.740288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076636+0000","last_change":"2024-05-08T21:21:52.094218+0000","last_active":"2024-05-08T21:21:53.076636+0000","last_peered":"2024-05-08T21:21:53.076636+0000","last_clean":"2024-05-08T21:21:53.076636+0000","last_became_active":"2024-05-08T21:21:52.093855+0000","last_became_peered":"2024-05-08T21:21:52.093855+0000","last_unstale":"2024-05-08T21:21:53.076636+0000","last_undegraded":"2024-05-08T21:21:53.076636+0000","last_fullsized":"2024-05-08T21:21:53.076636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:18:36.173550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076649+0000","last_change":"2024-05-08T21:21:52.093685+0000","last_active":"2024-05-08T21:21:53.076649+0000","last_peered":"2024-05-08T21:21:53.076649+0000","last_clean":"2024-05-08T21:21:53.076649+0000","last_became_active":"2024-05-08T21:21:52.093380+0000","last_became_peered":"2024-05-08T21:21:52.093380+0000","last_unstale":"2024-05-08T21:21:53.076649+0000","last_undegraded":"2024-05-08T21:21:53.076649+0000","last_fullsized":"2024-05-08T21:21:53.076649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:39:11.021535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603501+0000","last_change":"2024-05-08T21:21:52.077438+0000","last_active":"2024-05-08T21:21:55.603501+0000","last_peered":"2024-05-08T21:21:55.603501+0000","last_clean":"2024-05-08T21:21:55.603501+0000","last_became_active":"2024-05-08T21:21:52.077129+0000","last_became_peered":"2024-05-08T21:21:52.077129+0000","last_unstale":"2024-05-08T21:21:55.603501+0000","last_undegraded":"2024-05-08T21:21:55.603501+0000","last_fullsized":"2024-05-08T21:21:55.603501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:28:07.575847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073038+0000","last_change":"2024-05-08T21:21:52.096939+0000","last_active":"2024-05-08T21:21:53.073038+0000","last_peered":"2024-05-08T21:21:53.073038+0000","last_clean":"2024-05-08T21:21:53.073038+0000","last_became_active":"2024-05-08T21:21:52.096629+0000","last_became_peered":"2024-05-08T21:21:52.096629+0000","last_unstale":"2024-05-08T21:21:53.073038+0000","last_undegraded":"2024-05-08T21:21:53.073038+0000","last_fullsized":"2024-05-08T21:21:53.073038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:04:56.104718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603067+0000","last_change":"2024-05-08T21:21:52.088031+0000","last_active":"2024-05-08T21:21:55.603067+0000","last_peered":"2024-05-08T21:21:55.603067+0000","last_clean":"2024-05-08T21:21:55.603067+0000","last_became_active":"2024-05-08T21:21:52.087748+0000","last_became_peered":"2024-05-08T21:21:52.087748+0000","last_unstale":"2024-05-08T21:21:55.603067+0000","last_undegraded":"2024-05-08T21:21:55.603067+0000","last_fullsized":"2024-05-08T21:21:55.603067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:46:26.988461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604380+0000","last_change":"2024-05-08T21:21:52.096438+0000","last_active":"2024-05-08T21:21:55.604380+0000","last_peered":"2024-05-08T21:21:55.604380+0000","last_clean":"2024-05-08T21:21:55.604380+0000","last_became_active":"2024-05-08T21:21:52.095936+0000","last_became_peered":"2024-05-08T21:21:52.095936+0000","last_unstale":"2024-05-08T21:21:55.604380+0000","last_undegraded":"2024-05-08T21:21:55.604380+0000","last_fullsized":"2024-05-08T21:21:55.604380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:38:46.924064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603006+0000","last_change":"2024-05-08T21:21:52.095114+0000","last_active":"2024-05-08T21:21:55.603006+0000","last_peered":"2024-05-08T21:21:55.603006+0000","last_clean":"2024-05-08T21:21:55.603006+0000","last_became_active":"2024-05-08T21:21:52.094795+0000","last_became_peered":"2024-05-08T21:21:52.094795+0000","last_unstale":"2024-05-08T21:21:55.603006+0000","last_undegraded":"2024-05-08T21:21:55.603006+0000","last_fullsized":"2024-05-08T21:21:55.603006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:14:37.380612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603538+0000","last_change":"2024-05-08T21:21:52.084672+0000","last_active":"2024-05-08T21:21:55.603538+0000","last_peered":"2024-05-08T21:21:55.603538+0000","last_clean":"2024-05-08T21:21:55.603538+0000","last_became_active":"2024-05-08T21:21:52.084373+0000","last_became_peered":"2024-05-08T21:21:52.084373+0000","last_unstale":"2024-05-08T21:21:55.603538+0000","last_undegraded":"2024-05-08T21:21:55.603538+0000","last_fullsized":"2024-05-08T21:21:55.603538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:10:50.287200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.095184+0000","last_change":"2024-05-08T21:21:51.614469+0000","last_active":"2024-05-08T21:21:52.095184+0000","last_peered":"2024-05-08T21:21:52.095184+0000","last_clean":"2024-05-08T21:21:52.095184+0000","last_became_active":"2024-05-08T21:21:51.614173+0000","last_became_peered":"2024-05-08T21:21:51.614173+0000","last_unstale":"2024-05-08T21:21:52.095184+0000","last_undegraded":"2024-05-08T21:21:52.095184+0000","last_fullsized":"2024-05-08T21:21:52.095184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:00:55.530393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076753+0000","last_change":"2024-05-08T21:21:52.095181+0000","last_active":"2024-05-08T21:21:53.076753+0000","last_peered":"2024-05-08T21:21:53.076753+0000","last_clean":"2024-05-08T21:21:53.076753+0000","last_became_active":"2024-05-08T21:21:52.095007+0000","last_became_peered":"2024-05-08T21:21:52.095007+0000","last_unstale":"2024-05-08T21:21:53.076753+0000","last_undegraded":"2024-05-08T21:21:53.076753+0000","last_fullsized":"2024-05-08T21:21:53.076753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:28:07.575847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.091195+0000","last_change":"2024-05-08T21:21:51.615481+0000","last_active":"2024-05-08T21:21:52.091195+0000","last_peered":"2024-05-08T21:21:52.091195+0000","last_clean":"2024-05-08T21:21:52.091195+0000","last_became_active":"2024-05-08T21:21:51.614841+0000","last_became_peered":"2024-05-08T21:21:51.614841+0000","last_unstale":"2024-05-08T21:21:52.091195+0000","last_undegraded":"2024-05-08T21:21:52.091195+0000","last_fullsized":"2024-05-08T21:21:52.091195+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:10:09.131458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603223+0000","last_change":"2024-05-08T21:21:52.083798+0000","last_active":"2024-05-08T21:21:55.603223+0000","last_peered":"2024-05-08T21:21:55.603223+0000","last_clean":"2024-05-08T21:21:55.603223+0000","last_became_active":"2024-05-08T21:21:52.083361+0000","last_became_peered":"2024-05-08T21:21:52.083361+0000","last_unstale":"2024-05-08T21:21:55.603223+0000","last_undegraded":"2024-05-08T21:21:55.603223+0000","last_fullsized":"2024-05-08T21:21:55.603223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:38.782318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.094224+0000","last_change":"2024-05-08T21:21:51.610838+0000","last_active":"2024-05-08T21:21:52.094224+0000","last_peered":"2024-05-08T21:21:52.094224+0000","last_clean":"2024-05-08T21:21:52.094224+0000","last_became_active":"2024-05-08T21:21:51.610168+0000","last_became_peered":"2024-05-08T21:21:51.610168+0000","last_unstale":"2024-05-08T21:21:52.094224+0000","last_undegraded":"2024-05-08T21:21:52.094224+0000","last_fullsized":"2024-05-08T21:21:52.094224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:13.563899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602928+0000","last_change":"2024-05-08T21:21:52.101000+0000","last_active":"2024-05-08T21:21:55.602928+0000","last_peered":"2024-05-08T21:21:55.602928+0000","last_clean":"2024-05-08T21:21:55.602928+0000","last_became_active":"2024-05-08T21:21:52.100589+0000","last_became_peered":"2024-05-08T21:21:52.100589+0000","last_unstale":"2024-05-08T21:21:55.602928+0000","last_undegraded":"2024-05-08T21:21:55.602928+0000","last_fullsized":"2024-05-08T21:21:55.602928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:22:03.106073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.094794+0000","last_change":"2024-05-08T21:21:51.617528+0000","last_active":"2024-05-08T21:21:52.094794+0000","last_peered":"2024-05-08T21:21:52.094794+0000","last_clean":"2024-05-08T21:21:52.094794+0000","last_became_active":"2024-05-08T21:21:51.617225+0000","last_became_peered":"2024-05-08T21:21:51.617225+0000","last_unstale":"2024-05-08T21:21:52.094794+0000","last_undegraded":"2024-05-08T21:21:52.094794+0000","last_fullsized":"2024-05-08T21:21:52.094794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:05:44.822440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604076+0000","last_change":"2024-05-08T21:21:52.085423+0000","last_active":"2024-05-08T21:21:55.604076+0000","last_peered":"2024-05-08T21:21:55.604076+0000","last_clean":"2024-05-08T21:21:55.604076+0000","last_became_active":"2024-05-08T21:21:52.085144+0000","last_became_peered":"2024-05-08T21:21:52.085144+0000","last_unstale":"2024-05-08T21:21:55.604076+0000","last_undegraded":"2024-05-08T21:21:55.604076+0000","last_fullsized":"2024-05-08T21:21:55.604076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:54:03.930846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.092454+0000","last_change":"2024-05-08T21:21:51.611668+0000","last_active":"2024-05-08T21:21:52.092454+0000","last_peered":"2024-05-08T21:21:52.092454+0000","last_clean":"2024-05-08T21:21:52.092454+0000","last_became_active":"2024-05-08T21:21:51.611363+0000","last_became_peered":"2024-05-08T21:21:51.611363+0000","last_unstale":"2024-05-08T21:21:52.092454+0000","last_undegraded":"2024-05-08T21:21:52.092454+0000","last_fullsized":"2024-05-08T21:21:52.092454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:05.185930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076192+0000","last_change":"2024-05-08T21:21:52.092056+0000","last_active":"2024-05-08T21:21:53.076192+0000","last_peered":"2024-05-08T21:21:53.076192+0000","last_clean":"2024-05-08T21:21:53.076192+0000","last_became_active":"2024-05-08T21:21:52.086766+0000","last_became_peered":"2024-05-08T21:21:52.086766+0000","last_unstale":"2024-05-08T21:21:53.076192+0000","last_undegraded":"2024-05-08T21:21:53.076192+0000","last_fullsized":"2024-05-08T21:21:53.076192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:05:45.827530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.101711+0000","last_change":"2024-05-08T21:21:51.609576+0000","last_active":"2024-05-08T21:21:52.101711+0000","last_peered":"2024-05-08T21:21:52.101711+0000","last_clean":"2024-05-08T21:21:52.101711+0000","last_became_active":"2024-05-08T21:21:51.609004+0000","last_became_peered":"2024-05-08T21:21:51.609004+0000","last_unstale":"2024-05-08T21:21:52.101711+0000","last_undegraded":"2024-05-08T21:21:52.101711+0000","last_fullsized":"2024-05-08T21:21:52.101711+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:38:17.091010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603223+0000","last_change":"2024-05-08T21:21:52.093264+0000","last_active":"2024-05-08T21:21:55.603223+0000","last_peered":"2024-05-08T21:21:55.603223+0000","last_clean":"2024-05-08T21:21:55.603223+0000","last_became_active":"2024-05-08T21:21:52.092898+0000","last_became_peered":"2024-05-08T21:21:52.092898+0000","last_unstale":"2024-05-08T21:21:55.603223+0000","last_undegraded":"2024-05-08T21:21:55.603223+0000","last_fullsized":"2024-05-08T21:21:55.603223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:41:45.740288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073983+0000","last_change":"2024-05-08T21:21:51.609633+0000","last_active":"2024-05-08T21:21:52.073983+0000","last_peered":"2024-05-08T21:21:52.073983+0000","last_clean":"2024-05-08T21:21:52.073983+0000","last_became_active":"2024-05-08T21:21:51.609304+0000","last_became_peered":"2024-05-08T21:21:51.609304+0000","last_unstale":"2024-05-08T21:21:52.073983+0000","last_undegraded":"2024-05-08T21:21:52.073983+0000","last_fullsized":"2024-05-08T21:21:52.073983+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:20:59.795219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076482+0000","last_change":"2024-05-08T21:21:52.078368+0000","last_active":"2024-05-08T21:21:53.076482+0000","last_peered":"2024-05-08T21:21:53.076482+0000","last_clean":"2024-05-08T21:21:53.076482+0000","last_became_active":"2024-05-08T21:21:52.078095+0000","last_became_peered":"2024-05-08T21:21:52.078095+0000","last_unstale":"2024-05-08T21:21:53.076482+0000","last_undegraded":"2024-05-08T21:21:53.076482+0000","last_fullsized":"2024-05-08T21:21:53.076482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:38.782318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073269+0000","last_change":"2024-05-08T21:21:51.612841+0000","last_active":"2024-05-08T21:21:52.073269+0000","last_peered":"2024-05-08T21:21:52.073269+0000","last_clean":"2024-05-08T21:21:52.073269+0000","last_became_active":"2024-05-08T21:21:51.612563+0000","last_became_peered":"2024-05-08T21:21:51.612563+0000","last_unstale":"2024-05-08T21:21:52.073269+0000","last_undegraded":"2024-05-08T21:21:52.073269+0000","last_fullsized":"2024-05-08T21:21:52.073269+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:41.009726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073158+0000","last_change":"2024-05-08T21:21:52.088955+0000","last_active":"2024-05-08T21:21:53.073158+0000","last_peered":"2024-05-08T21:21:53.073158+0000","last_clean":"2024-05-08T21:21:53.073158+0000","last_became_active":"2024-05-08T21:21:52.088686+0000","last_became_peered":"2024-05-08T21:21:52.088686+0000","last_unstale":"2024-05-08T21:21:53.073158+0000","last_undegraded":"2024-05-08T21:21:53.073158+0000","last_fullsized":"2024-05-08T21:21:53.073158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:38.782318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.075045+0000","last_change":"2024-05-08T21:21:51.620459+0000","last_active":"2024-05-08T21:21:52.075045+0000","last_peered":"2024-05-08T21:21:52.075045+0000","last_clean":"2024-05-08T21:21:52.075045+0000","last_became_active":"2024-05-08T21:21:51.620147+0000","last_became_peered":"2024-05-08T21:21:51.620147+0000","last_unstale":"2024-05-08T21:21:52.075045+0000","last_undegraded":"2024-05-08T21:21:52.075045+0000","last_fullsized":"2024-05-08T21:21:52.075045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:13.563899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604099+0000","last_change":"2024-05-08T21:21:52.096162+0000","last_active":"2024-05-08T21:21:55.604099+0000","last_peered":"2024-05-08T21:21:55.604099+0000","last_clean":"2024-05-08T21:21:55.604099+0000","last_became_active":"2024-05-08T21:21:52.095548+0000","last_became_peered":"2024-05-08T21:21:52.095548+0000","last_unstale":"2024-05-08T21:21:55.604099+0000","last_undegraded":"2024-05-08T21:21:55.604099+0000","last_fullsized":"2024-05-08T21:21:55.604099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:33:44.560880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073170+0000","last_change":"2024-05-08T21:21:51.617174+0000","last_active":"2024-05-08T21:21:52.073170+0000","last_peered":"2024-05-08T21:21:52.073170+0000","last_clean":"2024-05-08T21:21:52.073170+0000","last_became_active":"2024-05-08T21:21:51.616894+0000","last_became_peered":"2024-05-08T21:21:51.616894+0000","last_unstale":"2024-05-08T21:21:52.073170+0000","last_undegraded":"2024-05-08T21:21:52.073170+0000","last_fullsized":"2024-05-08T21:21:52.073170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:38:17.091010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603455+0000","last_change":"2024-05-08T21:21:52.080625+0000","last_active":"2024-05-08T21:21:55.603455+0000","last_peered":"2024-05-08T21:21:55.603455+0000","last_clean":"2024-05-08T21:21:55.603455+0000","last_became_active":"2024-05-08T21:21:52.080347+0000","last_became_peered":"2024-05-08T21:21:52.080347+0000","last_unstale":"2024-05-08T21:21:55.603455+0000","last_undegraded":"2024-05-08T21:21:55.603455+0000","last_fullsized":"2024-05-08T21:21:55.603455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:04:56.104718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.087805+0000","last_change":"2024-05-08T21:21:51.617911+0000","last_active":"2024-05-08T21:21:52.087805+0000","last_peered":"2024-05-08T21:21:52.087805+0000","last_clean":"2024-05-08T21:21:52.087805+0000","last_became_active":"2024-05-08T21:21:51.617651+0000","last_became_peered":"2024-05-08T21:21:51.617651+0000","last_unstale":"2024-05-08T21:21:52.087805+0000","last_undegraded":"2024-05-08T21:21:52.087805+0000","last_fullsized":"2024-05-08T21:21:52.087805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:37:21.706277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076579+0000","last_change":"2024-05-08T21:21:52.090657+0000","last_active":"2024-05-08T21:21:53.076579+0000","last_peered":"2024-05-08T21:21:53.076579+0000","last_clean":"2024-05-08T21:21:53.076579+0000","last_became_active":"2024-05-08T21:21:52.090381+0000","last_became_peered":"2024-05-08T21:21:52.090381+0000","last_unstale":"2024-05-08T21:21:53.076579+0000","last_undegraded":"2024-05-08T21:21:53.076579+0000","last_fullsized":"2024-05-08T21:21:53.076579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:37:22.711367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.105960+0000","last_change":"2024-05-08T21:21:51.613700+0000","last_active":"2024-05-08T21:21:52.105960+0000","last_peered":"2024-05-08T21:21:52.105960+0000","last_clean":"2024-05-08T21:21:52.105960+0000","last_became_active":"2024-05-08T21:21:51.613457+0000","last_became_peered":"2024-05-08T21:21:51.613457+0000","last_unstale":"2024-05-08T21:21:52.105960+0000","last_undegraded":"2024-05-08T21:21:52.105960+0000","last_fullsized":"2024-05-08T21:21:52.105960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:00:55.530393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603663+0000","last_change":"2024-05-08T21:21:52.093097+0000","last_active":"2024-05-08T21:21:55.603663+0000","last_peered":"2024-05-08T21:21:55.603663+0000","last_clean":"2024-05-08T21:21:55.603663+0000","last_became_active":"2024-05-08T21:21:52.092769+0000","last_became_peered":"2024-05-08T21:21:52.092769+0000","last_unstale":"2024-05-08T21:21:55.603663+0000","last_undegraded":"2024-05-08T21:21:55.603663+0000","last_fullsized":"2024-05-08T21:21:55.603663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:38:46.924064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.100944+0000","last_change":"2024-05-08T21:21:51.614173+0000","last_active":"2024-05-08T21:21:52.100944+0000","last_peered":"2024-05-08T21:21:52.100944+0000","last_clean":"2024-05-08T21:21:52.100944+0000","last_became_active":"2024-05-08T21:21:51.613664+0000","last_became_peered":"2024-05-08T21:21:51.613664+0000","last_unstale":"2024-05-08T21:21:52.100944+0000","last_undegraded":"2024-05-08T21:21:52.100944+0000","last_fullsized":"2024-05-08T21:21:52.100944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:20:59.795219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602872+0000","last_change":"2024-05-08T21:21:52.081761+0000","last_active":"2024-05-08T21:21:55.602872+0000","last_peered":"2024-05-08T21:21:55.602872+0000","last_clean":"2024-05-08T21:21:55.602872+0000","last_became_active":"2024-05-08T21:21:52.081477+0000","last_became_peered":"2024-05-08T21:21:52.081477+0000","last_unstale":"2024-05-08T21:21:55.602872+0000","last_undegraded":"2024-05-08T21:21:55.602872+0000","last_fullsized":"2024-05-08T21:21:55.602872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:39:11.021535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.075759+0000","last_change":"2024-05-08T21:21:51.616956+0000","last_active":"2024-05-08T21:21:52.075759+0000","last_peered":"2024-05-08T21:21:52.075759+0000","last_clean":"2024-05-08T21:21:52.075759+0000","last_became_active":"2024-05-08T21:21:51.616670+0000","last_became_peered":"2024-05-08T21:21:51.616670+0000","last_unstale":"2024-05-08T21:21:52.075759+0000","last_undegraded":"2024-05-08T21:21:52.075759+0000","last_fullsized":"2024-05-08T21:21:52.075759+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:45:34.421641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603797+0000","last_change":"2024-05-08T21:21:52.100972+0000","last_active":"2024-05-08T21:21:55.603797+0000","last_peered":"2024-05-08T21:21:55.603797+0000","last_clean":"2024-05-08T21:21:55.603797+0000","last_became_active":"2024-05-08T21:21:52.100711+0000","last_became_peered":"2024-05-08T21:21:52.100711+0000","last_unstale":"2024-05-08T21:21:55.603797+0000","last_undegraded":"2024-05-08T21:21:55.603797+0000","last_fullsized":"2024-05-08T21:21:55.603797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:22:38.788235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073127+0000","last_change":"2024-05-08T21:21:51.609380+0000","last_active":"2024-05-08T21:21:52.073127+0000","last_peered":"2024-05-08T21:21:52.073127+0000","last_clean":"2024-05-08T21:21:52.073127+0000","last_became_active":"2024-05-08T21:21:51.608973+0000","last_became_peered":"2024-05-08T21:21:51.608973+0000","last_unstale":"2024-05-08T21:21:52.073127+0000","last_undegraded":"2024-05-08T21:21:52.073127+0000","last_fullsized":"2024-05-08T21:21:52.073127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:45:34.421641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602586+0000","last_change":"2024-05-08T21:21:52.088958+0000","last_active":"2024-05-08T21:21:55.602586+0000","last_peered":"2024-05-08T21:21:55.602586+0000","last_clean":"2024-05-08T21:21:55.602586+0000","last_became_active":"2024-05-08T21:21:52.088270+0000","last_became_peered":"2024-05-08T21:21:52.088270+0000","last_unstale":"2024-05-08T21:21:55.602586+0000","last_undegraded":"2024-05-08T21:21:55.602586+0000","last_fullsized":"2024-05-08T21:21:55.602586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:28:07.575847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.090786+0000","last_change":"2024-05-08T21:21:51.614824+0000","last_active":"2024-05-08T21:21:52.090786+0000","last_peered":"2024-05-08T21:21:52.090786+0000","last_clean":"2024-05-08T21:21:52.090786+0000","last_became_active":"2024-05-08T21:21:51.614526+0000","last_became_peered":"2024-05-08T21:21:51.614526+0000","last_unstale":"2024-05-08T21:21:52.090786+0000","last_undegraded":"2024-05-08T21:21:52.090786+0000","last_fullsized":"2024-05-08T21:21:52.090786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:41.009726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603517+0000","last_change":"2024-05-08T21:21:52.097367+0000","last_active":"2024-05-08T21:21:55.603517+0000","last_peered":"2024-05-08T21:21:55.603517+0000","last_clean":"2024-05-08T21:21:55.603517+0000","last_became_active":"2024-05-08T21:21:52.097047+0000","last_became_peered":"2024-05-08T21:21:52.097047+0000","last_unstale":"2024-05-08T21:21:55.603517+0000","last_undegraded":"2024-05-08T21:21:55.603517+0000","last_fullsized":"2024-05-08T21:21:55.603517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:46:26.988461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.100630+0000","last_change":"2024-05-08T21:21:51.615362+0000","last_active":"2024-05-08T21:21:52.100630+0000","last_peered":"2024-05-08T21:21:52.100630+0000","last_clean":"2024-05-08T21:21:52.100630+0000","last_became_active":"2024-05-08T21:21:51.615119+0000","last_became_peered":"2024-05-08T21:21:51.615119+0000","last_unstale":"2024-05-08T21:21:52.100630+0000","last_undegraded":"2024-05-08T21:21:52.100630+0000","last_fullsized":"2024-05-08T21:21:52.100630+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:10:09.131458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603677+0000","last_change":"2024-05-08T21:21:52.086491+0000","last_active":"2024-05-08T21:21:55.603677+0000","last_peered":"2024-05-08T21:21:55.603677+0000","last_clean":"2024-05-08T21:21:55.603677+0000","last_became_active":"2024-05-08T21:21:52.086134+0000","last_became_peered":"2024-05-08T21:21:52.086134+0000","last_unstale":"2024-05-08T21:21:55.603677+0000","last_undegraded":"2024-05-08T21:21:55.603677+0000","last_fullsized":"2024-05-08T21:21:55.603677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:41.471400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.070417+0000","last_change":"2024-05-08T21:21:51.617080+0000","last_active":"2024-05-08T21:21:52.070417+0000","last_peered":"2024-05-08T21:21:52.070417+0000","last_clean":"2024-05-08T21:21:52.070417+0000","last_became_active":"2024-05-08T21:21:51.615569+0000","last_became_peered":"2024-05-08T21:21:51.615569+0000","last_unstale":"2024-05-08T21:21:52.070417+0000","last_undegraded":"2024-05-08T21:21:52.070417+0000","last_fullsized":"2024-05-08T21:21:52.070417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:44:53.582574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604152+0000","last_change":"2024-05-08T21:21:52.089644+0000","last_active":"2024-05-08T21:21:55.604152+0000","last_peered":"2024-05-08T21:21:55.604152+0000","last_clean":"2024-05-08T21:21:55.604152+0000","last_became_active":"2024-05-08T21:21:52.089161+0000","last_became_peered":"2024-05-08T21:21:52.089161+0000","last_unstale":"2024-05-08T21:21:55.604152+0000","last_undegraded":"2024-05-08T21:21:55.604152+0000","last_fullsized":"2024-05-08T21:21:55.604152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:28:07.575847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.075661+0000","last_change":"2024-05-08T21:21:51.618799+0000","last_active":"2024-05-08T21:21:52.075661+0000","last_peered":"2024-05-08T21:21:52.075661+0000","last_clean":"2024-05-08T21:21:52.075661+0000","last_became_active":"2024-05-08T21:21:51.618584+0000","last_became_peered":"2024-05-08T21:21:51.618584+0000","last_unstale":"2024-05-08T21:21:52.075661+0000","last_undegraded":"2024-05-08T21:21:52.075661+0000","last_fullsized":"2024-05-08T21:21:52.075661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:44:53.582574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076748+0000","last_change":"2024-05-08T21:21:52.090935+0000","last_active":"2024-05-08T21:21:53.076748+0000","last_peered":"2024-05-08T21:21:53.076748+0000","last_clean":"2024-05-08T21:21:53.076748+0000","last_became_active":"2024-05-08T21:21:52.090545+0000","last_became_peered":"2024-05-08T21:21:52.090545+0000","last_unstale":"2024-05-08T21:21:53.076748+0000","last_undegraded":"2024-05-08T21:21:53.076748+0000","last_fullsized":"2024-05-08T21:21:53.076748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:14:37.380612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073078+0000","last_change":"2024-05-08T21:21:52.079445+0000","last_active":"2024-05-08T21:21:53.073078+0000","last_peered":"2024-05-08T21:21:53.073078+0000","last_clean":"2024-05-08T21:21:53.073078+0000","last_became_active":"2024-05-08T21:21:52.079055+0000","last_became_peered":"2024-05-08T21:21:52.079055+0000","last_unstale":"2024-05-08T21:21:53.073078+0000","last_undegraded":"2024-05-08T21:21:53.073078+0000","last_fullsized":"2024-05-08T21:21:53.073078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:28:07.575847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073936+0000","last_change":"2024-05-08T21:21:51.616872+0000","last_active":"2024-05-08T21:21:52.073936+0000","last_peered":"2024-05-08T21:21:52.073936+0000","last_clean":"2024-05-08T21:21:52.073936+0000","last_became_active":"2024-05-08T21:21:51.616576+0000","last_became_peered":"2024-05-08T21:21:51.616576+0000","last_unstale":"2024-05-08T21:21:52.073936+0000","last_undegraded":"2024-05-08T21:21:52.073936+0000","last_fullsized":"2024-05-08T21:21:52.073936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:38:17.091010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076646+0000","last_change":"2024-05-08T21:21:52.079441+0000","last_active":"2024-05-08T21:21:53.076646+0000","last_peered":"2024-05-08T21:21:53.076646+0000","last_clean":"2024-05-08T21:21:53.076646+0000","last_became_active":"2024-05-08T21:21:52.079170+0000","last_became_peered":"2024-05-08T21:21:52.079170+0000","last_unstale":"2024-05-08T21:21:53.076646+0000","last_undegraded":"2024-05-08T21:21:53.076646+0000","last_fullsized":"2024-05-08T21:21:53.076646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:41.471400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.072713+0000","last_change":"2024-05-08T21:21:51.613447+0000","last_active":"2024-05-08T21:21:52.072713+0000","last_peered":"2024-05-08T21:21:52.072713+0000","last_clean":"2024-05-08T21:21:52.072713+0000","last_became_active":"2024-05-08T21:21:51.613150+0000","last_became_peered":"2024-05-08T21:21:51.613150+0000","last_unstale":"2024-05-08T21:21:52.072713+0000","last_undegraded":"2024-05-08T21:21:52.072713+0000","last_fullsized":"2024-05-08T21:21:52.072713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:13.563899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073528+0000","last_change":"2024-05-08T21:21:42.333097+0000","last_active":"2024-05-08T21:21:52.073528+0000","last_peered":"2024-05-08T21:21:52.073528+0000","last_clean":"2024-05-08T21:21:52.073528+0000","last_became_active":"2024-05-08T21:21:42.332033+0000","last_became_peered":"2024-05-08T21:21:42.332033+0000","last_unstale":"2024-05-08T21:21:52.073528+0000","last_undegraded":"2024-05-08T21:21:52.073528+0000","last_fullsized":"2024-05-08T21:21:52.073528+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_clean_scrub_stamp":"2024-05-08T21:21:40.862622+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:39:24.772551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602793+0000","last_change":"2024-05-08T21:21:52.096265+0000","last_active":"2024-05-08T21:21:55.602793+0000","last_peered":"2024-05-08T21:21:55.602793+0000","last_clean":"2024-05-08T21:21:55.602793+0000","last_became_active":"2024-05-08T21:21:52.095929+0000","last_became_peered":"2024-05-08T21:21:52.095929+0000","last_unstale":"2024-05-08T21:21:55.602793+0000","last_undegraded":"2024-05-08T21:21:55.602793+0000","last_fullsized":"2024-05-08T21:21:55.602793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:42.014816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073633+0000","last_change":"2024-05-08T21:21:51.614632+0000","last_active":"2024-05-08T21:21:52.073633+0000","last_peered":"2024-05-08T21:21:52.073633+0000","last_clean":"2024-05-08T21:21:52.073633+0000","last_became_active":"2024-05-08T21:21:51.614368+0000","last_became_peered":"2024-05-08T21:21:51.614368+0000","last_unstale":"2024-05-08T21:21:52.073633+0000","last_undegraded":"2024-05-08T21:21:52.073633+0000","last_fullsized":"2024-05-08T21:21:52.073633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:14:36.375522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.075513+0000","last_change":"2024-05-08T21:21:51.615266+0000","last_active":"2024-05-08T21:21:52.075513+0000","last_peered":"2024-05-08T21:21:52.075513+0000","last_clean":"2024-05-08T21:21:52.075513+0000","last_became_active":"2024-05-08T21:21:51.614949+0000","last_became_peered":"2024-05-08T21:21:51.614949+0000","last_unstale":"2024-05-08T21:21:52.075513+0000","last_undegraded":"2024-05-08T21:21:52.075513+0000","last_fullsized":"2024-05-08T21:21:52.075513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:44:53.582574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073133+0000","last_change":"2024-05-08T21:21:52.079810+0000","last_active":"2024-05-08T21:21:53.073133+0000","last_peered":"2024-05-08T21:21:53.073133+0000","last_clean":"2024-05-08T21:21:53.073133+0000","last_became_active":"2024-05-08T21:21:52.079291+0000","last_became_peered":"2024-05-08T21:21:52.079291+0000","last_unstale":"2024-05-08T21:21:53.073133+0000","last_undegraded":"2024-05-08T21:21:53.073133+0000","last_fullsized":"2024-05-08T21:21:53.073133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:41:45.740288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073242+0000","last_change":"2024-05-08T21:21:51.613365+0000","last_active":"2024-05-08T21:21:53.073242+0000","last_peered":"2024-05-08T21:21:53.073242+0000","last_clean":"2024-05-08T21:21:53.073242+0000","last_became_active":"2024-05-08T21:21:51.613022+0000","last_became_peered":"2024-05-08T21:21:51.613022+0000","last_unstale":"2024-05-08T21:21:53.073242+0000","last_undegraded":"2024-05-08T21:21:53.073242+0000","last_fullsized":"2024-05-08T21:21:53.073242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:14:36.375522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076917+0000","last_change":"2024-05-08T21:21:52.082814+0000","last_active":"2024-05-08T21:21:53.076917+0000","last_peered":"2024-05-08T21:21:53.076917+0000","last_clean":"2024-05-08T21:21:53.076917+0000","last_became_active":"2024-05-08T21:21:52.082539+0000","last_became_peered":"2024-05-08T21:21:52.082539+0000","last_unstale":"2024-05-08T21:21:53.076917+0000","last_undegraded":"2024-05-08T21:21:53.076917+0000","last_fullsized":"2024-05-08T21:21:53.076917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:36:48.359422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.093644+0000","last_change":"2024-05-08T21:21:51.612996+0000","last_active":"2024-05-08T21:21:52.093644+0000","last_peered":"2024-05-08T21:21:52.093644+0000","last_clean":"2024-05-08T21:21:52.093644+0000","last_became_active":"2024-05-08T21:21:51.612671+0000","last_became_peered":"2024-05-08T21:21:51.612671+0000","last_unstale":"2024-05-08T21:21:52.093644+0000","last_undegraded":"2024-05-08T21:21:52.093644+0000","last_fullsized":"2024-05-08T21:21:52.093644+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:44:53.582574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603051+0000","last_change":"2024-05-08T21:21:52.075455+0000","last_active":"2024-05-08T21:21:55.603051+0000","last_peered":"2024-05-08T21:21:55.603051+0000","last_clean":"2024-05-08T21:21:55.603051+0000","last_became_active":"2024-05-08T21:21:52.075082+0000","last_became_peered":"2024-05-08T21:21:52.075082+0000","last_unstale":"2024-05-08T21:21:55.603051+0000","last_undegraded":"2024-05-08T21:21:55.603051+0000","last_fullsized":"2024-05-08T21:21:55.603051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:02.395932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073416+0000","last_change":"2024-05-08T21:21:51.614333+0000","last_active":"2024-05-08T21:21:52.073416+0000","last_peered":"2024-05-08T21:21:52.073416+0000","last_clean":"2024-05-08T21:21:52.073416+0000","last_became_active":"2024-05-08T21:21:51.614048+0000","last_became_peered":"2024-05-08T21:21:51.614048+0000","last_unstale":"2024-05-08T21:21:52.073416+0000","last_undegraded":"2024-05-08T21:21:52.073416+0000","last_fullsized":"2024-05-08T21:21:52.073416+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:37:21.706277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076323+0000","last_change":"2024-05-08T21:21:52.094009+0000","last_active":"2024-05-08T21:21:53.076323+0000","last_peered":"2024-05-08T21:21:53.076323+0000","last_clean":"2024-05-08T21:21:53.076323+0000","last_became_active":"2024-05-08T21:21:52.093730+0000","last_became_peered":"2024-05-08T21:21:52.093730+0000","last_unstale":"2024-05-08T21:21:53.076323+0000","last_undegraded":"2024-05-08T21:21:53.076323+0000","last_fullsized":"2024-05-08T21:21:53.076323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:14.568989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.068928+0000","last_change":"2024-05-08T21:21:51.620291+0000","last_active":"2024-05-08T21:21:52.068928+0000","last_peered":"2024-05-08T21:21:52.068928+0000","last_clean":"2024-05-08T21:21:52.068928+0000","last_became_active":"2024-05-08T21:21:51.619965+0000","last_became_peered":"2024-05-08T21:21:51.619965+0000","last_unstale":"2024-05-08T21:21:52.068928+0000","last_undegraded":"2024-05-08T21:21:52.068928+0000","last_fullsized":"2024-05-08T21:21:52.068928+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:05:44.822440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076439+0000","last_change":"2024-05-08T21:21:52.086946+0000","last_active":"2024-05-08T21:21:53.076439+0000","last_peered":"2024-05-08T21:21:53.076439+0000","last_clean":"2024-05-08T21:21:53.076439+0000","last_became_active":"2024-05-08T21:21:52.086651+0000","last_became_peered":"2024-05-08T21:21:52.086651+0000","last_unstale":"2024-05-08T21:21:53.076439+0000","last_undegraded":"2024-05-08T21:21:53.076439+0000","last_fullsized":"2024-05-08T21:21:53.076439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:22:38.788235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.068205+0000","last_change":"2024-05-08T21:21:51.616253+0000","last_active":"2024-05-08T21:21:52.068205+0000","last_peered":"2024-05-08T21:21:52.068205+0000","last_clean":"2024-05-08T21:21:52.068205+0000","last_became_active":"2024-05-08T21:21:51.615883+0000","last_became_peered":"2024-05-08T21:21:51.615883+0000","last_unstale":"2024-05-08T21:21:52.068205+0000","last_undegraded":"2024-05-08T21:21:52.068205+0000","last_fullsized":"2024-05-08T21:21:52.068205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:13.563899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076873+0000","last_change":"2024-05-08T21:21:52.081159+0000","last_active":"2024-05-08T21:21:53.076873+0000","last_peered":"2024-05-08T21:21:53.076873+0000","last_clean":"2024-05-08T21:21:53.076873+0000","last_became_active":"2024-05-08T21:21:52.080823+0000","last_became_peered":"2024-05-08T21:21:52.080823+0000","last_unstale":"2024-05-08T21:21:53.076873+0000","last_undegraded":"2024-05-08T21:21:53.076873+0000","last_fullsized":"2024-05-08T21:21:53.076873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:41:45.740288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.100647+0000","last_change":"2024-05-08T21:21:51.609896+0000","last_active":"2024-05-08T21:21:52.100647+0000","last_peered":"2024-05-08T21:21:52.100647+0000","last_clean":"2024-05-08T21:21:52.100647+0000","last_became_active":"2024-05-08T21:21:51.609409+0000","last_became_peered":"2024-05-08T21:21:51.609409+0000","last_unstale":"2024-05-08T21:21:52.100647+0000","last_undegraded":"2024-05-08T21:21:52.100647+0000","last_fullsized":"2024-05-08T21:21:52.100647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:38:17.091010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603571+0000","last_change":"2024-05-08T21:21:52.101873+0000","last_active":"2024-05-08T21:21:55.603571+0000","last_peered":"2024-05-08T21:21:55.603571+0000","last_clean":"2024-05-08T21:21:55.603571+0000","last_became_active":"2024-05-08T21:21:52.101487+0000","last_became_peered":"2024-05-08T21:21:52.101487+0000","last_unstale":"2024-05-08T21:21:55.603571+0000","last_undegraded":"2024-05-08T21:21:55.603571+0000","last_fullsized":"2024-05-08T21:21:55.603571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:02.395932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073673+0000","last_change":"2024-05-08T21:21:51.620697+0000","last_active":"2024-05-08T21:21:52.073673+0000","last_peered":"2024-05-08T21:21:52.073673+0000","last_clean":"2024-05-08T21:21:52.073673+0000","last_became_active":"2024-05-08T21:21:51.620370+0000","last_became_peered":"2024-05-08T21:21:51.620370+0000","last_unstale":"2024-05-08T21:21:52.073673+0000","last_undegraded":"2024-05-08T21:21:52.073673+0000","last_fullsized":"2024-05-08T21:21:52.073673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:00:55.530393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603751+0000","last_change":"2024-05-08T21:21:52.089908+0000","last_active":"2024-05-08T21:21:55.603751+0000","last_peered":"2024-05-08T21:21:55.603751+0000","last_clean":"2024-05-08T21:21:55.603751+0000","last_became_active":"2024-05-08T21:21:52.089610+0000","last_became_peered":"2024-05-08T21:21:52.089610+0000","last_unstale":"2024-05-08T21:21:55.603751+0000","last_undegraded":"2024-05-08T21:21:55.603751+0000","last_fullsized":"2024-05-08T21:21:55.603751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:26:01.523710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.089213+0000","last_change":"2024-05-08T21:21:51.610168+0000","last_active":"2024-05-08T21:21:52.089213+0000","last_peered":"2024-05-08T21:21:52.089213+0000","last_clean":"2024-05-08T21:21:52.089213+0000","last_became_active":"2024-05-08T21:21:51.609850+0000","last_became_peered":"2024-05-08T21:21:51.609850+0000","last_unstale":"2024-05-08T21:21:52.089213+0000","last_undegraded":"2024-05-08T21:21:52.089213+0000","last_fullsized":"2024-05-08T21:21:52.089213+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:41.009726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603664+0000","last_change":"2024-05-08T21:21:52.090014+0000","last_active":"2024-05-08T21:21:55.603664+0000","last_peered":"2024-05-08T21:21:55.603664+0000","last_clean":"2024-05-08T21:21:55.603664+0000","last_became_active":"2024-05-08T21:21:52.089630+0000","last_became_peered":"2024-05-08T21:21:52.089630+0000","last_unstale":"2024-05-08T21:21:55.603664+0000","last_undegraded":"2024-05-08T21:21:55.603664+0000","last_fullsized":"2024-05-08T21:21:55.603664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:26:01.523710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.088752+0000","last_change":"2024-05-08T21:21:51.613229+0000","last_active":"2024-05-08T21:21:52.088752+0000","last_peered":"2024-05-08T21:21:52.088752+0000","last_clean":"2024-05-08T21:21:52.088752+0000","last_became_active":"2024-05-08T21:21:51.612871+0000","last_became_peered":"2024-05-08T21:21:51.612871+0000","last_unstale":"2024-05-08T21:21:52.088752+0000","last_undegraded":"2024-05-08T21:21:52.088752+0000","last_fullsized":"2024-05-08T21:21:52.088752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:45:34.421641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604321+0000","last_change":"2024-05-08T21:21:52.097257+0000","last_active":"2024-05-08T21:21:55.604321+0000","last_peered":"2024-05-08T21:21:55.604321+0000","last_clean":"2024-05-08T21:21:55.604321+0000","last_became_active":"2024-05-08T21:21:52.096970+0000","last_became_peered":"2024-05-08T21:21:52.096970+0000","last_unstale":"2024-05-08T21:21:55.604321+0000","last_undegraded":"2024-05-08T21:21:55.604321+0000","last_fullsized":"2024-05-08T21:21:55.604321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:33:44.560880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602552+0000","last_change":"2024-05-08T21:21:52.089064+0000","last_active":"2024-05-08T21:21:55.602552+0000","last_peered":"2024-05-08T21:21:55.602552+0000","last_clean":"2024-05-08T21:21:55.602552+0000","last_became_active":"2024-05-08T21:21:52.088633+0000","last_became_peered":"2024-05-08T21:21:52.088633+0000","last_unstale":"2024-05-08T21:21:55.602552+0000","last_undegraded":"2024-05-08T21:21:55.602552+0000","last_fullsized":"2024-05-08T21:21:55.602552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:39:11.021535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602557+0000","last_change":"2024-05-08T21:21:52.087199+0000","last_active":"2024-05-08T21:21:55.602557+0000","last_peered":"2024-05-08T21:21:55.602557+0000","last_clean":"2024-05-08T21:21:55.602557+0000","last_became_active":"2024-05-08T21:21:52.086874+0000","last_became_peered":"2024-05-08T21:21:52.086874+0000","last_unstale":"2024-05-08T21:21:55.602557+0000","last_undegraded":"2024-05-08T21:21:55.602557+0000","last_fullsized":"2024-05-08T21:21:55.602557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:18:36.173550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603028+0000","last_change":"2024-05-08T21:21:52.090034+0000","last_active":"2024-05-08T21:21:55.603028+0000","last_peered":"2024-05-08T21:21:55.603028+0000","last_clean":"2024-05-08T21:21:55.603028+0000","last_became_active":"2024-05-08T21:21:52.089251+0000","last_became_peered":"2024-05-08T21:21:52.089251+0000","last_unstale":"2024-05-08T21:21:55.603028+0000","last_undegraded":"2024-05-08T21:21:55.603028+0000","last_fullsized":"2024-05-08T21:21:55.603028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:41.471400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603108+0000","last_change":"2024-05-08T21:21:52.085409+0000","last_active":"2024-05-08T21:21:55.603108+0000","last_peered":"2024-05-08T21:21:55.603108+0000","last_clean":"2024-05-08T21:21:55.603108+0000","last_became_active":"2024-05-08T21:21:52.085050+0000","last_became_peered":"2024-05-08T21:21:52.085050+0000","last_unstale":"2024-05-08T21:21:55.603108+0000","last_undegraded":"2024-05-08T21:21:55.603108+0000","last_fullsized":"2024-05-08T21:21:55.603108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:40:06.048132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076692+0000","last_change":"2024-05-08T21:21:52.100833+0000","last_active":"2024-05-08T21:21:53.076692+0000","last_peered":"2024-05-08T21:21:53.076692+0000","last_clean":"2024-05-08T21:21:53.076692+0000","last_became_active":"2024-05-08T21:21:52.100640+0000","last_became_peered":"2024-05-08T21:21:52.100640+0000","last_unstale":"2024-05-08T21:21:53.076692+0000","last_undegraded":"2024-05-08T21:21:53.076692+0000","last_fullsized":"2024-05-08T21:21:53.076692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:02.395932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073296+0000","last_change":"2024-05-08T21:21:52.080879+0000","last_active":"2024-05-08T21:21:53.073296+0000","last_peered":"2024-05-08T21:21:53.073296+0000","last_clean":"2024-05-08T21:21:53.073296+0000","last_became_active":"2024-05-08T21:21:52.080592+0000","last_became_peered":"2024-05-08T21:21:52.080592+0000","last_unstale":"2024-05-08T21:21:53.073296+0000","last_undegraded":"2024-05-08T21:21:53.073296+0000","last_fullsized":"2024-05-08T21:21:53.073296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:39:11.021535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603171+0000","last_change":"2024-05-08T21:21:52.092091+0000","last_active":"2024-05-08T21:21:55.603171+0000","last_peered":"2024-05-08T21:21:55.603171+0000","last_clean":"2024-05-08T21:21:55.603171+0000","last_became_active":"2024-05-08T21:21:52.091489+0000","last_became_peered":"2024-05-08T21:21:52.091489+0000","last_unstale":"2024-05-08T21:21:55.603171+0000","last_undegraded":"2024-05-08T21:21:55.603171+0000","last_fullsized":"2024-05-08T21:21:55.603171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:38.782318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076796+0000","last_change":"2024-05-08T21:21:52.087595+0000","last_active":"2024-05-08T21:21:53.076796+0000","last_peered":"2024-05-08T21:21:53.076796+0000","last_clean":"2024-05-08T21:21:53.076796+0000","last_became_active":"2024-05-08T21:21:52.087302+0000","last_became_peered":"2024-05-08T21:21:52.087302+0000","last_unstale":"2024-05-08T21:21:53.076796+0000","last_undegraded":"2024-05-08T21:21:53.076796+0000","last_fullsized":"2024-05-08T21:21:53.076796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:40:06.048132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603267+0000","last_change":"2024-05-08T21:21:52.089760+0000","last_active":"2024-05-08T21:21:55.603267+0000","last_peered":"2024-05-08T21:21:55.603267+0000","last_clean":"2024-05-08T21:21:55.603267+0000","last_became_active":"2024-05-08T21:21:52.088814+0000","last_became_peered":"2024-05-08T21:21:52.088814+0000","last_unstale":"2024-05-08T21:21:55.603267+0000","last_undegraded":"2024-05-08T21:21:55.603267+0000","last_fullsized":"2024-05-08T21:21:55.603267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:04:56.104718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076502+0000","last_change":"2024-05-08T21:21:52.089062+0000","last_active":"2024-05-08T21:21:53.076502+0000","last_peered":"2024-05-08T21:21:53.076502+0000","last_clean":"2024-05-08T21:21:53.076502+0000","last_became_active":"2024-05-08T21:21:52.088653+0000","last_became_peered":"2024-05-08T21:21:52.088653+0000","last_unstale":"2024-05-08T21:21:53.076502+0000","last_undegraded":"2024-05-08T21:21:53.076502+0000","last_fullsized":"2024-05-08T21:21:53.076502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:56:56.333047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602914+0000","last_change":"2024-05-08T21:21:52.096639+0000","last_active":"2024-05-08T21:21:55.602914+0000","last_peered":"2024-05-08T21:21:55.602914+0000","last_clean":"2024-05-08T21:21:55.602914+0000","last_became_active":"2024-05-08T21:21:52.096444+0000","last_became_peered":"2024-05-08T21:21:52.096444+0000","last_unstale":"2024-05-08T21:21:55.602914+0000","last_undegraded":"2024-05-08T21:21:55.602914+0000","last_fullsized":"2024-05-08T21:21:55.602914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:10:50.287200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603006+0000","last_change":"2024-05-08T21:21:52.082022+0000","last_active":"2024-05-08T21:21:55.603006+0000","last_peered":"2024-05-08T21:21:55.603006+0000","last_clean":"2024-05-08T21:21:55.603006+0000","last_became_active":"2024-05-08T21:21:52.081764+0000","last_became_peered":"2024-05-08T21:21:52.081764+0000","last_unstale":"2024-05-08T21:21:55.603006+0000","last_undegraded":"2024-05-08T21:21:55.603006+0000","last_fullsized":"2024-05-08T21:21:55.603006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:22:38.788235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602807+0000","last_change":"2024-05-08T21:21:52.101119+0000","last_active":"2024-05-08T21:21:55.602807+0000","last_peered":"2024-05-08T21:21:55.602807+0000","last_clean":"2024-05-08T21:21:55.602807+0000","last_became_active":"2024-05-08T21:21:52.100863+0000","last_became_peered":"2024-05-08T21:21:52.100863+0000","last_unstale":"2024-05-08T21:21:55.602807+0000","last_undegraded":"2024-05-08T21:21:55.602807+0000","last_fullsized":"2024-05-08T21:21:55.602807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:40:06.048132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":661904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":70988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T21:22:00.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T21:22:01.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.166+0000 7f0924361700 1 Processor -- start 2024-05-08T21:22:01.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.166+0000 7f0924361700 1 -- start start 2024-05-08T21:22:01.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.167+0000 7f0924361700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f091c0cd4e0 0x7f091c06f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:01.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.167+0000 7f0924361700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f091c06f130 0x7f091c0ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:01.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.167+0000 7f0924361700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f091c0ceb20 0x7f091c15e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:01.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.167+0000 7f0924361700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091c0cefb0 con 0x7f091c0cd4e0 2024-05-08T21:22:01.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.167+0000 7f0924361700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091c15c280 con 0x7f091c06f130 2024-05-08T21:22:01.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.167+0000 7f0924361700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091c06e780 con 0x7f091c0ceb20 2024-05-08T21:22:01.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.167+0000 7f09220fd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f091c0cd4e0 0x7f091c06f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:01.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.167+0000 7f09220fd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f091c0cd4e0 0x7f091c06f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52598/0 (socket says 172.21.15.3:52598) 2024-05-08T21:22:01.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.167+0000 7f09220fd700 1 -- 172.21.15.3:0/1762930180 learned_addr learned my addr 172.21.15.3:0/1762930180 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:01.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.167+0000 7f09218fc700 1 --2- 172.21.15.3:0/1762930180 >> v2:172.21.15.186:3300/0 conn(0x7f091c06f130 0x7f091c0ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:01.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.168+0000 7f09228fe700 1 --2- 172.21.15.3:0/1762930180 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f091c0ceb20 0x7f091c15e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:01.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.168+0000 7f09220fd700 1 -- 172.21.15.3:0/1762930180 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f091c0ceb20 msgr2=0x7f091c15e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:01.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.168+0000 7f09220fd700 1 --2- 172.21.15.3:0/1762930180 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f091c0ceb20 0x7f091c15e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.168+0000 7f09220fd700 1 -- 172.21.15.3:0/1762930180 >> v2:172.21.15.186:3300/0 conn(0x7f091c06f130 msgr2=0x7f091c0ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:01.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.168+0000 7f09220fd700 1 --2- 172.21.15.3:0/1762930180 >> v2:172.21.15.186:3300/0 conn(0x7f091c06f130 0x7f091c0ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.168+0000 7f09220fd700 1 -- 172.21.15.3:0/1762930180 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c06ea60 con 0x7f091c0cd4e0 2024-05-08T21:22:01.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.168+0000 7f09218fc700 1 --2- 172.21.15.3:0/1762930180 >> v2:172.21.15.186:3300/0 conn(0x7f091c06f130 0x7f091c0ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:01.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.168+0000 7f09220fd700 1 --2- 172.21.15.3:0/1762930180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f091c0cd4e0 0x7f091c06f910 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f091c0c3e20 tx=0x7f091800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab27b4b6cb0998b server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:01.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.168+0000 7f09210fb700 1 -- 172.21.15.3:0/1762930180 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091800dd30 con 0x7f091c0cd4e0 2024-05-08T21:22:01.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.168+0000 7f09210fb700 1 -- 172.21.15.3:0/1762930180 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f091800de90 con 0x7f091c0cd4e0 2024-05-08T21:22:01.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.168+0000 7f09210fb700 1 -- 172.21.15.3:0/1762930180 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0918010690 con 0x7f091c0cd4e0 2024-05-08T21:22:01.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.169+0000 7f0924361700 1 -- 172.21.15.3:0/1762930180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f091c0cd4e0 msgr2=0x7f091c06f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:01.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.169+0000 7f0924361700 1 --2- 172.21.15.3:0/1762930180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f091c0cd4e0 0x7f091c06f910 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f091c0c3e20 tx=0x7f091800be70 comp rx=0 tx=0).stop 2024-05-08T21:22:01.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.169+0000 7f0924361700 1 -- 172.21.15.3:0/1762930180 shutdown_connections 2024-05-08T21:22:01.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.169+0000 7f0924361700 1 --2- 172.21.15.3:0/1762930180 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f091c0ceb20 0x7f091c15e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.169+0000 7f0924361700 1 --2- 172.21.15.3:0/1762930180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f091c0cd4e0 0x7f091c06f910 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.169+0000 7f0924361700 1 --2- 172.21.15.3:0/1762930180 >> v2:172.21.15.186:3300/0 conn(0x7f091c06f130 0x7f091c0ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.169+0000 7f0924361700 1 -- 172.21.15.3:0/1762930180 >> 172.21.15.3:0/1762930180 conn(0x7f091c0d0320 msgr2=0x7f091c0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:01.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.169+0000 7f0924361700 1 -- 172.21.15.3:0/1762930180 shutdown_connections 2024-05-08T21:22:01.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.169+0000 7f0924361700 1 -- 172.21.15.3:0/1762930180 wait complete. 2024-05-08T21:22:01.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.170+0000 7f0924361700 1 Processor -- start 2024-05-08T21:22:01.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.170+0000 7f0924361700 1 -- start start 2024-05-08T21:22:01.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f0924361700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f091c06f130 0x7f091c150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:01.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f0924361700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f091c0cd4e0 0x7f091c0b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:01.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f0924361700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f091c0ceb20 0x7f091c0b5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:01.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f0924361700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c150d70 con 0x7f091c0ceb20 2024-05-08T21:22:01.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f0924361700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c0b48b0 con 0x7f091c06f130 2024-05-08T21:22:01.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f0924361700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091c0b3950 con 0x7f091c0cd4e0 2024-05-08T21:22:01.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f09228fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f091c0ceb20 0x7f091c0b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:01.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f09228fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f091c0ceb20 0x7f091c0b5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52608/0 (socket says 172.21.15.3:52608) 2024-05-08T21:22:01.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f09228fe700 1 -- 172.21.15.3:0/2480423668 learned_addr learned my addr 172.21.15.3:0/2480423668 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:01.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f09220fd700 1 --2- 172.21.15.3:0/2480423668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f091c06f130 0x7f091c150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:01.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f09218fc700 1 --2- 172.21.15.3:0/2480423668 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f091c0cd4e0 0x7f091c0b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:01.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f09228fe700 1 -- 172.21.15.3:0/2480423668 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f091c0cd4e0 msgr2=0x7f091c0b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:01.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f09228fe700 1 --2- 172.21.15.3:0/2480423668 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f091c0cd4e0 0x7f091c0b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f09228fe700 1 -- 172.21.15.3:0/2480423668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f091c06f130 msgr2=0x7f091c150830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:01.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f09228fe700 1 --2- 172.21.15.3:0/2480423668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f091c06f130 0x7f091c150830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f09228fe700 1 -- 172.21.15.3:0/2480423668 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091800b7f0 con 0x7f091c0ceb20 2024-05-08T21:22:01.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.171+0000 7f09218fc700 1 --2- 172.21.15.3:0/2480423668 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f091c0cd4e0 0x7f091c0b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:01.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.172+0000 7f09228fe700 1 --2- 172.21.15.3:0/2480423668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f091c0ceb20 0x7f091c0b5370 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0914007e60 tx=0x7f091400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:01.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.172+0000 7f0912ffd700 1 -- 172.21.15.3:0/2480423668 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091400ede0 con 0x7f091c0ceb20 2024-05-08T21:22:01.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.172+0000 7f0912ffd700 1 -- 172.21.15.3:0/2480423668 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0914014420 con 0x7f091c0ceb20 2024-05-08T21:22:01.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.172+0000 7f0924361700 1 -- 172.21.15.3:0/2480423668 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091c0b3c30 con 0x7f091c0ceb20 2024-05-08T21:22:01.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.172+0000 7f0912ffd700 1 -- 172.21.15.3:0/2480423668 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09140147d0 con 0x7f091c0ceb20 2024-05-08T21:22:01.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.172+0000 7f0924361700 1 -- 172.21.15.3:0/2480423668 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091c0b4e50 con 0x7f091c0ceb20 2024-05-08T21:22:01.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.173+0000 7f0924361700 1 -- 172.21.15.3:0/2480423668 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f091c068000 con 0x7f091c0ceb20 2024-05-08T21:22:01.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.173+0000 7f0912ffd700 1 -- 172.21.15.3:0/2480423668 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0914014930 con 0x7f091c0ceb20 2024-05-08T21:22:01.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.174+0000 7f0912ffd700 1 --2- 172.21.15.3:0/2480423668 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f090006acc0 0x7f090006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:01.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.174+0000 7f0912ffd700 1 -- 172.21.15.3:0/2480423668 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09140899f0 con 0x7f091c0ceb20 2024-05-08T21:22:01.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.174+0000 7f09220fd700 1 --2- 172.21.15.3:0/2480423668 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f090006acc0 0x7f090006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:01.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.174+0000 7f09220fd700 1 --2- 172.21.15.3:0/2480423668 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f090006acc0 0x7f090006d180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f091800beb0 tx=0x7f0918012000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:01.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.178+0000 7f0912ffd700 1 -- 172.21.15.3:0/2480423668 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0914054c60 con 0x7f091c0ceb20 2024-05-08T21:22:01.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.391+0000 7f0924361700 1 -- 172.21.15.3:0/2480423668 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f091c0cafb0 con 0x7f090006acc0 2024-05-08T21:22:01.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.399+0000 7f0912ffd700 1 -- 172.21.15.3:0/2480423668 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264535 (secure 0 0 0) 0x7f091c0cafb0 con 0x7f090006acc0 2024-05-08T21:22:01.404 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:01.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.403+0000 7f0924361700 1 -- 172.21.15.3:0/2480423668 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f090006acc0 msgr2=0x7f090006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.403+0000 7f0924361700 1 --2- 172.21.15.3:0/2480423668 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f090006acc0 0x7f090006d180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f091800beb0 tx=0x7f0918012000 comp rx=0 tx=0).stop 2024-05-08T21:22:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.403+0000 7f0924361700 1 -- 172.21.15.3:0/2480423668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f091c0ceb20 msgr2=0x7f091c0b5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.403+0000 7f0924361700 1 --2- 172.21.15.3:0/2480423668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f091c0ceb20 0x7f091c0b5370 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0914007e60 tx=0x7f091400e3c0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.403+0000 7f0924361700 1 -- 172.21.15.3:0/2480423668 shutdown_connections 2024-05-08T21:22:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.403+0000 7f0924361700 1 --2- 172.21.15.3:0/2480423668 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f090006acc0 0x7f090006d180 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.403+0000 7f0924361700 1 --2- 172.21.15.3:0/2480423668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f091c0ceb20 0x7f091c0b5370 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.403+0000 7f0924361700 1 --2- 172.21.15.3:0/2480423668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f091c06f130 0x7f091c150830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.403+0000 7f0924361700 1 --2- 172.21.15.3:0/2480423668 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f091c0cd4e0 0x7f091c0b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.403+0000 7f0924361700 1 -- 172.21.15.3:0/2480423668 >> 172.21.15.3:0/2480423668 conn(0x7f091c0d0320 msgr2=0x7f091c06dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.403+0000 7f0924361700 1 -- 172.21.15.3:0/2480423668 shutdown_connections 2024-05-08T21:22:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.403+0000 7f0924361700 1 -- 172.21.15.3:0/2480423668 wait complete. 2024-05-08T21:22:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-08T21:22:01.418 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-05-08T21:22:00.533048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218984,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755736,"statfs":{"total":773094113280,"available":772869873664,"internally_reserved":0,"allocated":2891776,"data_stored":1752644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.465691"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604264+0000","last_change":"2024-05-08T21:21:52.093899+0000","last_active":"2024-05-08T21:21:55.604264+0000","last_peered":"2024-05-08T21:21:55.604264+0000","last_clean":"2024-05-08T21:21:55.604264+0000","last_became_active":"2024-05-08T21:21:52.093609+0000","last_became_peered":"2024-05-08T21:21:52.093609+0000","last_unstale":"2024-05-08T21:21:55.604264+0000","last_undegraded":"2024-05-08T21:21:55.604264+0000","last_fullsized":"2024-05-08T21:21:55.604264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:22:03.106073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073419+0000","last_change":"2024-05-08T21:21:52.074817+0000","last_active":"2024-05-08T21:21:53.073419+0000","last_peered":"2024-05-08T21:21:53.073419+0000","last_clean":"2024-05-08T21:21:53.073419+0000","last_became_active":"2024-05-08T21:21:52.074544+0000","last_became_peered":"2024-05-08T21:21:52.074544+0000","last_unstale":"2024-05-08T21:21:53.073419+0000","last_undegraded":"2024-05-08T21:21:53.073419+0000","last_fullsized":"2024-05-08T21:21:53.073419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:22:38.788235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603137+0000","last_change":"2024-05-08T21:21:52.082843+0000","last_active":"2024-05-08T21:21:55.603137+0000","last_peered":"2024-05-08T21:21:55.603137+0000","last_clean":"2024-05-08T21:21:55.603137+0000","last_became_active":"2024-05-08T21:21:52.080398+0000","last_became_peered":"2024-05-08T21:21:52.080398+0000","last_unstale":"2024-05-08T21:21:55.603137+0000","last_undegraded":"2024-05-08T21:21:55.603137+0000","last_fullsized":"2024-05-08T21:21:55.603137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:36:48.359422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603875+0000","last_change":"2024-05-08T21:21:52.086712+0000","last_active":"2024-05-08T21:21:55.603875+0000","last_peered":"2024-05-08T21:21:55.603875+0000","last_clean":"2024-05-08T21:21:55.603875+0000","last_became_active":"2024-05-08T21:21:52.086445+0000","last_became_peered":"2024-05-08T21:21:52.086445+0000","last_unstale":"2024-05-08T21:21:55.603875+0000","last_undegraded":"2024-05-08T21:21:55.603875+0000","last_fullsized":"2024-05-08T21:21:55.603875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:22:38.788235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603913+0000","last_change":"2024-05-08T21:21:52.085542+0000","last_active":"2024-05-08T21:21:55.603913+0000","last_peered":"2024-05-08T21:21:55.603913+0000","last_clean":"2024-05-08T21:21:55.603913+0000","last_became_active":"2024-05-08T21:21:52.085253+0000","last_became_peered":"2024-05-08T21:21:52.085253+0000","last_unstale":"2024-05-08T21:21:55.603913+0000","last_undegraded":"2024-05-08T21:21:55.603913+0000","last_fullsized":"2024-05-08T21:21:55.603913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:54:03.930846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602797+0000","last_change":"2024-05-08T21:21:52.090302+0000","last_active":"2024-05-08T21:21:55.602797+0000","last_peered":"2024-05-08T21:21:55.602797+0000","last_clean":"2024-05-08T21:21:55.602797+0000","last_became_active":"2024-05-08T21:21:52.088764+0000","last_became_peered":"2024-05-08T21:21:52.088764+0000","last_unstale":"2024-05-08T21:21:55.602797+0000","last_undegraded":"2024-05-08T21:21:55.602797+0000","last_fullsized":"2024-05-08T21:21:55.602797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:09:59.496154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603367+0000","last_change":"2024-05-08T21:21:52.095147+0000","last_active":"2024-05-08T21:21:55.603367+0000","last_peered":"2024-05-08T21:21:55.603367+0000","last_clean":"2024-05-08T21:21:55.603367+0000","last_became_active":"2024-05-08T21:21:52.094954+0000","last_became_peered":"2024-05-08T21:21:52.094954+0000","last_unstale":"2024-05-08T21:21:55.603367+0000","last_undegraded":"2024-05-08T21:21:55.603367+0000","last_fullsized":"2024-05-08T21:21:55.603367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:09:59.496154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076360+0000","last_change":"2024-05-08T21:21:52.083974+0000","last_active":"2024-05-08T21:21:53.076360+0000","last_peered":"2024-05-08T21:21:53.076360+0000","last_clean":"2024-05-08T21:21:53.076360+0000","last_became_active":"2024-05-08T21:21:52.083702+0000","last_became_peered":"2024-05-08T21:21:52.083702+0000","last_unstale":"2024-05-08T21:21:53.076360+0000","last_undegraded":"2024-05-08T21:21:53.076360+0000","last_fullsized":"2024-05-08T21:21:53.076360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:54:03.930846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604376+0000","last_change":"2024-05-08T21:21:52.087758+0000","last_active":"2024-05-08T21:21:55.604376+0000","last_peered":"2024-05-08T21:21:55.604376+0000","last_clean":"2024-05-08T21:21:55.604376+0000","last_became_active":"2024-05-08T21:21:52.087489+0000","last_became_peered":"2024-05-08T21:21:52.087489+0000","last_unstale":"2024-05-08T21:21:55.604376+0000","last_undegraded":"2024-05-08T21:21:55.604376+0000","last_fullsized":"2024-05-08T21:21:55.604376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:08.050201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603328+0000","last_change":"2024-05-08T21:21:52.092265+0000","last_active":"2024-05-08T21:21:55.603328+0000","last_peered":"2024-05-08T21:21:55.603328+0000","last_clean":"2024-05-08T21:21:55.603328+0000","last_became_active":"2024-05-08T21:21:52.091813+0000","last_became_peered":"2024-05-08T21:21:52.091813+0000","last_unstale":"2024-05-08T21:21:55.603328+0000","last_undegraded":"2024-05-08T21:21:55.603328+0000","last_fullsized":"2024-05-08T21:21:55.603328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:04:56.104718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602830+0000","last_change":"2024-05-08T21:21:52.087586+0000","last_active":"2024-05-08T21:21:55.602830+0000","last_peered":"2024-05-08T21:21:55.602830+0000","last_clean":"2024-05-08T21:21:55.602830+0000","last_became_active":"2024-05-08T21:21:52.087302+0000","last_became_peered":"2024-05-08T21:21:52.087302+0000","last_unstale":"2024-05-08T21:21:55.602830+0000","last_undegraded":"2024-05-08T21:21:55.602830+0000","last_fullsized":"2024-05-08T21:21:55.602830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:41:45.740288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076636+0000","last_change":"2024-05-08T21:21:52.094218+0000","last_active":"2024-05-08T21:21:53.076636+0000","last_peered":"2024-05-08T21:21:53.076636+0000","last_clean":"2024-05-08T21:21:53.076636+0000","last_became_active":"2024-05-08T21:21:52.093855+0000","last_became_peered":"2024-05-08T21:21:52.093855+0000","last_unstale":"2024-05-08T21:21:53.076636+0000","last_undegraded":"2024-05-08T21:21:53.076636+0000","last_fullsized":"2024-05-08T21:21:53.076636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:18:36.173550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076649+0000","last_change":"2024-05-08T21:21:52.093685+0000","last_active":"2024-05-08T21:21:53.076649+0000","last_peered":"2024-05-08T21:21:53.076649+0000","last_clean":"2024-05-08T21:21:53.076649+0000","last_became_active":"2024-05-08T21:21:52.093380+0000","last_became_peered":"2024-05-08T21:21:52.093380+0000","last_unstale":"2024-05-08T21:21:53.076649+0000","last_undegraded":"2024-05-08T21:21:53.076649+0000","last_fullsized":"2024-05-08T21:21:53.076649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:39:11.021535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603501+0000","last_change":"2024-05-08T21:21:52.077438+0000","last_active":"2024-05-08T21:21:55.603501+0000","last_peered":"2024-05-08T21:21:55.603501+0000","last_clean":"2024-05-08T21:21:55.603501+0000","last_became_active":"2024-05-08T21:21:52.077129+0000","last_became_peered":"2024-05-08T21:21:52.077129+0000","last_unstale":"2024-05-08T21:21:55.603501+0000","last_undegraded":"2024-05-08T21:21:55.603501+0000","last_fullsized":"2024-05-08T21:21:55.603501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:28:07.575847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073038+0000","last_change":"2024-05-08T21:21:52.096939+0000","last_active":"2024-05-08T21:21:53.073038+0000","last_peered":"2024-05-08T21:21:53.073038+0000","last_clean":"2024-05-08T21:21:53.073038+0000","last_became_active":"2024-05-08T21:21:52.096629+0000","last_became_peered":"2024-05-08T21:21:52.096629+0000","last_unstale":"2024-05-08T21:21:53.073038+0000","last_undegraded":"2024-05-08T21:21:53.073038+0000","last_fullsized":"2024-05-08T21:21:53.073038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:04:56.104718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603067+0000","last_change":"2024-05-08T21:21:52.088031+0000","last_active":"2024-05-08T21:21:55.603067+0000","last_peered":"2024-05-08T21:21:55.603067+0000","last_clean":"2024-05-08T21:21:55.603067+0000","last_became_active":"2024-05-08T21:21:52.087748+0000","last_became_peered":"2024-05-08T21:21:52.087748+0000","last_unstale":"2024-05-08T21:21:55.603067+0000","last_undegraded":"2024-05-08T21:21:55.603067+0000","last_fullsized":"2024-05-08T21:21:55.603067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:46:26.988461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604380+0000","last_change":"2024-05-08T21:21:52.096438+0000","last_active":"2024-05-08T21:21:55.604380+0000","last_peered":"2024-05-08T21:21:55.604380+0000","last_clean":"2024-05-08T21:21:55.604380+0000","last_became_active":"2024-05-08T21:21:52.095936+0000","last_became_peered":"2024-05-08T21:21:52.095936+0000","last_unstale":"2024-05-08T21:21:55.604380+0000","last_undegraded":"2024-05-08T21:21:55.604380+0000","last_fullsized":"2024-05-08T21:21:55.604380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:38:46.924064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603006+0000","last_change":"2024-05-08T21:21:52.095114+0000","last_active":"2024-05-08T21:21:55.603006+0000","last_peered":"2024-05-08T21:21:55.603006+0000","last_clean":"2024-05-08T21:21:55.603006+0000","last_became_active":"2024-05-08T21:21:52.094795+0000","last_became_peered":"2024-05-08T21:21:52.094795+0000","last_unstale":"2024-05-08T21:21:55.603006+0000","last_undegraded":"2024-05-08T21:21:55.603006+0000","last_fullsized":"2024-05-08T21:21:55.603006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:14:37.380612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603538+0000","last_change":"2024-05-08T21:21:52.084672+0000","last_active":"2024-05-08T21:21:55.603538+0000","last_peered":"2024-05-08T21:21:55.603538+0000","last_clean":"2024-05-08T21:21:55.603538+0000","last_became_active":"2024-05-08T21:21:52.084373+0000","last_became_peered":"2024-05-08T21:21:52.084373+0000","last_unstale":"2024-05-08T21:21:55.603538+0000","last_undegraded":"2024-05-08T21:21:55.603538+0000","last_fullsized":"2024-05-08T21:21:55.603538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:10:50.287200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.095184+0000","last_change":"2024-05-08T21:21:51.614469+0000","last_active":"2024-05-08T21:21:52.095184+0000","last_peered":"2024-05-08T21:21:52.095184+0000","last_clean":"2024-05-08T21:21:52.095184+0000","last_became_active":"2024-05-08T21:21:51.614173+0000","last_became_peered":"2024-05-08T21:21:51.614173+0000","last_unstale":"2024-05-08T21:21:52.095184+0000","last_undegraded":"2024-05-08T21:21:52.095184+0000","last_fullsized":"2024-05-08T21:21:52.095184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:00:55.530393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076753+0000","last_change":"2024-05-08T21:21:52.095181+0000","last_active":"2024-05-08T21:21:53.076753+0000","last_peered":"2024-05-08T21:21:53.076753+0000","last_clean":"2024-05-08T21:21:53.076753+0000","last_became_active":"2024-05-08T21:21:52.095007+0000","last_became_peered":"2024-05-08T21:21:52.095007+0000","last_unstale":"2024-05-08T21:21:53.076753+0000","last_undegraded":"2024-05-08T21:21:53.076753+0000","last_fullsized":"2024-05-08T21:21:53.076753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:28:07.575847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.091195+0000","last_change":"2024-05-08T21:21:51.615481+0000","last_active":"2024-05-08T21:21:52.091195+0000","last_peered":"2024-05-08T21:21:52.091195+0000","last_clean":"2024-05-08T21:21:52.091195+0000","last_became_active":"2024-05-08T21:21:51.614841+0000","last_became_peered":"2024-05-08T21:21:51.614841+0000","last_unstale":"2024-05-08T21:21:52.091195+0000","last_undegraded":"2024-05-08T21:21:52.091195+0000","last_fullsized":"2024-05-08T21:21:52.091195+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:10:09.131458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603223+0000","last_change":"2024-05-08T21:21:52.083798+0000","last_active":"2024-05-08T21:21:55.603223+0000","last_peered":"2024-05-08T21:21:55.603223+0000","last_clean":"2024-05-08T21:21:55.603223+0000","last_became_active":"2024-05-08T21:21:52.083361+0000","last_became_peered":"2024-05-08T21:21:52.083361+0000","last_unstale":"2024-05-08T21:21:55.603223+0000","last_undegraded":"2024-05-08T21:21:55.603223+0000","last_fullsized":"2024-05-08T21:21:55.603223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:38.782318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.094224+0000","last_change":"2024-05-08T21:21:51.610838+0000","last_active":"2024-05-08T21:21:52.094224+0000","last_peered":"2024-05-08T21:21:52.094224+0000","last_clean":"2024-05-08T21:21:52.094224+0000","last_became_active":"2024-05-08T21:21:51.610168+0000","last_became_peered":"2024-05-08T21:21:51.610168+0000","last_unstale":"2024-05-08T21:21:52.094224+0000","last_undegraded":"2024-05-08T21:21:52.094224+0000","last_fullsized":"2024-05-08T21:21:52.094224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:13.563899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602928+0000","last_change":"2024-05-08T21:21:52.101000+0000","last_active":"2024-05-08T21:21:55.602928+0000","last_peered":"2024-05-08T21:21:55.602928+0000","last_clean":"2024-05-08T21:21:55.602928+0000","last_became_active":"2024-05-08T21:21:52.100589+0000","last_became_peered":"2024-05-08T21:21:52.100589+0000","last_unstale":"2024-05-08T21:21:55.602928+0000","last_undegraded":"2024-05-08T21:21:55.602928+0000","last_fullsized":"2024-05-08T21:21:55.602928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:22:03.106073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.094794+0000","last_change":"2024-05-08T21:21:51.617528+0000","last_active":"2024-05-08T21:21:52.094794+0000","last_peered":"2024-05-08T21:21:52.094794+0000","last_clean":"2024-05-08T21:21:52.094794+0000","last_became_active":"2024-05-08T21:21:51.617225+0000","last_became_peered":"2024-05-08T21:21:51.617225+0000","last_unstale":"2024-05-08T21:21:52.094794+0000","last_undegraded":"2024-05-08T21:21:52.094794+0000","last_fullsized":"2024-05-08T21:21:52.094794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:05:44.822440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604076+0000","last_change":"2024-05-08T21:21:52.085423+0000","last_active":"2024-05-08T21:21:55.604076+0000","last_peered":"2024-05-08T21:21:55.604076+0000","last_clean":"2024-05-08T21:21:55.604076+0000","last_became_active":"2024-05-08T21:21:52.085144+0000","last_became_peered":"2024-05-08T21:21:52.085144+0000","last_unstale":"2024-05-08T21:21:55.604076+0000","last_undegraded":"2024-05-08T21:21:55.604076+0000","last_fullsized":"2024-05-08T21:21:55.604076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:54:03.930846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.092454+0000","last_change":"2024-05-08T21:21:51.611668+0000","last_active":"2024-05-08T21:21:52.092454+0000","last_peered":"2024-05-08T21:21:52.092454+0000","last_clean":"2024-05-08T21:21:52.092454+0000","last_became_active":"2024-05-08T21:21:51.611363+0000","last_became_peered":"2024-05-08T21:21:51.611363+0000","last_unstale":"2024-05-08T21:21:52.092454+0000","last_undegraded":"2024-05-08T21:21:52.092454+0000","last_fullsized":"2024-05-08T21:21:52.092454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:05.185930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076192+0000","last_change":"2024-05-08T21:21:52.092056+0000","last_active":"2024-05-08T21:21:53.076192+0000","last_peered":"2024-05-08T21:21:53.076192+0000","last_clean":"2024-05-08T21:21:53.076192+0000","last_became_active":"2024-05-08T21:21:52.086766+0000","last_became_peered":"2024-05-08T21:21:52.086766+0000","last_unstale":"2024-05-08T21:21:53.076192+0000","last_undegraded":"2024-05-08T21:21:53.076192+0000","last_fullsized":"2024-05-08T21:21:53.076192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:05:45.827530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.101711+0000","last_change":"2024-05-08T21:21:51.609576+0000","last_active":"2024-05-08T21:21:52.101711+0000","last_peered":"2024-05-08T21:21:52.101711+0000","last_clean":"2024-05-08T21:21:52.101711+0000","last_became_active":"2024-05-08T21:21:51.609004+0000","last_became_peered":"2024-05-08T21:21:51.609004+0000","last_unstale":"2024-05-08T21:21:52.101711+0000","last_undegraded":"2024-05-08T21:21:52.101711+0000","last_fullsized":"2024-05-08T21:21:52.101711+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:38:17.091010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603223+0000","last_change":"2024-05-08T21:21:52.093264+0000","last_active":"2024-05-08T21:21:55.603223+0000","last_peered":"2024-05-08T21:21:55.603223+0000","last_clean":"2024-05-08T21:21:55.603223+0000","last_became_active":"2024-05-08T21:21:52.092898+0000","last_became_peered":"2024-05-08T21:21:52.092898+0000","last_unstale":"2024-05-08T21:21:55.603223+0000","last_undegraded":"2024-05-08T21:21:55.603223+0000","last_fullsized":"2024-05-08T21:21:55.603223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:41:45.740288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073983+0000","last_change":"2024-05-08T21:21:51.609633+0000","last_active":"2024-05-08T21:21:52.073983+0000","last_peered":"2024-05-08T21:21:52.073983+0000","last_clean":"2024-05-08T21:21:52.073983+0000","last_became_active":"2024-05-08T21:21:51.609304+0000","last_became_peered":"2024-05-08T21:21:51.609304+0000","last_unstale":"2024-05-08T21:21:52.073983+0000","last_undegraded":"2024-05-08T21:21:52.073983+0000","last_fullsized":"2024-05-08T21:21:52.073983+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:20:59.795219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076482+0000","last_change":"2024-05-08T21:21:52.078368+0000","last_active":"2024-05-08T21:21:53.076482+0000","last_peered":"2024-05-08T21:21:53.076482+0000","last_clean":"2024-05-08T21:21:53.076482+0000","last_became_active":"2024-05-08T21:21:52.078095+0000","last_became_peered":"2024-05-08T21:21:52.078095+0000","last_unstale":"2024-05-08T21:21:53.076482+0000","last_undegraded":"2024-05-08T21:21:53.076482+0000","last_fullsized":"2024-05-08T21:21:53.076482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:38.782318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073269+0000","last_change":"2024-05-08T21:21:51.612841+0000","last_active":"2024-05-08T21:21:52.073269+0000","last_peered":"2024-05-08T21:21:52.073269+0000","last_clean":"2024-05-08T21:21:52.073269+0000","last_became_active":"2024-05-08T21:21:51.612563+0000","last_became_peered":"2024-05-08T21:21:51.612563+0000","last_unstale":"2024-05-08T21:21:52.073269+0000","last_undegraded":"2024-05-08T21:21:52.073269+0000","last_fullsized":"2024-05-08T21:21:52.073269+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:41.009726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073158+0000","last_change":"2024-05-08T21:21:52.088955+0000","last_active":"2024-05-08T21:21:53.073158+0000","last_peered":"2024-05-08T21:21:53.073158+0000","last_clean":"2024-05-08T21:21:53.073158+0000","last_became_active":"2024-05-08T21:21:52.088686+0000","last_became_peered":"2024-05-08T21:21:52.088686+0000","last_unstale":"2024-05-08T21:21:53.073158+0000","last_undegraded":"2024-05-08T21:21:53.073158+0000","last_fullsized":"2024-05-08T21:21:53.073158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:38.782318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.075045+0000","last_change":"2024-05-08T21:21:51.620459+0000","last_active":"2024-05-08T21:21:52.075045+0000","last_peered":"2024-05-08T21:21:52.075045+0000","last_clean":"2024-05-08T21:21:52.075045+0000","last_became_active":"2024-05-08T21:21:51.620147+0000","last_became_peered":"2024-05-08T21:21:51.620147+0000","last_unstale":"2024-05-08T21:21:52.075045+0000","last_undegraded":"2024-05-08T21:21:52.075045+0000","last_fullsized":"2024-05-08T21:21:52.075045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:13.563899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604099+0000","last_change":"2024-05-08T21:21:52.096162+0000","last_active":"2024-05-08T21:21:55.604099+0000","last_peered":"2024-05-08T21:21:55.604099+0000","last_clean":"2024-05-08T21:21:55.604099+0000","last_became_active":"2024-05-08T21:21:52.095548+0000","last_became_peered":"2024-05-08T21:21:52.095548+0000","last_unstale":"2024-05-08T21:21:55.604099+0000","last_undegraded":"2024-05-08T21:21:55.604099+0000","last_fullsized":"2024-05-08T21:21:55.604099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:33:44.560880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073170+0000","last_change":"2024-05-08T21:21:51.617174+0000","last_active":"2024-05-08T21:21:52.073170+0000","last_peered":"2024-05-08T21:21:52.073170+0000","last_clean":"2024-05-08T21:21:52.073170+0000","last_became_active":"2024-05-08T21:21:51.616894+0000","last_became_peered":"2024-05-08T21:21:51.616894+0000","last_unstale":"2024-05-08T21:21:52.073170+0000","last_undegraded":"2024-05-08T21:21:52.073170+0000","last_fullsized":"2024-05-08T21:21:52.073170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:38:17.091010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603455+0000","last_change":"2024-05-08T21:21:52.080625+0000","last_active":"2024-05-08T21:21:55.603455+0000","last_peered":"2024-05-08T21:21:55.603455+0000","last_clean":"2024-05-08T21:21:55.603455+0000","last_became_active":"2024-05-08T21:21:52.080347+0000","last_became_peered":"2024-05-08T21:21:52.080347+0000","last_unstale":"2024-05-08T21:21:55.603455+0000","last_undegraded":"2024-05-08T21:21:55.603455+0000","last_fullsized":"2024-05-08T21:21:55.603455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:04:56.104718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.087805+0000","last_change":"2024-05-08T21:21:51.617911+0000","last_active":"2024-05-08T21:21:52.087805+0000","last_peered":"2024-05-08T21:21:52.087805+0000","last_clean":"2024-05-08T21:21:52.087805+0000","last_became_active":"2024-05-08T21:21:51.617651+0000","last_became_peered":"2024-05-08T21:21:51.617651+0000","last_unstale":"2024-05-08T21:21:52.087805+0000","last_undegraded":"2024-05-08T21:21:52.087805+0000","last_fullsized":"2024-05-08T21:21:52.087805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:37:21.706277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076579+0000","last_change":"2024-05-08T21:21:52.090657+0000","last_active":"2024-05-08T21:21:53.076579+0000","last_peered":"2024-05-08T21:21:53.076579+0000","last_clean":"2024-05-08T21:21:53.076579+0000","last_became_active":"2024-05-08T21:21:52.090381+0000","last_became_peered":"2024-05-08T21:21:52.090381+0000","last_unstale":"2024-05-08T21:21:53.076579+0000","last_undegraded":"2024-05-08T21:21:53.076579+0000","last_fullsized":"2024-05-08T21:21:53.076579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:37:22.711367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.105960+0000","last_change":"2024-05-08T21:21:51.613700+0000","last_active":"2024-05-08T21:21:52.105960+0000","last_peered":"2024-05-08T21:21:52.105960+0000","last_clean":"2024-05-08T21:21:52.105960+0000","last_became_active":"2024-05-08T21:21:51.613457+0000","last_became_peered":"2024-05-08T21:21:51.613457+0000","last_unstale":"2024-05-08T21:21:52.105960+0000","last_undegraded":"2024-05-08T21:21:52.105960+0000","last_fullsized":"2024-05-08T21:21:52.105960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:00:55.530393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603663+0000","last_change":"2024-05-08T21:21:52.093097+0000","last_active":"2024-05-08T21:21:55.603663+0000","last_peered":"2024-05-08T21:21:55.603663+0000","last_clean":"2024-05-08T21:21:55.603663+0000","last_became_active":"2024-05-08T21:21:52.092769+0000","last_became_peered":"2024-05-08T21:21:52.092769+0000","last_unstale":"2024-05-08T21:21:55.603663+0000","last_undegraded":"2024-05-08T21:21:55.603663+0000","last_fullsized":"2024-05-08T21:21:55.603663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:38:46.924064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.100944+0000","last_change":"2024-05-08T21:21:51.614173+0000","last_active":"2024-05-08T21:21:52.100944+0000","last_peered":"2024-05-08T21:21:52.100944+0000","last_clean":"2024-05-08T21:21:52.100944+0000","last_became_active":"2024-05-08T21:21:51.613664+0000","last_became_peered":"2024-05-08T21:21:51.613664+0000","last_unstale":"2024-05-08T21:21:52.100944+0000","last_undegraded":"2024-05-08T21:21:52.100944+0000","last_fullsized":"2024-05-08T21:21:52.100944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:20:59.795219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602872+0000","last_change":"2024-05-08T21:21:52.081761+0000","last_active":"2024-05-08T21:21:55.602872+0000","last_peered":"2024-05-08T21:21:55.602872+0000","last_clean":"2024-05-08T21:21:55.602872+0000","last_became_active":"2024-05-08T21:21:52.081477+0000","last_became_peered":"2024-05-08T21:21:52.081477+0000","last_unstale":"2024-05-08T21:21:55.602872+0000","last_undegraded":"2024-05-08T21:21:55.602872+0000","last_fullsized":"2024-05-08T21:21:55.602872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:39:11.021535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.075759+0000","last_change":"2024-05-08T21:21:51.616956+0000","last_active":"2024-05-08T21:21:52.075759+0000","last_peered":"2024-05-08T21:21:52.075759+0000","last_clean":"2024-05-08T21:21:52.075759+0000","last_became_active":"2024-05-08T21:21:51.616670+0000","last_became_peered":"2024-05-08T21:21:51.616670+0000","last_unstale":"2024-05-08T21:21:52.075759+0000","last_undegraded":"2024-05-08T21:21:52.075759+0000","last_fullsized":"2024-05-08T21:21:52.075759+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:45:34.421641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603797+0000","last_change":"2024-05-08T21:21:52.100972+0000","last_active":"2024-05-08T21:21:55.603797+0000","last_peered":"2024-05-08T21:21:55.603797+0000","last_clean":"2024-05-08T21:21:55.603797+0000","last_became_active":"2024-05-08T21:21:52.100711+0000","last_became_peered":"2024-05-08T21:21:52.100711+0000","last_unstale":"2024-05-08T21:21:55.603797+0000","last_undegraded":"2024-05-08T21:21:55.603797+0000","last_fullsized":"2024-05-08T21:21:55.603797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:22:38.788235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073127+0000","last_change":"2024-05-08T21:21:51.609380+0000","last_active":"2024-05-08T21:21:52.073127+0000","last_peered":"2024-05-08T21:21:52.073127+0000","last_clean":"2024-05-08T21:21:52.073127+0000","last_became_active":"2024-05-08T21:21:51.608973+0000","last_became_peered":"2024-05-08T21:21:51.608973+0000","last_unstale":"2024-05-08T21:21:52.073127+0000","last_undegraded":"2024-05-08T21:21:52.073127+0000","last_fullsized":"2024-05-08T21:21:52.073127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:45:34.421641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602586+0000","last_change":"2024-05-08T21:21:52.088958+0000","last_active":"2024-05-08T21:21:55.602586+0000","last_peered":"2024-05-08T21:21:55.602586+0000","last_clean":"2024-05-08T21:21:55.602586+0000","last_became_active":"2024-05-08T21:21:52.088270+0000","last_became_peered":"2024-05-08T21:21:52.088270+0000","last_unstale":"2024-05-08T21:21:55.602586+0000","last_undegraded":"2024-05-08T21:21:55.602586+0000","last_fullsized":"2024-05-08T21:21:55.602586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:28:07.575847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.090786+0000","last_change":"2024-05-08T21:21:51.614824+0000","last_active":"2024-05-08T21:21:52.090786+0000","last_peered":"2024-05-08T21:21:52.090786+0000","last_clean":"2024-05-08T21:21:52.090786+0000","last_became_active":"2024-05-08T21:21:51.614526+0000","last_became_peered":"2024-05-08T21:21:51.614526+0000","last_unstale":"2024-05-08T21:21:52.090786+0000","last_undegraded":"2024-05-08T21:21:52.090786+0000","last_fullsized":"2024-05-08T21:21:52.090786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:41.009726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603517+0000","last_change":"2024-05-08T21:21:52.097367+0000","last_active":"2024-05-08T21:21:55.603517+0000","last_peered":"2024-05-08T21:21:55.603517+0000","last_clean":"2024-05-08T21:21:55.603517+0000","last_became_active":"2024-05-08T21:21:52.097047+0000","last_became_peered":"2024-05-08T21:21:52.097047+0000","last_unstale":"2024-05-08T21:21:55.603517+0000","last_undegraded":"2024-05-08T21:21:55.603517+0000","last_fullsized":"2024-05-08T21:21:55.603517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:46:26.988461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.100630+0000","last_change":"2024-05-08T21:21:51.615362+0000","last_active":"2024-05-08T21:21:52.100630+0000","last_peered":"2024-05-08T21:21:52.100630+0000","last_clean":"2024-05-08T21:21:52.100630+0000","last_became_active":"2024-05-08T21:21:51.615119+0000","last_became_peered":"2024-05-08T21:21:51.615119+0000","last_unstale":"2024-05-08T21:21:52.100630+0000","last_undegraded":"2024-05-08T21:21:52.100630+0000","last_fullsized":"2024-05-08T21:21:52.100630+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:10:09.131458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603677+0000","last_change":"2024-05-08T21:21:52.086491+0000","last_active":"2024-05-08T21:21:55.603677+0000","last_peered":"2024-05-08T21:21:55.603677+0000","last_clean":"2024-05-08T21:21:55.603677+0000","last_became_active":"2024-05-08T21:21:52.086134+0000","last_became_peered":"2024-05-08T21:21:52.086134+0000","last_unstale":"2024-05-08T21:21:55.603677+0000","last_undegraded":"2024-05-08T21:21:55.603677+0000","last_fullsized":"2024-05-08T21:21:55.603677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:41.471400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.070417+0000","last_change":"2024-05-08T21:21:51.617080+0000","last_active":"2024-05-08T21:21:52.070417+0000","last_peered":"2024-05-08T21:21:52.070417+0000","last_clean":"2024-05-08T21:21:52.070417+0000","last_became_active":"2024-05-08T21:21:51.615569+0000","last_became_peered":"2024-05-08T21:21:51.615569+0000","last_unstale":"2024-05-08T21:21:52.070417+0000","last_undegraded":"2024-05-08T21:21:52.070417+0000","last_fullsized":"2024-05-08T21:21:52.070417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:44:53.582574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604152+0000","last_change":"2024-05-08T21:21:52.089644+0000","last_active":"2024-05-08T21:21:55.604152+0000","last_peered":"2024-05-08T21:21:55.604152+0000","last_clean":"2024-05-08T21:21:55.604152+0000","last_became_active":"2024-05-08T21:21:52.089161+0000","last_became_peered":"2024-05-08T21:21:52.089161+0000","last_unstale":"2024-05-08T21:21:55.604152+0000","last_undegraded":"2024-05-08T21:21:55.604152+0000","last_fullsized":"2024-05-08T21:21:55.604152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:28:07.575847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.075661+0000","last_change":"2024-05-08T21:21:51.618799+0000","last_active":"2024-05-08T21:21:52.075661+0000","last_peered":"2024-05-08T21:21:52.075661+0000","last_clean":"2024-05-08T21:21:52.075661+0000","last_became_active":"2024-05-08T21:21:51.618584+0000","last_became_peered":"2024-05-08T21:21:51.618584+0000","last_unstale":"2024-05-08T21:21:52.075661+0000","last_undegraded":"2024-05-08T21:21:52.075661+0000","last_fullsized":"2024-05-08T21:21:52.075661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:44:53.582574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076748+0000","last_change":"2024-05-08T21:21:52.090935+0000","last_active":"2024-05-08T21:21:53.076748+0000","last_peered":"2024-05-08T21:21:53.076748+0000","last_clean":"2024-05-08T21:21:53.076748+0000","last_became_active":"2024-05-08T21:21:52.090545+0000","last_became_peered":"2024-05-08T21:21:52.090545+0000","last_unstale":"2024-05-08T21:21:53.076748+0000","last_undegraded":"2024-05-08T21:21:53.076748+0000","last_fullsized":"2024-05-08T21:21:53.076748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:14:37.380612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073078+0000","last_change":"2024-05-08T21:21:52.079445+0000","last_active":"2024-05-08T21:21:53.073078+0000","last_peered":"2024-05-08T21:21:53.073078+0000","last_clean":"2024-05-08T21:21:53.073078+0000","last_became_active":"2024-05-08T21:21:52.079055+0000","last_became_peered":"2024-05-08T21:21:52.079055+0000","last_unstale":"2024-05-08T21:21:53.073078+0000","last_undegraded":"2024-05-08T21:21:53.073078+0000","last_fullsized":"2024-05-08T21:21:53.073078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:28:07.575847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073936+0000","last_change":"2024-05-08T21:21:51.616872+0000","last_active":"2024-05-08T21:21:52.073936+0000","last_peered":"2024-05-08T21:21:52.073936+0000","last_clean":"2024-05-08T21:21:52.073936+0000","last_became_active":"2024-05-08T21:21:51.616576+0000","last_became_peered":"2024-05-08T21:21:51.616576+0000","last_unstale":"2024-05-08T21:21:52.073936+0000","last_undegraded":"2024-05-08T21:21:52.073936+0000","last_fullsized":"2024-05-08T21:21:52.073936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:38:17.091010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076646+0000","last_change":"2024-05-08T21:21:52.079441+0000","last_active":"2024-05-08T21:21:53.076646+0000","last_peered":"2024-05-08T21:21:53.076646+0000","last_clean":"2024-05-08T21:21:53.076646+0000","last_became_active":"2024-05-08T21:21:52.079170+0000","last_became_peered":"2024-05-08T21:21:52.079170+0000","last_unstale":"2024-05-08T21:21:53.076646+0000","last_undegraded":"2024-05-08T21:21:53.076646+0000","last_fullsized":"2024-05-08T21:21:53.076646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:41.471400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.072713+0000","last_change":"2024-05-08T21:21:51.613447+0000","last_active":"2024-05-08T21:21:52.072713+0000","last_peered":"2024-05-08T21:21:52.072713+0000","last_clean":"2024-05-08T21:21:52.072713+0000","last_became_active":"2024-05-08T21:21:51.613150+0000","last_became_peered":"2024-05-08T21:21:51.613150+0000","last_unstale":"2024-05-08T21:21:52.072713+0000","last_undegraded":"2024-05-08T21:21:52.072713+0000","last_fullsized":"2024-05-08T21:21:52.072713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:13.563899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073528+0000","last_change":"2024-05-08T21:21:42.333097+0000","last_active":"2024-05-08T21:21:52.073528+0000","last_peered":"2024-05-08T21:21:52.073528+0000","last_clean":"2024-05-08T21:21:52.073528+0000","last_became_active":"2024-05-08T21:21:42.332033+0000","last_became_peered":"2024-05-08T21:21:42.332033+0000","last_unstale":"2024-05-08T21:21:52.073528+0000","last_undegraded":"2024-05-08T21:21:52.073528+0000","last_fullsized":"2024-05-08T21:21:52.073528+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_clean_scrub_stamp":"2024-05-08T21:21:40.862622+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:39:24.772551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602793+0000","last_change":"2024-05-08T21:21:52.096265+0000","last_active":"2024-05-08T21:21:55.602793+0000","last_peered":"2024-05-08T21:21:55.602793+0000","last_clean":"2024-05-08T21:21:55.602793+0000","last_became_active":"2024-05-08T21:21:52.095929+0000","last_became_peered":"2024-05-08T21:21:52.095929+0000","last_unstale":"2024-05-08T21:21:55.602793+0000","last_undegraded":"2024-05-08T21:21:55.602793+0000","last_fullsized":"2024-05-08T21:21:55.602793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:42.014816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073633+0000","last_change":"2024-05-08T21:21:51.614632+0000","last_active":"2024-05-08T21:21:52.073633+0000","last_peered":"2024-05-08T21:21:52.073633+0000","last_clean":"2024-05-08T21:21:52.073633+0000","last_became_active":"2024-05-08T21:21:51.614368+0000","last_became_peered":"2024-05-08T21:21:51.614368+0000","last_unstale":"2024-05-08T21:21:52.073633+0000","last_undegraded":"2024-05-08T21:21:52.073633+0000","last_fullsized":"2024-05-08T21:21:52.073633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:14:36.375522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.075513+0000","last_change":"2024-05-08T21:21:51.615266+0000","last_active":"2024-05-08T21:21:52.075513+0000","last_peered":"2024-05-08T21:21:52.075513+0000","last_clean":"2024-05-08T21:21:52.075513+0000","last_became_active":"2024-05-08T21:21:51.614949+0000","last_became_peered":"2024-05-08T21:21:51.614949+0000","last_unstale":"2024-05-08T21:21:52.075513+0000","last_undegraded":"2024-05-08T21:21:52.075513+0000","last_fullsized":"2024-05-08T21:21:52.075513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:44:53.582574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073133+0000","last_change":"2024-05-08T21:21:52.079810+0000","last_active":"2024-05-08T21:21:53.073133+0000","last_peered":"2024-05-08T21:21:53.073133+0000","last_clean":"2024-05-08T21:21:53.073133+0000","last_became_active":"2024-05-08T21:21:52.079291+0000","last_became_peered":"2024-05-08T21:21:52.079291+0000","last_unstale":"2024-05-08T21:21:53.073133+0000","last_undegraded":"2024-05-08T21:21:53.073133+0000","last_fullsized":"2024-05-08T21:21:53.073133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:41:45.740288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073242+0000","last_change":"2024-05-08T21:21:51.613365+0000","last_active":"2024-05-08T21:21:53.073242+0000","last_peered":"2024-05-08T21:21:53.073242+0000","last_clean":"2024-05-08T21:21:53.073242+0000","last_became_active":"2024-05-08T21:21:51.613022+0000","last_became_peered":"2024-05-08T21:21:51.613022+0000","last_unstale":"2024-05-08T21:21:53.073242+0000","last_undegraded":"2024-05-08T21:21:53.073242+0000","last_fullsized":"2024-05-08T21:21:53.073242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:14:36.375522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076917+0000","last_change":"2024-05-08T21:21:52.082814+0000","last_active":"2024-05-08T21:21:53.076917+0000","last_peered":"2024-05-08T21:21:53.076917+0000","last_clean":"2024-05-08T21:21:53.076917+0000","last_became_active":"2024-05-08T21:21:52.082539+0000","last_became_peered":"2024-05-08T21:21:52.082539+0000","last_unstale":"2024-05-08T21:21:53.076917+0000","last_undegraded":"2024-05-08T21:21:53.076917+0000","last_fullsized":"2024-05-08T21:21:53.076917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:36:48.359422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.093644+0000","last_change":"2024-05-08T21:21:51.612996+0000","last_active":"2024-05-08T21:21:52.093644+0000","last_peered":"2024-05-08T21:21:52.093644+0000","last_clean":"2024-05-08T21:21:52.093644+0000","last_became_active":"2024-05-08T21:21:51.612671+0000","last_became_peered":"2024-05-08T21:21:51.612671+0000","last_unstale":"2024-05-08T21:21:52.093644+0000","last_undegraded":"2024-05-08T21:21:52.093644+0000","last_fullsized":"2024-05-08T21:21:52.093644+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:44:53.582574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603051+0000","last_change":"2024-05-08T21:21:52.075455+0000","last_active":"2024-05-08T21:21:55.603051+0000","last_peered":"2024-05-08T21:21:55.603051+0000","last_clean":"2024-05-08T21:21:55.603051+0000","last_became_active":"2024-05-08T21:21:52.075082+0000","last_became_peered":"2024-05-08T21:21:52.075082+0000","last_unstale":"2024-05-08T21:21:55.603051+0000","last_undegraded":"2024-05-08T21:21:55.603051+0000","last_fullsized":"2024-05-08T21:21:55.603051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:02.395932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073416+0000","last_change":"2024-05-08T21:21:51.614333+0000","last_active":"2024-05-08T21:21:52.073416+0000","last_peered":"2024-05-08T21:21:52.073416+0000","last_clean":"2024-05-08T21:21:52.073416+0000","last_became_active":"2024-05-08T21:21:51.614048+0000","last_became_peered":"2024-05-08T21:21:51.614048+0000","last_unstale":"2024-05-08T21:21:52.073416+0000","last_undegraded":"2024-05-08T21:21:52.073416+0000","last_fullsized":"2024-05-08T21:21:52.073416+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:37:21.706277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076323+0000","last_change":"2024-05-08T21:21:52.094009+0000","last_active":"2024-05-08T21:21:53.076323+0000","last_peered":"2024-05-08T21:21:53.076323+0000","last_clean":"2024-05-08T21:21:53.076323+0000","last_became_active":"2024-05-08T21:21:52.093730+0000","last_became_peered":"2024-05-08T21:21:52.093730+0000","last_unstale":"2024-05-08T21:21:53.076323+0000","last_undegraded":"2024-05-08T21:21:53.076323+0000","last_fullsized":"2024-05-08T21:21:53.076323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:14.568989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.068928+0000","last_change":"2024-05-08T21:21:51.620291+0000","last_active":"2024-05-08T21:21:52.068928+0000","last_peered":"2024-05-08T21:21:52.068928+0000","last_clean":"2024-05-08T21:21:52.068928+0000","last_became_active":"2024-05-08T21:21:51.619965+0000","last_became_peered":"2024-05-08T21:21:51.619965+0000","last_unstale":"2024-05-08T21:21:52.068928+0000","last_undegraded":"2024-05-08T21:21:52.068928+0000","last_fullsized":"2024-05-08T21:21:52.068928+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:05:44.822440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076439+0000","last_change":"2024-05-08T21:21:52.086946+0000","last_active":"2024-05-08T21:21:53.076439+0000","last_peered":"2024-05-08T21:21:53.076439+0000","last_clean":"2024-05-08T21:21:53.076439+0000","last_became_active":"2024-05-08T21:21:52.086651+0000","last_became_peered":"2024-05-08T21:21:52.086651+0000","last_unstale":"2024-05-08T21:21:53.076439+0000","last_undegraded":"2024-05-08T21:21:53.076439+0000","last_fullsized":"2024-05-08T21:21:53.076439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:22:38.788235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.068205+0000","last_change":"2024-05-08T21:21:51.616253+0000","last_active":"2024-05-08T21:21:52.068205+0000","last_peered":"2024-05-08T21:21:52.068205+0000","last_clean":"2024-05-08T21:21:52.068205+0000","last_became_active":"2024-05-08T21:21:51.615883+0000","last_became_peered":"2024-05-08T21:21:51.615883+0000","last_unstale":"2024-05-08T21:21:52.068205+0000","last_undegraded":"2024-05-08T21:21:52.068205+0000","last_fullsized":"2024-05-08T21:21:52.068205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:13.563899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076873+0000","last_change":"2024-05-08T21:21:52.081159+0000","last_active":"2024-05-08T21:21:53.076873+0000","last_peered":"2024-05-08T21:21:53.076873+0000","last_clean":"2024-05-08T21:21:53.076873+0000","last_became_active":"2024-05-08T21:21:52.080823+0000","last_became_peered":"2024-05-08T21:21:52.080823+0000","last_unstale":"2024-05-08T21:21:53.076873+0000","last_undegraded":"2024-05-08T21:21:53.076873+0000","last_fullsized":"2024-05-08T21:21:53.076873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:41:45.740288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.100647+0000","last_change":"2024-05-08T21:21:51.609896+0000","last_active":"2024-05-08T21:21:52.100647+0000","last_peered":"2024-05-08T21:21:52.100647+0000","last_clean":"2024-05-08T21:21:52.100647+0000","last_became_active":"2024-05-08T21:21:51.609409+0000","last_became_peered":"2024-05-08T21:21:51.609409+0000","last_unstale":"2024-05-08T21:21:52.100647+0000","last_undegraded":"2024-05-08T21:21:52.100647+0000","last_fullsized":"2024-05-08T21:21:52.100647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:38:17.091010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603571+0000","last_change":"2024-05-08T21:21:52.101873+0000","last_active":"2024-05-08T21:21:55.603571+0000","last_peered":"2024-05-08T21:21:55.603571+0000","last_clean":"2024-05-08T21:21:55.603571+0000","last_became_active":"2024-05-08T21:21:52.101487+0000","last_became_peered":"2024-05-08T21:21:52.101487+0000","last_unstale":"2024-05-08T21:21:55.603571+0000","last_undegraded":"2024-05-08T21:21:55.603571+0000","last_fullsized":"2024-05-08T21:21:55.603571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:02.395932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.073673+0000","last_change":"2024-05-08T21:21:51.620697+0000","last_active":"2024-05-08T21:21:52.073673+0000","last_peered":"2024-05-08T21:21:52.073673+0000","last_clean":"2024-05-08T21:21:52.073673+0000","last_became_active":"2024-05-08T21:21:51.620370+0000","last_became_peered":"2024-05-08T21:21:51.620370+0000","last_unstale":"2024-05-08T21:21:52.073673+0000","last_undegraded":"2024-05-08T21:21:52.073673+0000","last_fullsized":"2024-05-08T21:21:52.073673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:00:55.530393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603751+0000","last_change":"2024-05-08T21:21:52.089908+0000","last_active":"2024-05-08T21:21:55.603751+0000","last_peered":"2024-05-08T21:21:55.603751+0000","last_clean":"2024-05-08T21:21:55.603751+0000","last_became_active":"2024-05-08T21:21:52.089610+0000","last_became_peered":"2024-05-08T21:21:52.089610+0000","last_unstale":"2024-05-08T21:21:55.603751+0000","last_undegraded":"2024-05-08T21:21:55.603751+0000","last_fullsized":"2024-05-08T21:21:55.603751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:26:01.523710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.089213+0000","last_change":"2024-05-08T21:21:51.610168+0000","last_active":"2024-05-08T21:21:52.089213+0000","last_peered":"2024-05-08T21:21:52.089213+0000","last_clean":"2024-05-08T21:21:52.089213+0000","last_became_active":"2024-05-08T21:21:51.609850+0000","last_became_peered":"2024-05-08T21:21:51.609850+0000","last_unstale":"2024-05-08T21:21:52.089213+0000","last_undegraded":"2024-05-08T21:21:52.089213+0000","last_fullsized":"2024-05-08T21:21:52.089213+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:41.009726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603664+0000","last_change":"2024-05-08T21:21:52.090014+0000","last_active":"2024-05-08T21:21:55.603664+0000","last_peered":"2024-05-08T21:21:55.603664+0000","last_clean":"2024-05-08T21:21:55.603664+0000","last_became_active":"2024-05-08T21:21:52.089630+0000","last_became_peered":"2024-05-08T21:21:52.089630+0000","last_unstale":"2024-05-08T21:21:55.603664+0000","last_undegraded":"2024-05-08T21:21:55.603664+0000","last_fullsized":"2024-05-08T21:21:55.603664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:26:01.523710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T21:21:52.088752+0000","last_change":"2024-05-08T21:21:51.613229+0000","last_active":"2024-05-08T21:21:52.088752+0000","last_peered":"2024-05-08T21:21:52.088752+0000","last_clean":"2024-05-08T21:21:52.088752+0000","last_became_active":"2024-05-08T21:21:51.612871+0000","last_became_peered":"2024-05-08T21:21:51.612871+0000","last_unstale":"2024-05-08T21:21:52.088752+0000","last_undegraded":"2024-05-08T21:21:52.088752+0000","last_fullsized":"2024-05-08T21:21:52.088752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:50.586265+0000","last_clean_scrub_stamp":"2024-05-08T21:21:50.586265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:45:34.421641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.604321+0000","last_change":"2024-05-08T21:21:52.097257+0000","last_active":"2024-05-08T21:21:55.604321+0000","last_peered":"2024-05-08T21:21:55.604321+0000","last_clean":"2024-05-08T21:21:55.604321+0000","last_became_active":"2024-05-08T21:21:52.096970+0000","last_became_peered":"2024-05-08T21:21:52.096970+0000","last_unstale":"2024-05-08T21:21:55.604321+0000","last_undegraded":"2024-05-08T21:21:55.604321+0000","last_fullsized":"2024-05-08T21:21:55.604321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:33:44.560880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602552+0000","last_change":"2024-05-08T21:21:52.089064+0000","last_active":"2024-05-08T21:21:55.602552+0000","last_peered":"2024-05-08T21:21:55.602552+0000","last_clean":"2024-05-08T21:21:55.602552+0000","last_became_active":"2024-05-08T21:21:52.088633+0000","last_became_peered":"2024-05-08T21:21:52.088633+0000","last_unstale":"2024-05-08T21:21:55.602552+0000","last_undegraded":"2024-05-08T21:21:55.602552+0000","last_fullsized":"2024-05-08T21:21:55.602552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:39:11.021535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602557+0000","last_change":"2024-05-08T21:21:52.087199+0000","last_active":"2024-05-08T21:21:55.602557+0000","last_peered":"2024-05-08T21:21:55.602557+0000","last_clean":"2024-05-08T21:21:55.602557+0000","last_became_active":"2024-05-08T21:21:52.086874+0000","last_became_peered":"2024-05-08T21:21:52.086874+0000","last_unstale":"2024-05-08T21:21:55.602557+0000","last_undegraded":"2024-05-08T21:21:55.602557+0000","last_fullsized":"2024-05-08T21:21:55.602557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:18:36.173550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603028+0000","last_change":"2024-05-08T21:21:52.090034+0000","last_active":"2024-05-08T21:21:55.603028+0000","last_peered":"2024-05-08T21:21:55.603028+0000","last_clean":"2024-05-08T21:21:55.603028+0000","last_became_active":"2024-05-08T21:21:52.089251+0000","last_became_peered":"2024-05-08T21:21:52.089251+0000","last_unstale":"2024-05-08T21:21:55.603028+0000","last_undegraded":"2024-05-08T21:21:55.603028+0000","last_fullsized":"2024-05-08T21:21:55.603028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:41.471400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603108+0000","last_change":"2024-05-08T21:21:52.085409+0000","last_active":"2024-05-08T21:21:55.603108+0000","last_peered":"2024-05-08T21:21:55.603108+0000","last_clean":"2024-05-08T21:21:55.603108+0000","last_became_active":"2024-05-08T21:21:52.085050+0000","last_became_peered":"2024-05-08T21:21:52.085050+0000","last_unstale":"2024-05-08T21:21:55.603108+0000","last_undegraded":"2024-05-08T21:21:55.603108+0000","last_fullsized":"2024-05-08T21:21:55.603108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:40:06.048132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076692+0000","last_change":"2024-05-08T21:21:52.100833+0000","last_active":"2024-05-08T21:21:53.076692+0000","last_peered":"2024-05-08T21:21:53.076692+0000","last_clean":"2024-05-08T21:21:53.076692+0000","last_became_active":"2024-05-08T21:21:52.100640+0000","last_became_peered":"2024-05-08T21:21:52.100640+0000","last_unstale":"2024-05-08T21:21:53.076692+0000","last_undegraded":"2024-05-08T21:21:53.076692+0000","last_fullsized":"2024-05-08T21:21:53.076692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:02.395932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.073296+0000","last_change":"2024-05-08T21:21:52.080879+0000","last_active":"2024-05-08T21:21:53.073296+0000","last_peered":"2024-05-08T21:21:53.073296+0000","last_clean":"2024-05-08T21:21:53.073296+0000","last_became_active":"2024-05-08T21:21:52.080592+0000","last_became_peered":"2024-05-08T21:21:52.080592+0000","last_unstale":"2024-05-08T21:21:53.073296+0000","last_undegraded":"2024-05-08T21:21:53.073296+0000","last_fullsized":"2024-05-08T21:21:53.073296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:39:11.021535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603171+0000","last_change":"2024-05-08T21:21:52.092091+0000","last_active":"2024-05-08T21:21:55.603171+0000","last_peered":"2024-05-08T21:21:55.603171+0000","last_clean":"2024-05-08T21:21:55.603171+0000","last_became_active":"2024-05-08T21:21:52.091489+0000","last_became_peered":"2024-05-08T21:21:52.091489+0000","last_unstale":"2024-05-08T21:21:55.603171+0000","last_undegraded":"2024-05-08T21:21:55.603171+0000","last_fullsized":"2024-05-08T21:21:55.603171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:38.782318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076796+0000","last_change":"2024-05-08T21:21:52.087595+0000","last_active":"2024-05-08T21:21:53.076796+0000","last_peered":"2024-05-08T21:21:53.076796+0000","last_clean":"2024-05-08T21:21:53.076796+0000","last_became_active":"2024-05-08T21:21:52.087302+0000","last_became_peered":"2024-05-08T21:21:52.087302+0000","last_unstale":"2024-05-08T21:21:53.076796+0000","last_undegraded":"2024-05-08T21:21:53.076796+0000","last_fullsized":"2024-05-08T21:21:53.076796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:40:06.048132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603267+0000","last_change":"2024-05-08T21:21:52.089760+0000","last_active":"2024-05-08T21:21:55.603267+0000","last_peered":"2024-05-08T21:21:55.603267+0000","last_clean":"2024-05-08T21:21:55.603267+0000","last_became_active":"2024-05-08T21:21:52.088814+0000","last_became_peered":"2024-05-08T21:21:52.088814+0000","last_unstale":"2024-05-08T21:21:55.603267+0000","last_undegraded":"2024-05-08T21:21:55.603267+0000","last_fullsized":"2024-05-08T21:21:55.603267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:04:56.104718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:53.076502+0000","last_change":"2024-05-08T21:21:52.089062+0000","last_active":"2024-05-08T21:21:53.076502+0000","last_peered":"2024-05-08T21:21:53.076502+0000","last_clean":"2024-05-08T21:21:53.076502+0000","last_became_active":"2024-05-08T21:21:52.088653+0000","last_became_peered":"2024-05-08T21:21:52.088653+0000","last_unstale":"2024-05-08T21:21:53.076502+0000","last_undegraded":"2024-05-08T21:21:53.076502+0000","last_fullsized":"2024-05-08T21:21:53.076502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:56:56.333047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602914+0000","last_change":"2024-05-08T21:21:52.096639+0000","last_active":"2024-05-08T21:21:55.602914+0000","last_peered":"2024-05-08T21:21:55.602914+0000","last_clean":"2024-05-08T21:21:55.602914+0000","last_became_active":"2024-05-08T21:21:52.096444+0000","last_became_peered":"2024-05-08T21:21:52.096444+0000","last_unstale":"2024-05-08T21:21:55.602914+0000","last_undegraded":"2024-05-08T21:21:55.602914+0000","last_fullsized":"2024-05-08T21:21:55.602914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:10:50.287200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.603006+0000","last_change":"2024-05-08T21:21:52.082022+0000","last_active":"2024-05-08T21:21:55.603006+0000","last_peered":"2024-05-08T21:21:55.603006+0000","last_clean":"2024-05-08T21:21:55.603006+0000","last_became_active":"2024-05-08T21:21:52.081764+0000","last_became_peered":"2024-05-08T21:21:52.081764+0000","last_unstale":"2024-05-08T21:21:55.603006+0000","last_undegraded":"2024-05-08T21:21:55.603006+0000","last_fullsized":"2024-05-08T21:21:55.603006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:22:38.788235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T21:21:55.602807+0000","last_change":"2024-05-08T21:21:52.101119+0000","last_active":"2024-05-08T21:21:55.602807+0000","last_peered":"2024-05-08T21:21:55.602807+0000","last_clean":"2024-05-08T21:21:55.602807+0000","last_became_active":"2024-05-08T21:21:52.100863+0000","last_became_peered":"2024-05-08T21:21:52.100863+0000","last_unstale":"2024-05-08T21:21:55.602807+0000","last_undegraded":"2024-05-08T21:21:55.602807+0000","last_fullsized":"2024-05-08T21:21:55.602807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:51.591355+0000","last_clean_scrub_stamp":"2024-05-08T21:21:51.591355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:40:06.048132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":661904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":70988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T21:22:01.422 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-08T21:22:01.422 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-08T21:22:01.422 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-08T21:22:01.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-08T21:22:01.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.595+0000 7f17c580b700 1 Processor -- start 2024-05-08T21:22:01.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.595+0000 7f17c580b700 1 -- start start 2024-05-08T21:22:01.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.595+0000 7f17c580b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17c015d780 0x7f17c00b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:01.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.595+0000 7f17c580b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f17c00d13e0 0x7f17c00d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:01.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.595+0000 7f17c580b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c00b4430 con 0x7f17c015db60 2024-05-08T21:22:01.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.595+0000 7f17c580b700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c00d45c0 con 0x7f17c00d13e0 2024-05-08T21:22:01.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.596+0000 7f17c580b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c00b3820 con 0x7f17c015d780 2024-05-08T21:22:01.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.596+0000 7f17bf7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f17c015db60 0x7f17c00ce3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58016/0 (socket says 172.21.15.3:58016) 2024-05-08T21:22:01.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.596+0000 7f17bf7fe700 1 -- 172.21.15.3:0/4089057156 learned_addr learned my addr 172.21.15.3:0/4089057156 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:01.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.596+0000 7f17be7fc700 1 --2- 172.21.15.3:0/4089057156 >> v2:172.21.15.186:3300/0 conn(0x7f17c00d13e0 0x7f17c00d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:01.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.596+0000 7f17beffd700 1 --2- 172.21.15.3:0/4089057156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17c015d780 0x7f17c00b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:01.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.596+0000 7f17bdffb700 1 -- 172.21.15.3:0/4089057156 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516188890 0 0) 0x7f17c00b4430 con 0x7f17c015db60 2024-05-08T21:22:01.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.596+0000 7f17bdffb700 1 -- 172.21.15.3:0/4089057156 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a80035b0 con 0x7f17c015db60 2024-05-08T21:22:01.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.596+0000 7f17be7fc700 1 -- 172.21.15.3:0/4089057156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17c015d780 msgr2=0x7f17c00b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:01.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.596+0000 7f17be7fc700 1 --2- 172.21.15.3:0/4089057156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17c015d780 0x7f17c00b48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.596+0000 7f17be7fc700 1 -- 172.21.15.3:0/4089057156 >> v1:172.21.15.3:6789/0 conn(0x7f17c015db60 legacy=0x7f17c00ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:01.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.596+0000 7f17be7fc700 1 -- 172.21.15.3:0/4089057156 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c00b3b00 con 0x7f17c00d13e0 2024-05-08T21:22:01.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.596+0000 7f17be7fc700 1 --2- 172.21.15.3:0/4089057156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17c00d13e0 0x7f17c00d1be0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009840 tx=0x7f17b4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5932723899094fec server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:01.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.597+0000 7f17bdffb700 1 -- 172.21.15.3:0/4089057156 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b4011830 con 0x7f17c00d13e0 2024-05-08T21:22:01.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.597+0000 7f17bdffb700 1 -- 172.21.15.3:0/4089057156 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b4011990 con 0x7f17c00d13e0 2024-05-08T21:22:01.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.597+0000 7f17bdffb700 1 -- 172.21.15.3:0/4089057156 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b4011d20 con 0x7f17c00d13e0 2024-05-08T21:22:01.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.597+0000 7f17c580b700 1 -- 172.21.15.3:0/4089057156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17c00d13e0 msgr2=0x7f17c00d1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:01.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.597+0000 7f17c580b700 1 --2- 172.21.15.3:0/4089057156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17c00d13e0 0x7f17c00d1be0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009840 tx=0x7f17b4009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:01.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.597+0000 7f17c580b700 1 -- 172.21.15.3:0/4089057156 shutdown_connections 2024-05-08T21:22:01.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.597+0000 7f17c580b700 1 --2- 172.21.15.3:0/4089057156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17c015d780 0x7f17c00b48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.597+0000 7f17c580b700 1 --2- 172.21.15.3:0/4089057156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17c00d13e0 0x7f17c00d1be0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.597+0000 7f17c580b700 1 -- 172.21.15.3:0/4089057156 >> 172.21.15.3:0/4089057156 conn(0x7f17c008afe0 msgr2=0x7f17c00d3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:01.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.597+0000 7f17c580b700 1 -- 172.21.15.3:0/4089057156 shutdown_connections 2024-05-08T21:22:01.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.597+0000 7f17c580b700 1 -- 172.21.15.3:0/4089057156 wait complete. 2024-05-08T21:22:01.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.599+0000 7f17c580b700 1 Processor -- start 2024-05-08T21:22:01.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.599+0000 7f17c580b700 1 -- start start 2024-05-08T21:22:01.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.600+0000 7f17c580b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17c00d13e0 0x7f17c014d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:01.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.600+0000 7f17c580b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c015d780 0x7f17c00c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:01.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.600+0000 7f17c580b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17c015db60 0x7f17c0158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:01.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.600+0000 7f17c580b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c014df10 con 0x7f17c015d780 2024-05-08T21:22:01.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.600+0000 7f17c580b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c0156ea0 con 0x7f17c015db60 2024-05-08T21:22:01.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.600+0000 7f17c580b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c0157990 con 0x7f17c00d13e0 2024-05-08T21:22:01.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.600+0000 7f17be7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c015d780 0x7f17c00c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:01.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.600+0000 7f17be7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c015d780 0x7f17c00c79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39656/0 (socket says 172.21.15.3:39656) 2024-05-08T21:22:01.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.600+0000 7f17be7fc700 1 -- 172.21.15.3:0/3784170838 learned_addr learned my addr 172.21.15.3:0/3784170838 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:01.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.601+0000 7f17beffd700 1 --2- 172.21.15.3:0/3784170838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17c00d13e0 0x7f17c014d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:01.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.601+0000 7f17bf7fe700 1 --2- 172.21.15.3:0/3784170838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17c015db60 0x7f17c0158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:01.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.601+0000 7f17be7fc700 1 -- 172.21.15.3:0/3784170838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17c00d13e0 msgr2=0x7f17c014d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:01.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.601+0000 7f17be7fc700 1 --2- 172.21.15.3:0/3784170838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17c00d13e0 0x7f17c014d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.601+0000 7f17be7fc700 1 -- 172.21.15.3:0/3784170838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17c015db60 msgr2=0x7f17c0158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:01.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.601+0000 7f17be7fc700 1 --2- 172.21.15.3:0/3784170838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17c015db60 0x7f17c0158a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.601+0000 7f17be7fc700 1 -- 172.21.15.3:0/3784170838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b40094a0 con 0x7f17c015d780 2024-05-08T21:22:01.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.601+0000 7f17beffd700 1 --2- 172.21.15.3:0/3784170838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17c00d13e0 0x7f17c014d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:01.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.601+0000 7f17be7fc700 1 --2- 172.21.15.3:0/3784170838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c015d780 0x7f17c00c79b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f17b4000c00 tx=0x7f17b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:01.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.601+0000 7f17a7fff700 1 -- 172.21.15.3:0/3784170838 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b40045d0 con 0x7f17c015d780 2024-05-08T21:22:01.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.601+0000 7f17a7fff700 1 -- 172.21.15.3:0/3784170838 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b401f070 con 0x7f17c015d780 2024-05-08T21:22:01.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.601+0000 7f17a7fff700 1 -- 172.21.15.3:0/3784170838 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b40038e0 con 0x7f17c015d780 2024-05-08T21:22:01.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.601+0000 7f17c580b700 1 -- 172.21.15.3:0/3784170838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c0157c70 con 0x7f17c015d780 2024-05-08T21:22:01.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.601+0000 7f17c580b700 1 -- 172.21.15.3:0/3784170838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c0158230 con 0x7f17c015d780 2024-05-08T21:22:01.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.602+0000 7f17c580b700 1 -- 172.21.15.3:0/3784170838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c00d1a50 con 0x7f17c015d780 2024-05-08T21:22:01.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.603+0000 7f17a7fff700 1 -- 172.21.15.3:0/3784170838 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f17b4003a40 con 0x7f17c015d780 2024-05-08T21:22:01.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.603+0000 7f17a7fff700 1 --2- 172.21.15.3:0/3784170838 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f17ac06ac90 0x7f17ac06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:01.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.603+0000 7f17a7fff700 1 -- 172.21.15.3:0/3784170838 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17b4021070 con 0x7f17c015d780 2024-05-08T21:22:01.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.603+0000 7f17beffd700 1 --2- 172.21.15.3:0/3784170838 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f17ac06ac90 0x7f17ac06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:01.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.604+0000 7f17beffd700 1 --2- 172.21.15.3:0/3784170838 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f17ac06ac90 0x7f17ac06d150 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f17b0009b40 tx=0x7f17b0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:01.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.608+0000 7f17a7fff700 1 -- 172.21.15.3:0/3784170838 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f17b4055db0 con 0x7f17c015d780 2024-05-08T21:22:01.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.877+0000 7f17c580b700 1 -- 172.21.15.3:0/3784170838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f17c00678f0 con 0x7f17c015d780 2024-05-08T21:22:01.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.877+0000 7f17a7fff700 1 -- 172.21.15.3:0/3784170838 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f17b402b040 con 0x7f17c015d780 2024-05-08T21:22:01.879 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:01.879 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-08T21:22:01.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.879+0000 7f17c580b700 1 -- 172.21.15.3:0/3784170838 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f17ac06ac90 msgr2=0x7f17ac06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:01.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.879+0000 7f17c580b700 1 --2- 172.21.15.3:0/3784170838 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f17ac06ac90 0x7f17ac06d150 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f17b0009b40 tx=0x7f17b0008040 comp rx=0 tx=0).stop 2024-05-08T21:22:01.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.879+0000 7f17c580b700 1 -- 172.21.15.3:0/3784170838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c015d780 msgr2=0x7f17c00c79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:01.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.879+0000 7f17c580b700 1 --2- 172.21.15.3:0/3784170838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c015d780 0x7f17c00c79b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f17b4000c00 tx=0x7f17b4004750 comp rx=0 tx=0).stop 2024-05-08T21:22:01.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.879+0000 7f17c580b700 1 -- 172.21.15.3:0/3784170838 shutdown_connections 2024-05-08T21:22:01.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.879+0000 7f17c580b700 1 --2- 172.21.15.3:0/3784170838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17c00d13e0 0x7f17c014d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.879+0000 7f17c580b700 1 --2- 172.21.15.3:0/3784170838 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f17ac06ac90 0x7f17ac06d150 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.879+0000 7f17c580b700 1 --2- 172.21.15.3:0/3784170838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17c015db60 0x7f17c0158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.879+0000 7f17c580b700 1 --2- 172.21.15.3:0/3784170838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c015d780 0x7f17c00c79b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:01.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.879+0000 7f17c580b700 1 -- 172.21.15.3:0/3784170838 >> 172.21.15.3:0/3784170838 conn(0x7f17c008afe0 msgr2=0x7f17c00b0cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:01.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.880+0000 7f17c580b700 1 -- 172.21.15.3:0/3784170838 shutdown_connections 2024-05-08T21:22:01.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:01.880+0000 7f17c580b700 1 -- 172.21.15.3:0/3784170838 wait complete. 2024-05-08T21:22:01.895 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-08T21:22:01.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:02.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.030+0000 7f03a1a4e700 1 Processor -- start 2024-05-08T21:22:02.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.030+0000 7f03a1a4e700 1 -- start start 2024-05-08T21:22:02.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.030+0000 7f03a1a4e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f039c099cc0 0x7f039c0d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.030+0000 7f03a1a4e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f039c09a0a0 0x7f039c0d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f03a1a4e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f039c15d710 0x7f039c15dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f03a1a4e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f039c06c820 con 0x7f039c15d710 2024-05-08T21:22:02.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f03a1a4e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f039c06bc10 con 0x7f039c099cc0 2024-05-08T21:22:02.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f03a1a4e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f039c06bef0 con 0x7f039c09a0a0 2024-05-08T21:22:02.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f039b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f039c15d710 0x7f039c15dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f039b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f039c15d710 0x7f039c15dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39658/0 (socket says 172.21.15.3:39658) 2024-05-08T21:22:02.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f039b7fe700 1 -- 172.21.15.3:0/279207589 learned_addr learned my addr 172.21.15.3:0/279207589 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:02.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f039affd700 1 --2- 172.21.15.3:0/279207589 >> v2:172.21.15.186:3300/0 conn(0x7f039c099cc0 0x7f039c0d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f039a7fc700 1 --2- 172.21.15.3:0/279207589 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f039c09a0a0 0x7f039c0d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f039b7fe700 1 -- 172.21.15.3:0/279207589 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f039c09a0a0 msgr2=0x7f039c0d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f039b7fe700 1 --2- 172.21.15.3:0/279207589 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f039c09a0a0 0x7f039c0d0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f039b7fe700 1 -- 172.21.15.3:0/279207589 >> v2:172.21.15.186:3300/0 conn(0x7f039c099cc0 msgr2=0x7f039c0d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f039b7fe700 1 --2- 172.21.15.3:0/279207589 >> v2:172.21.15.186:3300/0 conn(0x7f039c099cc0 0x7f039c0d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f039b7fe700 1 -- 172.21.15.3:0/279207589 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c06c1d0 con 0x7f039c15d710 2024-05-08T21:22:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f039affd700 1 --2- 172.21.15.3:0/279207589 >> v2:172.21.15.186:3300/0 conn(0x7f039c099cc0 0x7f039c0d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f039b7fe700 1 --2- 172.21.15.3:0/279207589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f039c15d710 0x7f039c15dba0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f039c0d2510 tx=0x7f039400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b461e0823c7bf5 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f0399ffb700 1 -- 172.21.15.3:0/279207589 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039400e070 con 0x7f039c15d710 2024-05-08T21:22:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f0399ffb700 1 -- 172.21.15.3:0/279207589 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039400b910 con 0x7f039c15d710 2024-05-08T21:22:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.031+0000 7f0399ffb700 1 -- 172.21.15.3:0/279207589 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039400bcc0 con 0x7f039c15d710 2024-05-08T21:22:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.032+0000 7f03a1a4e700 1 -- 172.21.15.3:0/279207589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f039c15d710 msgr2=0x7f039c15dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.032+0000 7f03a1a4e700 1 --2- 172.21.15.3:0/279207589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f039c15d710 0x7f039c15dba0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f039c0d2510 tx=0x7f039400b330 comp rx=0 tx=0).stop 2024-05-08T21:22:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.032+0000 7f03a1a4e700 1 -- 172.21.15.3:0/279207589 shutdown_connections 2024-05-08T21:22:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.032+0000 7f03a1a4e700 1 --2- 172.21.15.3:0/279207589 >> v2:172.21.15.186:3300/0 conn(0x7f039c099cc0 0x7f039c0d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.032+0000 7f03a1a4e700 1 --2- 172.21.15.3:0/279207589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f039c15d710 0x7f039c15dba0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.032+0000 7f03a1a4e700 1 --2- 172.21.15.3:0/279207589 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f039c09a0a0 0x7f039c0d0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.032+0000 7f03a1a4e700 1 -- 172.21.15.3:0/279207589 >> 172.21.15.3:0/279207589 conn(0x7f039c0b2f80 msgr2=0x7f039c0b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:02.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.032+0000 7f03a1a4e700 1 -- 172.21.15.3:0/279207589 shutdown_connections 2024-05-08T21:22:02.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.032+0000 7f03a1a4e700 1 -- 172.21.15.3:0/279207589 wait complete. 2024-05-08T21:22:02.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.034+0000 7f03a1a4e700 1 Processor -- start 2024-05-08T21:22:02.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.034+0000 7f03a1a4e700 1 -- start start 2024-05-08T21:22:02.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.035+0000 7f03a1a4e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f039c099cc0 0x7f039c15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.035+0000 7f03a1a4e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f039c09a0a0 0x7f039c0ce000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.035+0000 7f03a1a4e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f039c15d710 0x7f039c09ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.035+0000 7f03a1a4e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c15d000 con 0x7f039c09a0a0 2024-05-08T21:22:02.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.035+0000 7f03a1a4e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c0cc520 con 0x7f039c099cc0 2024-05-08T21:22:02.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.035+0000 7f03a1a4e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f039c0cc800 con 0x7f039c15d710 2024-05-08T21:22:02.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.035+0000 7f039a7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f039c09a0a0 0x7f039c0ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.035+0000 7f039a7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f039c09a0a0 0x7f039c0ce000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39674/0 (socket says 172.21.15.3:39674) 2024-05-08T21:22:02.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.035+0000 7f039a7fc700 1 -- 172.21.15.3:0/1052348806 learned_addr learned my addr 172.21.15.3:0/1052348806 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:02.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.035+0000 7f039affd700 1 --2- 172.21.15.3:0/1052348806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f039c099cc0 0x7f039c15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.035+0000 7f039b7fe700 1 --2- 172.21.15.3:0/1052348806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f039c15d710 0x7f039c09ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.036+0000 7f039a7fc700 1 -- 172.21.15.3:0/1052348806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f039c15d710 msgr2=0x7f039c09ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:02.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.036+0000 7f039a7fc700 1 --2- 172.21.15.3:0/1052348806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f039c15d710 0x7f039c09ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.036+0000 7f039a7fc700 1 -- 172.21.15.3:0/1052348806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f039c099cc0 msgr2=0x7f039c15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:02.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.036+0000 7f039a7fc700 1 --2- 172.21.15.3:0/1052348806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f039c099cc0 0x7f039c15cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.036+0000 7f039a7fc700 1 -- 172.21.15.3:0/1052348806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038c0094a0 con 0x7f039c09a0a0 2024-05-08T21:22:02.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.036+0000 7f039affd700 1 --2- 172.21.15.3:0/1052348806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f039c099cc0 0x7f039c15cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:02.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.036+0000 7f039a7fc700 1 --2- 172.21.15.3:0/1052348806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f039c09a0a0 0x7f039c0ce000 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f039000e9d0 tx=0x7f039000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:02.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.036+0000 7f0383fff700 1 -- 172.21.15.3:0/1052348806 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0390009730 con 0x7f039c09a0a0 2024-05-08T21:22:02.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.036+0000 7f0383fff700 1 -- 172.21.15.3:0/1052348806 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039000ee70 con 0x7f039c09a0a0 2024-05-08T21:22:02.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.036+0000 7f03a1a4e700 1 -- 172.21.15.3:0/1052348806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0394009b30 con 0x7f039c09a0a0 2024-05-08T21:22:02.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.036+0000 7f0383fff700 1 -- 172.21.15.3:0/1052348806 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039000ca70 con 0x7f039c09a0a0 2024-05-08T21:22:02.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.036+0000 7f03a1a4e700 1 -- 172.21.15.3:0/1052348806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f039c0ccdc0 con 0x7f039c09a0a0 2024-05-08T21:22:02.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.037+0000 7f03a1a4e700 1 -- 172.21.15.3:0/1052348806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f039c0678f0 con 0x7f039c09a0a0 2024-05-08T21:22:02.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.037+0000 7f0383fff700 1 -- 172.21.15.3:0/1052348806 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f039000cbd0 con 0x7f039c09a0a0 2024-05-08T21:22:02.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.038+0000 7f0383fff700 1 --2- 172.21.15.3:0/1052348806 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f038806ac90 0x7f038806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.038+0000 7f0383fff700 1 -- 172.21.15.3:0/1052348806 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0390089310 con 0x7f039c09a0a0 2024-05-08T21:22:02.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.038+0000 7f039affd700 1 --2- 172.21.15.3:0/1052348806 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f038806ac90 0x7f038806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.039+0000 7f039affd700 1 --2- 172.21.15.3:0/1052348806 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f038806ac90 0x7f038806d150 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f038c009470 tx=0x7f038c009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:02.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.042+0000 7f0383fff700 1 -- 172.21.15.3:0/1052348806 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f03900544f0 con 0x7f039c09a0a0 2024-05-08T21:22:02.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.262+0000 7f03a1a4e700 1 -- 172.21.15.3:0/1052348806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f039c0b38c0 con 0x7f039c09a0a0 2024-05-08T21:22:02.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.263+0000 7f0383fff700 1 -- 172.21.15.3:0/1052348806 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15989 (secure 0 0 0) 0x7f0390057b10 con 0x7f039c09a0a0 2024-05-08T21:22:02.265 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:02.265 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:53.064342+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:02.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.265+0000 7f03a1a4e700 1 -- 172.21.15.3:0/1052348806 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f038806ac90 msgr2=0x7f038806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:02.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.265+0000 7f03a1a4e700 1 --2- 172.21.15.3:0/1052348806 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f038806ac90 0x7f038806d150 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f038c009470 tx=0x7f038c009210 comp rx=0 tx=0).stop 2024-05-08T21:22:02.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.265+0000 7f03a1a4e700 1 -- 172.21.15.3:0/1052348806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f039c09a0a0 msgr2=0x7f039c0ce000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:02.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.265+0000 7f03a1a4e700 1 --2- 172.21.15.3:0/1052348806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f039c09a0a0 0x7f039c0ce000 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f039000e9d0 tx=0x7f039000ece0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.265+0000 7f03a1a4e700 1 -- 172.21.15.3:0/1052348806 shutdown_connections 2024-05-08T21:22:02.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.265+0000 7f03a1a4e700 1 --2- 172.21.15.3:0/1052348806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f039c15d710 0x7f039c09ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.265+0000 7f03a1a4e700 1 --2- 172.21.15.3:0/1052348806 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f038806ac90 0x7f038806d150 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.265+0000 7f03a1a4e700 1 --2- 172.21.15.3:0/1052348806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f039c099cc0 0x7f039c15cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.265+0000 7f03a1a4e700 1 --2- 172.21.15.3:0/1052348806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f039c09a0a0 0x7f039c0ce000 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.265+0000 7f03a1a4e700 1 -- 172.21.15.3:0/1052348806 >> 172.21.15.3:0/1052348806 conn(0x7f039c0b2f80 msgr2=0x7f039c0c9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:02.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.266+0000 7f03a1a4e700 1 -- 172.21.15.3:0/1052348806 shutdown_connections 2024-05-08T21:22:02.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.266+0000 7f03a1a4e700 1 -- 172.21.15.3:0/1052348806 wait complete. 2024-05-08T21:22:02.281 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-08T21:21:49.638998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:21:51.024632+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:22:02.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:22:02.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.419+0000 7f5279675700 1 Processor -- start 2024-05-08T21:22:02.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.419+0000 7f5279675700 1 -- start start 2024-05-08T21:22:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.420+0000 7f5279675700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f527415dde0 0x7f52740cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.420+0000 7f5279675700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f52740d2500 0x7f52740d04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.420+0000 7f5279675700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52740ceef0 con 0x7f52740b4c80 2024-05-08T21:22:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.420+0000 7f5279675700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f527415e1c0 con 0x7f52740d2500 2024-05-08T21:22:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.420+0000 7f5279675700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52740cf140 con 0x7f527415dde0 2024-05-08T21:22:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.420+0000 7f5272ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f527415dde0 0x7f52740cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.420+0000 7f52737fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f52740d2500 0x7f52740d04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.421+0000 7f52727fc700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f52740b4c80 0x7f52740b2e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:51382/0 (socket says 172.21.15.3:51382) 2024-05-08T21:22:02.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.421+0000 7f52727fc700 1 -- 172.21.15.3:0/2699456270 learned_addr learned my addr 172.21.15.3:0/2699456270 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:02.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.421+0000 7f5271ffb700 1 -- 172.21.15.3:0/2699456270 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494493179 0 0) 0x7f52740ceef0 con 0x7f52740b4c80 2024-05-08T21:22:02.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.421+0000 7f5271ffb700 1 -- 172.21.15.3:0/2699456270 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52600035b0 con 0x7f52740b4c80 2024-05-08T21:22:02.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.421+0000 7f52737fe700 1 -- 172.21.15.3:0/2699456270 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f527415dde0 msgr2=0x7f52740cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.421+0000 7f52737fe700 1 --2- 172.21.15.3:0/2699456270 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f527415dde0 0x7f52740cffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.421+0000 7f52737fe700 1 -- 172.21.15.3:0/2699456270 >> v1:172.21.15.186:6789/0 conn(0x7f52740b4c80 legacy=0x7f52740b2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.421+0000 7f52737fe700 1 -- 172.21.15.3:0/2699456270 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52740cf420 con 0x7f52740d2500 2024-05-08T21:22:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.421+0000 7f5272ffd700 1 --2- 172.21.15.3:0/2699456270 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f527415dde0 0x7f52740cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.421+0000 7f52737fe700 1 --2- 172.21.15.3:0/2699456270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52740d2500 0x7f52740d04e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f526c009d90 tx=0x7f526c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5768729836d689d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.421+0000 7f5271ffb700 1 -- 172.21.15.3:0/2699456270 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c00e070 con 0x7f52740d2500 2024-05-08T21:22:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.421+0000 7f5271ffb700 1 -- 172.21.15.3:0/2699456270 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526c00bb20 con 0x7f52740d2500 2024-05-08T21:22:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.421+0000 7f5271ffb700 1 -- 172.21.15.3:0/2699456270 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c012430 con 0x7f52740d2500 2024-05-08T21:22:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.422+0000 7f5279675700 1 -- 172.21.15.3:0/2699456270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52740d2500 msgr2=0x7f52740d04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.422+0000 7f5279675700 1 --2- 172.21.15.3:0/2699456270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52740d2500 0x7f52740d04e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f526c009d90 tx=0x7f526c00b330 comp rx=0 tx=0).stop 2024-05-08T21:22:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.422+0000 7f5279675700 1 -- 172.21.15.3:0/2699456270 shutdown_connections 2024-05-08T21:22:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.422+0000 7f5279675700 1 --2- 172.21.15.3:0/2699456270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52740d2500 0x7f52740d04e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.422+0000 7f5279675700 1 --2- 172.21.15.3:0/2699456270 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f527415dde0 0x7f52740cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.422+0000 7f5279675700 1 -- 172.21.15.3:0/2699456270 >> 172.21.15.3:0/2699456270 conn(0x7f527408afe0 msgr2=0x7f52740d35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.422+0000 7f5279675700 1 -- 172.21.15.3:0/2699456270 shutdown_connections 2024-05-08T21:22:02.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.422+0000 7f5279675700 1 -- 172.21.15.3:0/2699456270 wait complete. 2024-05-08T21:22:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.423+0000 7f5279675700 1 Processor -- start 2024-05-08T21:22:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.424+0000 7f5279675700 1 -- start start 2024-05-08T21:22:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.424+0000 7f5279675700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52740b4c80 0x7f52740cd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.424+0000 7f5279675700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52740d2500 0x7f527406bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.424+0000 7f5279675700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f527415dde0 0x7f52740cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.424+0000 7f5279675700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52740ccbe0 con 0x7f527415dde0 2024-05-08T21:22:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.424+0000 7f5279675700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52740cbf60 con 0x7f52740b4c80 2024-05-08T21:22:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.424+0000 7f5279675700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52740cc240 con 0x7f52740d2500 2024-05-08T21:22:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.424+0000 7f52737fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f527415dde0 0x7f52740cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.424+0000 7f52737fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f527415dde0 0x7f52740cb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39690/0 (socket says 172.21.15.3:39690) 2024-05-08T21:22:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.424+0000 7f52737fe700 1 -- 172.21.15.3:0/715314057 learned_addr learned my addr 172.21.15.3:0/715314057 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.425+0000 7f52727fc700 1 --2- 172.21.15.3:0/715314057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52740d2500 0x7f527406bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.425+0000 7f5272ffd700 1 --2- 172.21.15.3:0/715314057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52740b4c80 0x7f52740cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.425+0000 7f52737fe700 1 -- 172.21.15.3:0/715314057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52740d2500 msgr2=0x7f527406bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.425+0000 7f52737fe700 1 --2- 172.21.15.3:0/715314057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52740d2500 0x7f527406bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.425+0000 7f52737fe700 1 -- 172.21.15.3:0/715314057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52740b4c80 msgr2=0x7f52740cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.425+0000 7f52737fe700 1 --2- 172.21.15.3:0/715314057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52740b4c80 0x7f52740cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.425+0000 7f52737fe700 1 -- 172.21.15.3:0/715314057 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52640094a0 con 0x7f527415dde0 2024-05-08T21:22:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.425+0000 7f52727fc700 1 --2- 172.21.15.3:0/715314057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52740d2500 0x7f527406bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.425+0000 7f5272ffd700 1 --2- 172.21.15.3:0/715314057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52740b4c80 0x7f52740cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.425+0000 7f52737fe700 1 --2- 172.21.15.3:0/715314057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f527415dde0 0x7f52740cb4a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f526c00b370 tx=0x7f526c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.425+0000 7f526b7fe700 1 -- 172.21.15.3:0/715314057 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c012430 con 0x7f527415dde0 2024-05-08T21:22:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.425+0000 7f526b7fe700 1 -- 172.21.15.3:0/715314057 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526c020070 con 0x7f527415dde0 2024-05-08T21:22:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.425+0000 7f5279675700 1 -- 172.21.15.3:0/715314057 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f526c009a70 con 0x7f527415dde0 2024-05-08T21:22:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.425+0000 7f526b7fe700 1 -- 172.21.15.3:0/715314057 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c004280 con 0x7f527415dde0 2024-05-08T21:22:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.425+0000 7f5279675700 1 -- 172.21.15.3:0/715314057 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52740cc800 con 0x7f527415dde0 2024-05-08T21:22:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.426+0000 7f5279675700 1 -- 172.21.15.3:0/715314057 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5274074bc0 con 0x7f527415dde0 2024-05-08T21:22:02.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.427+0000 7f526b7fe700 1 -- 172.21.15.3:0/715314057 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f526c0043e0 con 0x7f527415dde0 2024-05-08T21:22:02.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.427+0000 7f526b7fe700 1 --2- 172.21.15.3:0/715314057 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f525406ac90 0x7f525406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.427+0000 7f526b7fe700 1 -- 172.21.15.3:0/715314057 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f526c022070 con 0x7f527415dde0 2024-05-08T21:22:02.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.427+0000 7f5272ffd700 1 --2- 172.21.15.3:0/715314057 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f525406ac90 0x7f525406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.428+0000 7f5272ffd700 1 --2- 172.21.15.3:0/715314057 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f525406ac90 0x7f525406d150 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5264009d20 tx=0x7f5264008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:02.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.432+0000 7f526b7fe700 1 -- 172.21.15.3:0/715314057 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f526c056600 con 0x7f527415dde0 2024-05-08T21:22:02.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.650+0000 7f5279675700 1 -- 172.21.15.3:0/715314057 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f52740b2450 con 0x7f527415dde0 2024-05-08T21:22:02.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.650+0000 7f526b7fe700 1 -- 172.21.15.3:0/715314057 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f526c01ae60 con 0x7f527415dde0 2024-05-08T21:22:02.652 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:22:02.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.653+0000 7f5279675700 1 -- 172.21.15.3:0/715314057 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f525406ac90 msgr2=0x7f525406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:02.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.653+0000 7f5279675700 1 --2- 172.21.15.3:0/715314057 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f525406ac90 0x7f525406d150 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5264009d20 tx=0x7f5264008040 comp rx=0 tx=0).stop 2024-05-08T21:22:02.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.653+0000 7f5279675700 1 -- 172.21.15.3:0/715314057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f527415dde0 msgr2=0x7f52740cb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:02.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.653+0000 7f5279675700 1 --2- 172.21.15.3:0/715314057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f527415dde0 0x7f52740cb4a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f526c00b370 tx=0x7f526c004ad0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.653+0000 7f5279675700 1 -- 172.21.15.3:0/715314057 shutdown_connections 2024-05-08T21:22:02.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.653+0000 7f5279675700 1 --2- 172.21.15.3:0/715314057 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f525406ac90 0x7f525406d150 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.653+0000 7f5279675700 1 --2- 172.21.15.3:0/715314057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f527415dde0 0x7f52740cb4a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.653+0000 7f5279675700 1 --2- 172.21.15.3:0/715314057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52740b4c80 0x7f52740cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.653+0000 7f5279675700 1 --2- 172.21.15.3:0/715314057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52740d2500 0x7f527406bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.653+0000 7f5279675700 1 -- 172.21.15.3:0/715314057 >> 172.21.15.3:0/715314057 conn(0x7f527408afe0 msgr2=0x7f527415caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:02.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.653+0000 7f5279675700 1 -- 172.21.15.3:0/715314057 shutdown_connections 2024-05-08T21:22:02.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.653+0000 7f5279675700 1 -- 172.21.15.3:0/715314057 wait complete. 2024-05-08T21:22:02.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-08T21:22:02.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.804+0000 7fad60e23700 1 Processor -- start 2024-05-08T21:22:02.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.804+0000 7fad60e23700 1 -- start start 2024-05-08T21:22:02.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.805+0000 7fad60e23700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fad5c09a160 0x7fad5c168570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.805+0000 7fad60e23700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5c06d010 con 0x7fad5c09a790 2024-05-08T21:22:02.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.805+0000 7fad60e23700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5c15de40 con 0x7fad5c15da60 2024-05-08T21:22:02.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.805+0000 7fad60e23700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad5c0d0b20 con 0x7fad5c09a160 2024-05-08T21:22:02.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.806+0000 7fad5ad9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fad5c09a160 0x7fad5c168570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.806+0000 7fad59d9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fad5c09a790 0x7fad5c099a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58024/0 (socket says 172.21.15.3:58024) 2024-05-08T21:22:02.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.806+0000 7fad59d9b700 1 -- 172.21.15.3:0/3343997314 learned_addr learned my addr 172.21.15.3:0/3343997314 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:02.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.806+0000 7fad5959a700 1 -- 172.21.15.3:0/3343997314 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315197844 0 0) 0x7fad5c06d010 con 0x7fad5c09a790 2024-05-08T21:22:02.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.806+0000 7fad5959a700 1 -- 172.21.15.3:0/3343997314 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad440035b0 con 0x7fad5c09a790 2024-05-08T21:22:02.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.806+0000 7fad5ad9d700 1 -- 172.21.15.3:0/3343997314 >> v1:172.21.15.186:6789/0 conn(0x7fad5c15da60 legacy=0x7fad5c0d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:02.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.806+0000 7fad5959a700 1 -- 172.21.15.3:0/3343997314 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556932734 0 0) 0x7fad5c15de40 con 0x7fad5c15da60 2024-05-08T21:22:02.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.806+0000 7fad5ad9d700 1 -- 172.21.15.3:0/3343997314 >> v1:172.21.15.3:6789/0 conn(0x7fad5c09a790 legacy=0x7fad5c099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:02.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.806+0000 7fad5ad9d700 1 -- 172.21.15.3:0/3343997314 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5c0cbc40 con 0x7fad5c09a160 2024-05-08T21:22:02.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.806+0000 7fad5ad9d700 1 --2- 172.21.15.3:0/3343997314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad5c09a160 0x7fad5c168570 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fad54009d90 tx=0x7fad5400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f496980e435b2cb7 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:02.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.806+0000 7fad5959a700 1 -- 172.21.15.3:0/3343997314 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5400e070 con 0x7fad5c09a160 2024-05-08T21:22:02.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.806+0000 7fad5959a700 1 -- 172.21.15.3:0/3343997314 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5400bb20 con 0x7fad5c09a160 2024-05-08T21:22:02.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.806+0000 7fad5959a700 1 -- 172.21.15.3:0/3343997314 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5400bed0 con 0x7fad5c09a160 2024-05-08T21:22:02.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.807+0000 7fad60e23700 1 -- 172.21.15.3:0/3343997314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad5c09a160 msgr2=0x7fad5c168570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:02.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.807+0000 7fad60e23700 1 --2- 172.21.15.3:0/3343997314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad5c09a160 0x7fad5c168570 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fad54009d90 tx=0x7fad5400b330 comp rx=0 tx=0).stop 2024-05-08T21:22:02.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.807+0000 7fad60e23700 1 -- 172.21.15.3:0/3343997314 shutdown_connections 2024-05-08T21:22:02.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.807+0000 7fad60e23700 1 --2- 172.21.15.3:0/3343997314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad5c09a160 0x7fad5c168570 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.807+0000 7fad60e23700 1 -- 172.21.15.3:0/3343997314 >> 172.21.15.3:0/3343997314 conn(0x7fad5c0b2f60 msgr2=0x7fad5c0b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:02.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.807+0000 7fad60e23700 1 -- 172.21.15.3:0/3343997314 shutdown_connections 2024-05-08T21:22:02.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.807+0000 7fad60e23700 1 -- 172.21.15.3:0/3343997314 wait complete. 2024-05-08T21:22:02.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.809+0000 7fad60e23700 1 Processor -- start 2024-05-08T21:22:02.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.809+0000 7fad60e23700 1 -- start start 2024-05-08T21:22:02.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.809+0000 7fad60e23700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad5c09a160 0x7fad5c0b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.809+0000 7fad60e23700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad5c09a790 0x7fad5c1f2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.809+0000 7fad60e23700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad5c15da60 0x7fad5c06c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.809+0000 7fad60e23700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c0b7240 con 0x7fad5c15da60 2024-05-08T21:22:02.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.809+0000 7fad60e23700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c0b6c80 con 0x7fad5c09a790 2024-05-08T21:22:02.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.809+0000 7fad60e23700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c0b6f60 con 0x7fad5c09a160 2024-05-08T21:22:02.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad5ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad5c15da60 0x7fad5c06c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad5ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad5c15da60 0x7fad5c06c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39700/0 (socket says 172.21.15.3:39700) 2024-05-08T21:22:02.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad5ad9d700 1 -- 172.21.15.3:0/2905728023 learned_addr learned my addr 172.21.15.3:0/2905728023 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:02.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad59d9b700 1 --2- 172.21.15.3:0/2905728023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad5c09a790 0x7fad5c1f2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad5a59c700 1 --2- 172.21.15.3:0/2905728023 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad5c09a160 0x7fad5c0b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad5ad9d700 1 -- 172.21.15.3:0/2905728023 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad5c09a160 msgr2=0x7fad5c0b62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad5ad9d700 1 --2- 172.21.15.3:0/2905728023 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad5c09a160 0x7fad5c0b62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad5ad9d700 1 -- 172.21.15.3:0/2905728023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad5c09a790 msgr2=0x7fad5c1f2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad5ad9d700 1 --2- 172.21.15.3:0/2905728023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad5c09a790 0x7fad5c1f2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad5ad9d700 1 -- 172.21.15.3:0/2905728023 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad54009a70 con 0x7fad5c15da60 2024-05-08T21:22:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad59d9b700 1 --2- 172.21.15.3:0/2905728023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad5c09a790 0x7fad5c1f2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad5ad9d700 1 --2- 172.21.15.3:0/2905728023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad5c15da60 0x7fad5c06c1a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fad54014040 tx=0x7fad54004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad4affd700 1 -- 172.21.15.3:0/2905728023 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5401b810 con 0x7fad5c15da60 2024-05-08T21:22:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad4affd700 1 -- 172.21.15.3:0/2905728023 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad54020070 con 0x7fad5c15da60 2024-05-08T21:22:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad60e23700 1 -- 172.21.15.3:0/2905728023 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5c15e1d0 con 0x7fad5c15da60 2024-05-08T21:22:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad4affd700 1 -- 172.21.15.3:0/2905728023 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5401bbb0 con 0x7fad5c15da60 2024-05-08T21:22:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.810+0000 7fad60e23700 1 -- 172.21.15.3:0/2905728023 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5c0b6820 con 0x7fad5c15da60 2024-05-08T21:22:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.811+0000 7fad60e23700 1 -- 172.21.15.3:0/2905728023 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5c15e030 con 0x7fad5c15da60 2024-05-08T21:22:02.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.812+0000 7fad4affd700 1 -- 172.21.15.3:0/2905728023 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fad5401bd10 con 0x7fad5c15da60 2024-05-08T21:22:02.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.812+0000 7fad4affd700 1 --2- 172.21.15.3:0/2905728023 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad3c06ac90 0x7fad3c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:02.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.812+0000 7fad4affd700 1 -- 172.21.15.3:0/2905728023 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad54022070 con 0x7fad5c15da60 2024-05-08T21:22:02.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.813+0000 7fad5a59c700 1 --2- 172.21.15.3:0/2905728023 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad3c06ac90 0x7fad3c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:02.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.813+0000 7fad5a59c700 1 --2- 172.21.15.3:0/2905728023 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad3c06ac90 0x7fad3c06d150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fad4c000f60 tx=0x7fad4c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:02.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:02.817+0000 7fad4affd700 1 -- 172.21.15.3:0/2905728023 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fad5c15e030 con 0x7fad5c15da60 2024-05-08T21:22:03.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.033+0000 7fad60e23700 1 -- 172.21.15.3:0/2905728023 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fad5c15e6f0 con 0x7fad5c15da60 2024-05-08T21:22:03.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.034+0000 7fad4affd700 1 -- 172.21.15.3:0/2905728023 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fad5408b020 con 0x7fad5c15da60 2024-05-08T21:22:03.035 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:22:03.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.036+0000 7fad60e23700 1 -- 172.21.15.3:0/2905728023 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad3c06ac90 msgr2=0x7fad3c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:03.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.036+0000 7fad60e23700 1 --2- 172.21.15.3:0/2905728023 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad3c06ac90 0x7fad3c06d150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fad4c000f60 tx=0x7fad4c009040 comp rx=0 tx=0).stop 2024-05-08T21:22:03.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.036+0000 7fad60e23700 1 -- 172.21.15.3:0/2905728023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad5c15da60 msgr2=0x7fad5c06c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:03.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.036+0000 7fad60e23700 1 --2- 172.21.15.3:0/2905728023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad5c15da60 0x7fad5c06c1a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fad54014040 tx=0x7fad54004ad0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.036+0000 7fad60e23700 1 -- 172.21.15.3:0/2905728023 shutdown_connections 2024-05-08T21:22:03.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.036+0000 7fad60e23700 1 --2- 172.21.15.3:0/2905728023 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad5c09a160 0x7fad5c0b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.036+0000 7fad60e23700 1 --2- 172.21.15.3:0/2905728023 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad3c06ac90 0x7fad3c06d150 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.036+0000 7fad60e23700 1 --2- 172.21.15.3:0/2905728023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad5c15da60 0x7fad5c06c1a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.036+0000 7fad60e23700 1 --2- 172.21.15.3:0/2905728023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad5c09a790 0x7fad5c1f2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.036+0000 7fad60e23700 1 -- 172.21.15.3:0/2905728023 >> 172.21.15.3:0/2905728023 conn(0x7fad5c0b2f60 msgr2=0x7fad5c06ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:03.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.036+0000 7fad60e23700 1 -- 172.21.15.3:0/2905728023 shutdown_connections 2024-05-08T21:22:03.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.036+0000 7fad60e23700 1 -- 172.21.15.3:0/2905728023 wait complete. 2024-05-08T21:22:03.052 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:22:03.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.189+0000 7f2435ec5700 1 Processor -- start 2024-05-08T21:22:03.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.189+0000 7f2435ec5700 1 -- start start 2024-05-08T21:22:03.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.190+0000 7f2435ec5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24300b5010 0x7f243015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:03.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.190+0000 7f2435ec5700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f24300b53f0 0x7f243015f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:03.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.190+0000 7f2435ec5700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f243006d700 con 0x7f243015f660 2024-05-08T21:22:03.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.190+0000 7f2435ec5700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f243015cab0 con 0x7f24300b53f0 2024-05-08T21:22:03.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.190+0000 7f2435ec5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24300cdcf0 con 0x7f24300b5010 2024-05-08T21:22:03.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.190+0000 7f242effd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f24300b53f0 0x7f243015f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:03.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.190+0000 7f242ffff700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f243015f660 0x7f243006db90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:51408/0 (socket says 172.21.15.3:51408) 2024-05-08T21:22:03.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.190+0000 7f242f7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24300b5010 0x7f243015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:03.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.190+0000 7f242ffff700 1 -- 172.21.15.3:0/1347008049 learned_addr learned my addr 172.21.15.3:0/1347008049 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:03.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.190+0000 7f242effd700 1 --2- 172.21.15.3:0/1347008049 >> v2:172.21.15.186:3300/0 conn(0x7f24300b53f0 0x7f243015f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:57934/0 (socket says 172.21.15.3:57934) 2024-05-08T21:22:03.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.191+0000 7f242effd700 1 -- 172.21.15.3:0/1347008049 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24300b5010 msgr2=0x7f243015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:03.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.191+0000 7f242effd700 1 --2- 172.21.15.3:0/1347008049 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24300b5010 0x7f243015c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.191+0000 7f242effd700 1 -- 172.21.15.3:0/1347008049 >> v1:172.21.15.186:6789/0 conn(0x7f243015f660 legacy=0x7f243006db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:03.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.191+0000 7f242effd700 1 -- 172.21.15.3:0/1347008049 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24300cdfd0 con 0x7f24300b53f0 2024-05-08T21:22:03.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.191+0000 7f242e7fc700 1 -- 172.21.15.3:0/1347008049 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363231651 0 0) 0x7f243006d700 con 0x7f243015f660 2024-05-08T21:22:03.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.191+0000 7f242f7fe700 1 --2- 172.21.15.3:0/1347008049 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24300b5010 0x7f243015c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:03.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.191+0000 7f242effd700 1 --2- 172.21.15.3:0/1347008049 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24300b53f0 0x7f243015f120 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2424009840 tx=0x7f2424009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a709a4a43821c67 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:03.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.191+0000 7f242e7fc700 1 -- 172.21.15.3:0/1347008049 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424011830 con 0x7f24300b53f0 2024-05-08T21:22:03.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.191+0000 7f242e7fc700 1 -- 172.21.15.3:0/1347008049 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2424011990 con 0x7f24300b53f0 2024-05-08T21:22:03.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.191+0000 7f242e7fc700 1 -- 172.21.15.3:0/1347008049 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424011d20 con 0x7f24300b53f0 2024-05-08T21:22:03.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.192+0000 7f2435ec5700 1 -- 172.21.15.3:0/1347008049 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24300b53f0 msgr2=0x7f243015f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:03.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.192+0000 7f2435ec5700 1 --2- 172.21.15.3:0/1347008049 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24300b53f0 0x7f243015f120 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2424009840 tx=0x7f2424009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:03.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.192+0000 7f2435ec5700 1 -- 172.21.15.3:0/1347008049 shutdown_connections 2024-05-08T21:22:03.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.192+0000 7f2435ec5700 1 --2- 172.21.15.3:0/1347008049 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24300b5010 0x7f243015c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.192+0000 7f2435ec5700 1 --2- 172.21.15.3:0/1347008049 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24300b53f0 0x7f243015f120 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.192+0000 7f2435ec5700 1 -- 172.21.15.3:0/1347008049 >> 172.21.15.3:0/1347008049 conn(0x7f24300722d0 msgr2=0x7f243006a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:03.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.192+0000 7f2435ec5700 1 -- 172.21.15.3:0/1347008049 shutdown_connections 2024-05-08T21:22:03.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.192+0000 7f2435ec5700 1 -- 172.21.15.3:0/1347008049 wait complete. 2024-05-08T21:22:03.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.193+0000 7f2435ec5700 1 Processor -- start 2024-05-08T21:22:03.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.193+0000 7f2435ec5700 1 -- start start 2024-05-08T21:22:03.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.194+0000 7f2435ec5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24300b5010 0x7f243013fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:03.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.194+0000 7f2435ec5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24300b53f0 0x7f2430153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:03.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.194+0000 7f2435ec5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f243015f660 0x7f2430159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:03.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.194+0000 7f2435ec5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243015a3c0 con 0x7f243015f660 2024-05-08T21:22:03.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.194+0000 7f2435ec5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243015a6a0 con 0x7f24300b53f0 2024-05-08T21:22:03.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.194+0000 7f2435ec5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f243015a980 con 0x7f24300b5010 2024-05-08T21:22:03.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.194+0000 7f242ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f243015f660 0x7f2430159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:03.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.194+0000 7f242ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f243015f660 0x7f2430159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39716/0 (socket says 172.21.15.3:39716) 2024-05-08T21:22:03.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.194+0000 7f242ffff700 1 -- 172.21.15.3:0/1282331503 learned_addr learned my addr 172.21.15.3:0/1282331503 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:03.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.194+0000 7f242effd700 1 --2- 172.21.15.3:0/1282331503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24300b53f0 0x7f2430153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:03.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.194+0000 7f242f7fe700 1 --2- 172.21.15.3:0/1282331503 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24300b5010 0x7f243013fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:03.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.195+0000 7f242ffff700 1 -- 172.21.15.3:0/1282331503 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24300b5010 msgr2=0x7f243013fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:03.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.195+0000 7f242ffff700 1 --2- 172.21.15.3:0/1282331503 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24300b5010 0x7f243013fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.195+0000 7f242ffff700 1 -- 172.21.15.3:0/1282331503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24300b53f0 msgr2=0x7f2430153d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:03.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.195+0000 7f242ffff700 1 --2- 172.21.15.3:0/1282331503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24300b53f0 0x7f2430153d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.195+0000 7f242ffff700 1 -- 172.21.15.3:0/1282331503 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24200094a0 con 0x7f243015f660 2024-05-08T21:22:03.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.195+0000 7f242ffff700 1 --2- 172.21.15.3:0/1282331503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f243015f660 0x7f2430159b20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2428002230 tx=0x7f242800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:03.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.195+0000 7f2417fff700 1 -- 172.21.15.3:0/1282331503 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242800aac0 con 0x7f243015f660 2024-05-08T21:22:03.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.195+0000 7f2417fff700 1 -- 172.21.15.3:0/1282331503 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242800ac40 con 0x7f243015f660 2024-05-08T21:22:03.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.195+0000 7f2435ec5700 1 -- 172.21.15.3:0/1282331503 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24240094a0 con 0x7f243015f660 2024-05-08T21:22:03.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.195+0000 7f2417fff700 1 -- 172.21.15.3:0/1282331503 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24280070d0 con 0x7f243015f660 2024-05-08T21:22:03.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.195+0000 7f2435ec5700 1 -- 172.21.15.3:0/1282331503 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f243015af40 con 0x7f243015f660 2024-05-08T21:22:03.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.196+0000 7f2435ec5700 1 -- 172.21.15.3:0/1282331503 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2430074bc0 con 0x7f243015f660 2024-05-08T21:22:03.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.197+0000 7f2417fff700 1 -- 172.21.15.3:0/1282331503 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2428007250 con 0x7f243015f660 2024-05-08T21:22:03.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.197+0000 7f2417fff700 1 --2- 172.21.15.3:0/1282331503 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f241c06ac90 0x7f241c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:03.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.197+0000 7f2417fff700 1 -- 172.21.15.3:0/1282331503 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2428088e60 con 0x7f243015f660 2024-05-08T21:22:03.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.197+0000 7f242f7fe700 1 --2- 172.21.15.3:0/1282331503 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f241c06ac90 0x7f241c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:03.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.198+0000 7f242f7fe700 1 --2- 172.21.15.3:0/1282331503 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f241c06ac90 0x7f241c06d150 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2420009cd0 tx=0x7f2420009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:03.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.202+0000 7f2417fff700 1 -- 172.21.15.3:0/1282331503 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2428053fc0 con 0x7f243015f660 2024-05-08T21:22:03.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.416+0000 7f2435ec5700 1 -- 172.21.15.3:0/1282331503 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f24300666c0 con 0x7f243015f660 2024-05-08T21:22:03.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.417+0000 7f2417fff700 1 -- 172.21.15.3:0/1282331503 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f24280180a0 con 0x7f243015f660 2024-05-08T21:22:03.419 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:22:03.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.419+0000 7f2435ec5700 1 -- 172.21.15.3:0/1282331503 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f241c06ac90 msgr2=0x7f241c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:03.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.419+0000 7f2435ec5700 1 --2- 172.21.15.3:0/1282331503 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f241c06ac90 0x7f241c06d150 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2420009cd0 tx=0x7f2420009210 comp rx=0 tx=0).stop 2024-05-08T21:22:03.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.419+0000 7f2435ec5700 1 -- 172.21.15.3:0/1282331503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f243015f660 msgr2=0x7f2430159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:03.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.419+0000 7f2435ec5700 1 --2- 172.21.15.3:0/1282331503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f243015f660 0x7f2430159b20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2428002230 tx=0x7f242800cf50 comp rx=0 tx=0).stop 2024-05-08T21:22:03.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.419+0000 7f2435ec5700 1 -- 172.21.15.3:0/1282331503 shutdown_connections 2024-05-08T21:22:03.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.419+0000 7f2435ec5700 1 --2- 172.21.15.3:0/1282331503 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24300b5010 0x7f243013fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.419+0000 7f2435ec5700 1 --2- 172.21.15.3:0/1282331503 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f241c06ac90 0x7f241c06d150 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.419+0000 7f2435ec5700 1 --2- 172.21.15.3:0/1282331503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f243015f660 0x7f2430159b20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.419+0000 7f2435ec5700 1 --2- 172.21.15.3:0/1282331503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24300b53f0 0x7f2430153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.419+0000 7f2435ec5700 1 -- 172.21.15.3:0/1282331503 >> 172.21.15.3:0/1282331503 conn(0x7f24300722d0 msgr2=0x7f24300af000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:03.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.419+0000 7f2435ec5700 1 -- 172.21.15.3:0/1282331503 shutdown_connections 2024-05-08T21:22:03.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.419+0000 7f2435ec5700 1 -- 172.21.15.3:0/1282331503 wait complete. 2024-05-08T21:22:03.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:22:03.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.569+0000 7f547c9f1700 1 Processor -- start 2024-05-08T21:22:03.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.569+0000 7f547c9f1700 1 -- start start 2024-05-08T21:22:03.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.570+0000 7f547c9f1700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f54780b5630 0x7f5478166480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:03.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.570+0000 7f547c9f1700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54780cfe20 con 0x7f54780b2340 2024-05-08T21:22:03.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.570+0000 7f547c9f1700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f547809ad40 con 0x7f547815ef60 2024-05-08T21:22:03.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.570+0000 7f547c9f1700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54780cd400 con 0x7f54780b5630 2024-05-08T21:22:03.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.570+0000 7f5476d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f54780b5630 0x7f5478166480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:03.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.570+0000 7f5476d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f54780b5630 0x7f5478166480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39728/0 (socket says 172.21.15.3:39728) 2024-05-08T21:22:03.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.570+0000 7f5475d9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f54780b2340 0x7f54780cf710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58038/0 (socket says 172.21.15.3:58038) 2024-05-08T21:22:03.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.570+0000 7f5476d9d700 1 -- 172.21.15.3:0/838098414 learned_addr learned my addr 172.21.15.3:0/838098414 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:03.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.570+0000 7f5476d9d700 1 -- 172.21.15.3:0/838098414 >> v1:172.21.15.186:6789/0 conn(0x7f547815ef60 legacy=0x7f54780cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:03.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.570+0000 7f5476d9d700 1 -- 172.21.15.3:0/838098414 >> v1:172.21.15.3:6789/0 conn(0x7f54780b2340 legacy=0x7f54780cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:03.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.570+0000 7f547559a700 1 -- 172.21.15.3:0/838098414 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2759683775 0 0) 0x7f54780cfe20 con 0x7f54780b2340 2024-05-08T21:22:03.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.570+0000 7f5476d9d700 1 -- 172.21.15.3:0/838098414 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54780ccb60 con 0x7f54780b5630 2024-05-08T21:22:03.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.571+0000 7f5476d9d700 1 --2- 172.21.15.3:0/838098414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54780b5630 0x7f5478166480 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5470009e50 tx=0x7f547000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f14302b107cf314 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:03.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.571+0000 7f547559a700 1 -- 172.21.15.3:0/838098414 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547000e070 con 0x7f54780b5630 2024-05-08T21:22:03.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.571+0000 7f547559a700 1 -- 172.21.15.3:0/838098414 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547000b9c0 con 0x7f54780b5630 2024-05-08T21:22:03.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.571+0000 7f547559a700 1 -- 172.21.15.3:0/838098414 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547000bd70 con 0x7f54780b5630 2024-05-08T21:22:03.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.571+0000 7f547c9f1700 1 -- 172.21.15.3:0/838098414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54780b5630 msgr2=0x7f5478166480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:03.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.571+0000 7f547c9f1700 1 --2- 172.21.15.3:0/838098414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54780b5630 0x7f5478166480 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5470009e50 tx=0x7f547000b330 comp rx=0 tx=0).stop 2024-05-08T21:22:03.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.571+0000 7f547c9f1700 1 -- 172.21.15.3:0/838098414 shutdown_connections 2024-05-08T21:22:03.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.571+0000 7f547c9f1700 1 --2- 172.21.15.3:0/838098414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54780b5630 0x7f5478166480 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.571+0000 7f547c9f1700 1 -- 172.21.15.3:0/838098414 >> 172.21.15.3:0/838098414 conn(0x7f54780b2f20 msgr2=0x7f54780b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:03.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.571+0000 7f547c9f1700 1 -- 172.21.15.3:0/838098414 shutdown_connections 2024-05-08T21:22:03.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.571+0000 7f547c9f1700 1 -- 172.21.15.3:0/838098414 wait complete. 2024-05-08T21:22:03.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.573+0000 7f547c9f1700 1 Processor -- start 2024-05-08T21:22:03.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.573+0000 7f547c9f1700 1 -- start start 2024-05-08T21:22:03.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f547c9f1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54780b2340 0x7f547814a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:03.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f547c9f1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54780b5630 0x7f54781f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:03.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f547c9f1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f547815ef60 0x7f54781f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:03.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f547c9f1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547806d490 con 0x7f54780b2340 2024-05-08T21:22:03.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f547c9f1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54781558a0 con 0x7f547815ef60 2024-05-08T21:22:03.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f547c9f1700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547814acd0 con 0x7f54780b5630 2024-05-08T21:22:03.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f547659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54780b2340 0x7f547814a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:03.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f547659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54780b2340 0x7f547814a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39730/0 (socket says 172.21.15.3:39730) 2024-05-08T21:22:03.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f547659c700 1 -- 172.21.15.3:0/1305636596 learned_addr learned my addr 172.21.15.3:0/1305636596 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:03.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f5475d9b700 1 --2- 172.21.15.3:0/1305636596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54780b5630 0x7f54781f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:03.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f5476d9d700 1 --2- 172.21.15.3:0/1305636596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f547815ef60 0x7f54781f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:03.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f547659c700 1 -- 172.21.15.3:0/1305636596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54780b5630 msgr2=0x7f54781f66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:03.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f547659c700 1 --2- 172.21.15.3:0/1305636596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54780b5630 0x7f54781f66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f547659c700 1 -- 172.21.15.3:0/1305636596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f547815ef60 msgr2=0x7f54781f6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:03.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f547659c700 1 --2- 172.21.15.3:0/1305636596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f547815ef60 0x7f54781f6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f547659c700 1 -- 172.21.15.3:0/1305636596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5470009b30 con 0x7f54780b2340 2024-05-08T21:22:03.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f5475d9b700 1 --2- 172.21.15.3:0/1305636596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54780b5630 0x7f54781f66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:03.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f5476d9d700 1 --2- 172.21.15.3:0/1305636596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f547815ef60 0x7f54781f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:03.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.574+0000 7f547659c700 1 --2- 172.21.15.3:0/1305636596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54780b2340 0x7f547814a790 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f546800c480 tx=0x7f546800c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:03.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.575+0000 7f5466ffd700 1 -- 172.21.15.3:0/1305636596 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54680052d0 con 0x7f54780b2340 2024-05-08T21:22:03.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.575+0000 7f5466ffd700 1 -- 172.21.15.3:0/1305636596 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5468005430 con 0x7f54780b2340 2024-05-08T21:22:03.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.575+0000 7f547c9f1700 1 -- 172.21.15.3:0/1305636596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f547806d1b0 con 0x7f54780b2340 2024-05-08T21:22:03.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.575+0000 7f5466ffd700 1 -- 172.21.15.3:0/1305636596 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54680038c0 con 0x7f54780b2340 2024-05-08T21:22:03.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.575+0000 7f547c9f1700 1 -- 172.21.15.3:0/1305636596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5478155530 con 0x7f54780b2340 2024-05-08T21:22:03.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.576+0000 7f547c9f1700 1 -- 172.21.15.3:0/1305636596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5478068000 con 0x7f54780b2340 2024-05-08T21:22:03.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.576+0000 7f5466ffd700 1 -- 172.21.15.3:0/1305636596 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5468003a20 con 0x7f54780b2340 2024-05-08T21:22:03.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.577+0000 7f5466ffd700 1 --2- 172.21.15.3:0/1305636596 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f545806ac90 0x7f545806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:03.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.577+0000 7f5466ffd700 1 -- 172.21.15.3:0/1305636596 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54680886e0 con 0x7f54780b2340 2024-05-08T21:22:03.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.577+0000 7f5475d9b700 1 --2- 172.21.15.3:0/1305636596 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f545806ac90 0x7f545806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:03.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.578+0000 7f5475d9b700 1 --2- 172.21.15.3:0/1305636596 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f545806ac90 0x7f545806d150 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f547814e2d0 tx=0x7f546c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:03.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.582+0000 7f5466ffd700 1 -- 172.21.15.3:0/1305636596 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f5468053950 con 0x7f54780b2340 2024-05-08T21:22:03.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.846+0000 7f547c9f1700 1 -- 172.21.15.3:0/1305636596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54780b4ff0 con 0x7f54780b2340 2024-05-08T21:22:03.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.847+0000 7f5466ffd700 1 -- 172.21.15.3:0/1305636596 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f5468056f70 con 0x7f54780b2340 2024-05-08T21:22:03.848 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:03.849 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/2839854293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2839854293},{"type":"v1","addr":"172.21.15.3:6834","nonce":2839854293}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1878326567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":1878326567},{"type":"v1","addr":"172.21.15.3:6835","nonce":1878326567}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/3624849830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3624849830},{"type":"v1","addr":"172.21.15.186:6836","nonce":3624849830}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:21:52.062548+0000","modified":"2024-05-08T21:21:55.079806+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6837/1462886680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1462886680},{"type":"v1","addr":"172.21.15.186:6837","nonce":1462886680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-05-08T21:22:03.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.849+0000 7f547c9f1700 1 -- 172.21.15.3:0/1305636596 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f545806ac90 msgr2=0x7f545806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:03.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.849+0000 7f547c9f1700 1 --2- 172.21.15.3:0/1305636596 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f545806ac90 0x7f545806d150 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f547814e2d0 tx=0x7f546c009040 comp rx=0 tx=0).stop 2024-05-08T21:22:03.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.849+0000 7f547c9f1700 1 -- 172.21.15.3:0/1305636596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54780b2340 msgr2=0x7f547814a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:03.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.849+0000 7f547c9f1700 1 --2- 172.21.15.3:0/1305636596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54780b2340 0x7f547814a790 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f546800c480 tx=0x7f546800c840 comp rx=0 tx=0).stop 2024-05-08T21:22:03.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.849+0000 7f547c9f1700 1 -- 172.21.15.3:0/1305636596 shutdown_connections 2024-05-08T21:22:03.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.849+0000 7f547c9f1700 1 --2- 172.21.15.3:0/1305636596 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f545806ac90 0x7f545806d150 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.849+0000 7f547c9f1700 1 --2- 172.21.15.3:0/1305636596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f547815ef60 0x7f54781f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.849+0000 7f547c9f1700 1 --2- 172.21.15.3:0/1305636596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54780b2340 0x7f547814a790 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.849+0000 7f547c9f1700 1 --2- 172.21.15.3:0/1305636596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54780b5630 0x7f54781f66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:03.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.849+0000 7f547c9f1700 1 -- 172.21.15.3:0/1305636596 >> 172.21.15.3:0/1305636596 conn(0x7f54780b2f20 msgr2=0x7f547815cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:03.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.849+0000 7f547c9f1700 1 -- 172.21.15.3:0/1305636596 shutdown_connections 2024-05-08T21:22:03.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.849+0000 7f547c9f1700 1 -- 172.21.15.3:0/1305636596 wait complete. 2024-05-08T21:22:03.852 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 8 2024-05-08T21:22:03.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:04.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.999+0000 7f48d02b5700 1 Processor -- start 2024-05-08T21:22:04.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:03.999+0000 7f48d02b5700 1 -- start start 2024-05-08T21:22:04.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.000+0000 7f48d02b5700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f48c815db00 0x7f48c806dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.000+0000 7f48d02b5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c80ce120 0x7f48c806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.000+0000 7f48d02b5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f48c806e700 0x7f48c815e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.000+0000 7f48d02b5700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c806fbf0 con 0x7f48c806e700 2024-05-08T21:22:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.000+0000 7f48d02b5700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c806edb0 con 0x7f48c815db00 2024-05-08T21:22:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.000+0000 7f48d02b5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c806f090 con 0x7f48c80ce120 2024-05-08T21:22:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.000+0000 7f48ce852700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f48c806e700 0x7f48c815e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.000+0000 7f48ce852700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f48c806e700 0x7f48c815e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39738/0 (socket says 172.21.15.3:39738) 2024-05-08T21:22:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.000+0000 7f48ce852700 1 -- 172.21.15.3:0/954930960 learned_addr learned my addr 172.21.15.3:0/954930960 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.000+0000 7f48ce051700 1 --2- 172.21.15.3:0/954930960 >> v2:172.21.15.186:3300/0 conn(0x7f48c815db00 0x7f48c806dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.000+0000 7f48cd850700 1 --2- 172.21.15.3:0/954930960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c80ce120 0x7f48c806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.000+0000 7f48ce852700 1 -- 172.21.15.3:0/954930960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c80ce120 msgr2=0x7f48c806e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.000+0000 7f48ce852700 1 --2- 172.21.15.3:0/954930960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c80ce120 0x7f48c806e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.001+0000 7f48ce852700 1 -- 172.21.15.3:0/954930960 >> v2:172.21.15.186:3300/0 conn(0x7f48c815db00 msgr2=0x7f48c806dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.001+0000 7f48ce852700 1 --2- 172.21.15.3:0/954930960 >> v2:172.21.15.186:3300/0 conn(0x7f48c815db00 0x7f48c806dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.001+0000 7f48ce852700 1 -- 172.21.15.3:0/954930960 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c806f370 con 0x7f48c806e700 2024-05-08T21:22:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.001+0000 7f48ce852700 1 --2- 172.21.15.3:0/954930960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c806e700 0x7f48c815e310 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f48c8070650 tx=0x7f48c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcbc9d6e5318805a server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.001+0000 7f48cd04f700 1 -- 172.21.15.3:0/954930960 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c000e070 con 0x7f48c806e700 2024-05-08T21:22:04.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.001+0000 7f48cd04f700 1 -- 172.21.15.3:0/954930960 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c000b910 con 0x7f48c806e700 2024-05-08T21:22:04.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.001+0000 7f48cd04f700 1 -- 172.21.15.3:0/954930960 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c000bcc0 con 0x7f48c806e700 2024-05-08T21:22:04.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.001+0000 7f48d02b5700 1 -- 172.21.15.3:0/954930960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c806e700 msgr2=0x7f48c815e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:04.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.001+0000 7f48d02b5700 1 --2- 172.21.15.3:0/954930960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c806e700 0x7f48c815e310 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f48c8070650 tx=0x7f48c000b330 comp rx=0 tx=0).stop 2024-05-08T21:22:04.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.002+0000 7f48d02b5700 1 -- 172.21.15.3:0/954930960 shutdown_connections 2024-05-08T21:22:04.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.002+0000 7f48d02b5700 1 --2- 172.21.15.3:0/954930960 >> v2:172.21.15.186:3300/0 conn(0x7f48c815db00 0x7f48c806dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.002+0000 7f48d02b5700 1 --2- 172.21.15.3:0/954930960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c806e700 0x7f48c815e310 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.002+0000 7f48d02b5700 1 --2- 172.21.15.3:0/954930960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c80ce120 0x7f48c806e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.002+0000 7f48d02b5700 1 -- 172.21.15.3:0/954930960 >> 172.21.15.3:0/954930960 conn(0x7f48c8099a50 msgr2=0x7f48c80d0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:04.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.002+0000 7f48d02b5700 1 -- 172.21.15.3:0/954930960 shutdown_connections 2024-05-08T21:22:04.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.002+0000 7f48d02b5700 1 -- 172.21.15.3:0/954930960 wait complete. 2024-05-08T21:22:04.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.003+0000 7f48d02b5700 1 Processor -- start 2024-05-08T21:22:04.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.003+0000 7f48d02b5700 1 -- start start 2024-05-08T21:22:04.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.004+0000 7f48d02b5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c806e700 0x7f48c80c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.004+0000 7f48d02b5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c80ce120 0x7f48c81584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.004+0000 7f48d02b5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48c815db00 0x7f48c80c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.004+0000 7f48d02b5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c80c8370 con 0x7f48c806e700 2024-05-08T21:22:04.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.004+0000 7f48d02b5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c80c8650 con 0x7f48c815db00 2024-05-08T21:22:04.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.004+0000 7f48d02b5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c80c8930 con 0x7f48c80ce120 2024-05-08T21:22:04.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.005+0000 7f48ce051700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c806e700 0x7f48c80c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.005+0000 7f48ce051700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c806e700 0x7f48c80c9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39746/0 (socket says 172.21.15.3:39746) 2024-05-08T21:22:04.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.005+0000 7f48ce051700 1 -- 172.21.15.3:0/1749586030 learned_addr learned my addr 172.21.15.3:0/1749586030 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:04.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.005+0000 7f48cd850700 1 --2- 172.21.15.3:0/1749586030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c80ce120 0x7f48c81584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.005+0000 7f48ce852700 1 --2- 172.21.15.3:0/1749586030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48c815db00 0x7f48c80c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.005+0000 7f48ce051700 1 -- 172.21.15.3:0/1749586030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c80ce120 msgr2=0x7f48c81584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:04.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.005+0000 7f48ce051700 1 --2- 172.21.15.3:0/1749586030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c80ce120 0x7f48c81584e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.005+0000 7f48ce051700 1 -- 172.21.15.3:0/1749586030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48c815db00 msgr2=0x7f48c80c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:04.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.005+0000 7f48ce051700 1 --2- 172.21.15.3:0/1749586030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48c815db00 0x7f48c80c5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.005+0000 7f48ce051700 1 -- 172.21.15.3:0/1749586030 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c0009b30 con 0x7f48c806e700 2024-05-08T21:22:04.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.005+0000 7f48ce051700 1 --2- 172.21.15.3:0/1749586030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c806e700 0x7f48c80c9410 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f48c400e9b0 tx=0x7f48c400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:04.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.005+0000 7f48beffd700 1 -- 172.21.15.3:0/1749586030 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c4009710 con 0x7f48c806e700 2024-05-08T21:22:04.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.005+0000 7f48beffd700 1 -- 172.21.15.3:0/1749586030 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c400ee50 con 0x7f48c806e700 2024-05-08T21:22:04.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.005+0000 7f48d02b5700 1 -- 172.21.15.3:0/1749586030 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c80c8c10 con 0x7f48c806e700 2024-05-08T21:22:04.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.005+0000 7f48beffd700 1 -- 172.21.15.3:0/1749586030 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c400ca50 con 0x7f48c806e700 2024-05-08T21:22:04.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.005+0000 7f48d02b5700 1 -- 172.21.15.3:0/1749586030 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c8158a20 con 0x7f48c806e700 2024-05-08T21:22:04.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.006+0000 7f48d02b5700 1 -- 172.21.15.3:0/1749586030 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c8068000 con 0x7f48c806e700 2024-05-08T21:22:04.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.007+0000 7f48beffd700 1 -- 172.21.15.3:0/1749586030 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f48c400cbb0 con 0x7f48c806e700 2024-05-08T21:22:04.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.007+0000 7f48beffd700 1 --2- 172.21.15.3:0/1749586030 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48ac06ac90 0x7f48ac06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.008+0000 7f48beffd700 1 -- 172.21.15.3:0/1749586030 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f48c4089240 con 0x7f48c806e700 2024-05-08T21:22:04.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.008+0000 7f48cd850700 1 --2- 172.21.15.3:0/1749586030 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48ac06ac90 0x7f48ac06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.008+0000 7f48cd850700 1 --2- 172.21.15.3:0/1749586030 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48ac06ac90 0x7f48ac06d150 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f48c80b61c0 tx=0x7f48b8009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:04.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.012+0000 7f48beffd700 1 -- 172.21.15.3:0/1749586030 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f48c40544b0 con 0x7f48c806e700 2024-05-08T21:22:04.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.234+0000 7f48d02b5700 1 -- 172.21.15.3:0/1749586030 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48c80b61f0 con 0x7f48c806e700 2024-05-08T21:22:04.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.235+0000 7f48beffd700 1 -- 172.21.15.3:0/1749586030 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15989 (secure 0 0 0) 0x7f48c400ce60 con 0x7f48c806e700 2024-05-08T21:22:04.237 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:04.237 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:53.064342+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:04.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.237+0000 7f48d02b5700 1 -- 172.21.15.3:0/1749586030 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48ac06ac90 msgr2=0x7f48ac06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:04.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.237+0000 7f48d02b5700 1 --2- 172.21.15.3:0/1749586030 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48ac06ac90 0x7f48ac06d150 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f48c80b61c0 tx=0x7f48b8009210 comp rx=0 tx=0).stop 2024-05-08T21:22:04.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.237+0000 7f48d02b5700 1 -- 172.21.15.3:0/1749586030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c806e700 msgr2=0x7f48c80c9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:04.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.237+0000 7f48d02b5700 1 --2- 172.21.15.3:0/1749586030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c806e700 0x7f48c80c9410 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f48c400e9b0 tx=0x7f48c400ecc0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.237+0000 7f48d02b5700 1 -- 172.21.15.3:0/1749586030 shutdown_connections 2024-05-08T21:22:04.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.237+0000 7f48d02b5700 1 --2- 172.21.15.3:0/1749586030 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48ac06ac90 0x7f48ac06d150 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.237+0000 7f48d02b5700 1 --2- 172.21.15.3:0/1749586030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48c815db00 0x7f48c80c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.237+0000 7f48d02b5700 1 --2- 172.21.15.3:0/1749586030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c806e700 0x7f48c80c9410 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.237+0000 7f48d02b5700 1 --2- 172.21.15.3:0/1749586030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c80ce120 0x7f48c81584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.237+0000 7f48d02b5700 1 -- 172.21.15.3:0/1749586030 >> 172.21.15.3:0/1749586030 conn(0x7f48c8099a50 msgr2=0x7f48c815c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:04.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.237+0000 7f48d02b5700 1 -- 172.21.15.3:0/1749586030 shutdown_connections 2024-05-08T21:22:04.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.238+0000 7f48d02b5700 1 -- 172.21.15.3:0/1749586030 wait complete. 2024-05-08T21:22:04.253 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-08T21:21:49.638998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:21:51.024632+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:22:04.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:22:04.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.390+0000 7fd1ba673700 1 Processor -- start 2024-05-08T21:22:04.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.390+0000 7fd1ba673700 1 -- start start 2024-05-08T21:22:04.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.391+0000 7fd1ba673700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1b40cbf50 0x7fd1b4161d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.391+0000 7fd1ba673700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd1b4162240 0x7fd1b415e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.391+0000 7fd1ba673700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b40cf970 con 0x7fd1b40d0f10 2024-05-08T21:22:04.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.391+0000 7fd1ba673700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b40cfc50 con 0x7fd1b4162240 2024-05-08T21:22:04.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.391+0000 7fd1ba673700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b40cff30 con 0x7fd1b40cbf50 2024-05-08T21:22:04.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.391+0000 7fd1b37fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd1b4162240 0x7fd1b415e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.391+0000 7fd1b8c10700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd1b40d0f10 0x7fd1b406a960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58044/0 (socket says 172.21.15.3:58044) 2024-05-08T21:22:04.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.391+0000 7fd1b8c10700 1 -- 172.21.15.3:0/2254528060 learned_addr learned my addr 172.21.15.3:0/2254528060 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:04.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.392+0000 7fd1b3fff700 1 --2- 172.21.15.3:0/2254528060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1b40cbf50 0x7fd1b4161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.392+0000 7fd1b37fe700 1 -- 172.21.15.3:0/2254528060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1b40cbf50 msgr2=0x7fd1b4161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:04.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.392+0000 7fd1b37fe700 1 --2- 172.21.15.3:0/2254528060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1b40cbf50 0x7fd1b4161d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.392+0000 7fd1b37fe700 1 -- 172.21.15.3:0/2254528060 >> v1:172.21.15.3:6789/0 conn(0x7fd1b40d0f10 legacy=0x7fd1b406a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:04.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.392+0000 7fd1b37fe700 1 -- 172.21.15.3:0/2254528060 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b41617e0 con 0x7fd1b4162240 2024-05-08T21:22:04.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.392+0000 7fd1b2ffd700 1 -- 172.21.15.3:0/2254528060 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44345784 0 0) 0x7fd1b40cf970 con 0x7fd1b40d0f10 2024-05-08T21:22:04.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.392+0000 7fd1b37fe700 1 --2- 172.21.15.3:0/2254528060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b4162240 0x7fd1b415e050 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8009840 tx=0x7fd1a8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88af27afa3f29efa server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:04.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.392+0000 7fd1b2ffd700 1 -- 172.21.15.3:0/2254528060 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8011830 con 0x7fd1b4162240 2024-05-08T21:22:04.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.392+0000 7fd1b2ffd700 1 -- 172.21.15.3:0/2254528060 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a8011990 con 0x7fd1b4162240 2024-05-08T21:22:04.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.392+0000 7fd1b2ffd700 1 -- 172.21.15.3:0/2254528060 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8011d20 con 0x7fd1b4162240 2024-05-08T21:22:04.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.393+0000 7fd1ba673700 1 -- 172.21.15.3:0/2254528060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b4162240 msgr2=0x7fd1b415e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:04.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.393+0000 7fd1ba673700 1 --2- 172.21.15.3:0/2254528060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b4162240 0x7fd1b415e050 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8009840 tx=0x7fd1a8009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:04.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.393+0000 7fd1ba673700 1 -- 172.21.15.3:0/2254528060 shutdown_connections 2024-05-08T21:22:04.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.393+0000 7fd1ba673700 1 --2- 172.21.15.3:0/2254528060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1b40cbf50 0x7fd1b4161d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.393+0000 7fd1ba673700 1 --2- 172.21.15.3:0/2254528060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b4162240 0x7fd1b415e050 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.393+0000 7fd1ba673700 1 -- 172.21.15.3:0/2254528060 >> 172.21.15.3:0/2254528060 conn(0x7fd1b4099a50 msgr2=0x7fd1b40d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:04.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.393+0000 7fd1ba673700 1 -- 172.21.15.3:0/2254528060 shutdown_connections 2024-05-08T21:22:04.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.393+0000 7fd1ba673700 1 -- 172.21.15.3:0/2254528060 wait complete. 2024-05-08T21:22:04.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.394+0000 7fd1ba673700 1 Processor -- start 2024-05-08T21:22:04.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.394+0000 7fd1ba673700 1 -- start start 2024-05-08T21:22:04.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.395+0000 7fd1ba673700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1b40cbf50 0x7fd1b4155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.395+0000 7fd1ba673700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b40d0f10 0x7fd1b41f3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.395+0000 7fd1ba673700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1b4162240 0x7fd1b40b4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.395+0000 7fd1ba673700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b41f3ac0 con 0x7fd1b40d0f10 2024-05-08T21:22:04.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.395+0000 7fd1ba673700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b4157180 con 0x7fd1b40cbf50 2024-05-08T21:22:04.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.395+0000 7fd1ba673700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b4156570 con 0x7fd1b4162240 2024-05-08T21:22:04.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.395+0000 7fd1b37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b40d0f10 0x7fd1b41f3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.395+0000 7fd1b37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b40d0f10 0x7fd1b41f3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39766/0 (socket says 172.21.15.3:39766) 2024-05-08T21:22:04.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.395+0000 7fd1b37fe700 1 -- 172.21.15.3:0/3474607033 learned_addr learned my addr 172.21.15.3:0/3474607033 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:04.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.395+0000 7fd1b8c10700 1 --2- 172.21.15.3:0/3474607033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1b4162240 0x7fd1b40b4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.396+0000 7fd1b3fff700 1 --2- 172.21.15.3:0/3474607033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1b40cbf50 0x7fd1b4155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.396+0000 7fd1b37fe700 1 -- 172.21.15.3:0/3474607033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1b4162240 msgr2=0x7fd1b40b4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:04.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.396+0000 7fd1b37fe700 1 --2- 172.21.15.3:0/3474607033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1b4162240 0x7fd1b40b4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.396+0000 7fd1b37fe700 1 -- 172.21.15.3:0/3474607033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1b40cbf50 msgr2=0x7fd1b4155890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:04.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.396+0000 7fd1b37fe700 1 --2- 172.21.15.3:0/3474607033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1b40cbf50 0x7fd1b4155890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.396+0000 7fd1b37fe700 1 -- 172.21.15.3:0/3474607033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a80094a0 con 0x7fd1b40d0f10 2024-05-08T21:22:04.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.396+0000 7fd1b8c10700 1 --2- 172.21.15.3:0/3474607033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1b4162240 0x7fd1b40b4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:04.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.396+0000 7fd1b3fff700 1 --2- 172.21.15.3:0/3474607033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1b40cbf50 0x7fd1b4155890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:04.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.396+0000 7fd1b37fe700 1 --2- 172.21.15.3:0/3474607033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b40d0f10 0x7fd1b41f3580 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8014040 tx=0x7fd1a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:04.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.396+0000 7fd1b0ff9700 1 -- 172.21.15.3:0/3474607033 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a80038a0 con 0x7fd1b40d0f10 2024-05-08T21:22:04.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.396+0000 7fd1b0ff9700 1 -- 172.21.15.3:0/3474607033 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a801e070 con 0x7fd1b40d0f10 2024-05-08T21:22:04.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.396+0000 7fd1b0ff9700 1 -- 172.21.15.3:0/3474607033 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8003c20 con 0x7fd1b40d0f10 2024-05-08T21:22:04.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.396+0000 7fd1ba673700 1 -- 172.21.15.3:0/3474607033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b4156850 con 0x7fd1b40d0f10 2024-05-08T21:22:04.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.396+0000 7fd1ba673700 1 -- 172.21.15.3:0/3474607033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b4156e10 con 0x7fd1b40d0f10 2024-05-08T21:22:04.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.397+0000 7fd1ba673700 1 -- 172.21.15.3:0/3474607033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b4156fa0 con 0x7fd1b40d0f10 2024-05-08T21:22:04.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.397+0000 7fd1b0ff9700 1 -- 172.21.15.3:0/3474607033 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd1a802d450 con 0x7fd1b40d0f10 2024-05-08T21:22:04.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.398+0000 7fd1b0ff9700 1 --2- 172.21.15.3:0/3474607033 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1a006ac90 0x7fd1a006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.398+0000 7fd1b0ff9700 1 -- 172.21.15.3:0/3474607033 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1a808a320 con 0x7fd1b40d0f10 2024-05-08T21:22:04.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.398+0000 7fd1b3fff700 1 --2- 172.21.15.3:0/3474607033 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1a006ac90 0x7fd1a006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.399+0000 7fd1b3fff700 1 --2- 172.21.15.3:0/3474607033 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1a006ac90 0x7fd1a006d150 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009b60 tx=0x7fd1a4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:04.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.403+0000 7fd1b0ff9700 1 -- 172.21.15.3:0/3474607033 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd1a8055590 con 0x7fd1b40d0f10 2024-05-08T21:22:04.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.617+0000 7fd1ba673700 1 -- 172.21.15.3:0/3474607033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd1b4074bc0 con 0x7fd1b40d0f10 2024-05-08T21:22:04.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.617+0000 7fd1b0ff9700 1 -- 172.21.15.3:0/3474607033 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd1a8058bb0 con 0x7fd1b40d0f10 2024-05-08T21:22:04.619 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:22:04.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.619+0000 7fd1ba673700 1 -- 172.21.15.3:0/3474607033 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1a006ac90 msgr2=0x7fd1a006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:04.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.619+0000 7fd1ba673700 1 --2- 172.21.15.3:0/3474607033 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1a006ac90 0x7fd1a006d150 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009b60 tx=0x7fd1a4008040 comp rx=0 tx=0).stop 2024-05-08T21:22:04.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.619+0000 7fd1ba673700 1 -- 172.21.15.3:0/3474607033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b40d0f10 msgr2=0x7fd1b41f3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:04.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.619+0000 7fd1ba673700 1 --2- 172.21.15.3:0/3474607033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b40d0f10 0x7fd1b41f3580 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8014040 tx=0x7fd1a8004750 comp rx=0 tx=0).stop 2024-05-08T21:22:04.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.619+0000 7fd1ba673700 1 -- 172.21.15.3:0/3474607033 shutdown_connections 2024-05-08T21:22:04.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.619+0000 7fd1ba673700 1 --2- 172.21.15.3:0/3474607033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1b4162240 0x7fd1b40b4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.619+0000 7fd1ba673700 1 --2- 172.21.15.3:0/3474607033 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1a006ac90 0x7fd1a006d150 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.619+0000 7fd1ba673700 1 --2- 172.21.15.3:0/3474607033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1b40cbf50 0x7fd1b4155890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.619+0000 7fd1ba673700 1 --2- 172.21.15.3:0/3474607033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b40d0f10 0x7fd1b41f3580 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.619+0000 7fd1ba673700 1 -- 172.21.15.3:0/3474607033 >> 172.21.15.3:0/3474607033 conn(0x7fd1b4099a50 msgr2=0x7fd1b40ade80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:04.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.619+0000 7fd1ba673700 1 -- 172.21.15.3:0/3474607033 shutdown_connections 2024-05-08T21:22:04.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.619+0000 7fd1ba673700 1 -- 172.21.15.3:0/3474607033 wait complete. 2024-05-08T21:22:04.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-08T21:22:04.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.772+0000 7f12c5a68700 1 Processor -- start 2024-05-08T21:22:04.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.772+0000 7f12c5a68700 1 -- start start 2024-05-08T21:22:04.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.773+0000 7f12c5a68700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f12c015d880 0x7f12c00af990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.773+0000 7f12c5a68700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c00b3870 con 0x7f12c006f8c0 2024-05-08T21:22:04.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.773+0000 7f12c5a68700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c00b4ee0 con 0x7f12c006e7e0 2024-05-08T21:22:04.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.773+0000 7f12c5a68700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c00b32f0 con 0x7f12c015d880 2024-05-08T21:22:04.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.773+0000 7f12bf7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f12c015d880 0x7f12c00af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.773+0000 7f12beffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f12c006f8c0 0x7f12c00ad260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58046/0 (socket says 172.21.15.3:58046) 2024-05-08T21:22:04.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.773+0000 7f12beffd700 1 -- 172.21.15.3:0/153353335 learned_addr learned my addr 172.21.15.3:0/153353335 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:04.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.773+0000 7f12bf7fe700 1 -- 172.21.15.3:0/153353335 >> v1:172.21.15.186:6789/0 conn(0x7f12c006e7e0 legacy=0x7f12c006a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:04.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.773+0000 7f12bf7fe700 1 -- 172.21.15.3:0/153353335 >> v1:172.21.15.3:6789/0 conn(0x7f12c006f8c0 legacy=0x7f12c00ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.773+0000 7f12bf7fe700 1 -- 172.21.15.3:0/153353335 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c006fd50 con 0x7f12c015d880 2024-05-08T21:22:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.773+0000 7f12be7fc700 1 -- 172.21.15.3:0/153353335 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300171343 0 0) 0x7f12c00b3870 con 0x7f12c006f8c0 2024-05-08T21:22:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.773+0000 7f12bf7fe700 1 --2- 172.21.15.3:0/153353335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c015d880 0x7f12c00af990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009840 tx=0x7f12b0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b7177a47f0620da server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.774+0000 7f12be7fc700 1 -- 172.21.15.3:0/153353335 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b0011830 con 0x7f12c015d880 2024-05-08T21:22:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.774+0000 7f12be7fc700 1 -- 172.21.15.3:0/153353335 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b0011990 con 0x7f12c015d880 2024-05-08T21:22:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.774+0000 7f12be7fc700 1 -- 172.21.15.3:0/153353335 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b0011d20 con 0x7f12c015d880 2024-05-08T21:22:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.774+0000 7f12c5a68700 1 -- 172.21.15.3:0/153353335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c015d880 msgr2=0x7f12c00af990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.774+0000 7f12c5a68700 1 --2- 172.21.15.3:0/153353335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c015d880 0x7f12c00af990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009840 tx=0x7f12b0009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.774+0000 7f12c5a68700 1 -- 172.21.15.3:0/153353335 shutdown_connections 2024-05-08T21:22:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.774+0000 7f12c5a68700 1 --2- 172.21.15.3:0/153353335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c015d880 0x7f12c00af990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.774+0000 7f12c5a68700 1 -- 172.21.15.3:0/153353335 >> 172.21.15.3:0/153353335 conn(0x7f12c00b5810 msgr2=0x7f12c00b5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.774+0000 7f12c5a68700 1 -- 172.21.15.3:0/153353335 shutdown_connections 2024-05-08T21:22:04.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.774+0000 7f12c5a68700 1 -- 172.21.15.3:0/153353335 wait complete. 2024-05-08T21:22:04.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.775+0000 7f12c5a68700 1 Processor -- start 2024-05-08T21:22:04.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.776+0000 7f12c5a68700 1 -- start start 2024-05-08T21:22:04.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.776+0000 7f12c5a68700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c006e7e0 0x7f12c009ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.776+0000 7f12c5a68700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12c006f8c0 0x7f12c01f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.776+0000 7f12c5a68700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12c015d880 0x7f12c00ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.776+0000 7f12c5a68700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c009b5b0 con 0x7f12c006e7e0 2024-05-08T21:22:04.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.776+0000 7f12c5a68700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c01f3fc0 con 0x7f12c006f8c0 2024-05-08T21:22:04.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.776+0000 7f12c5a68700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c009a9a0 con 0x7f12c015d880 2024-05-08T21:22:04.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12bf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c006e7e0 0x7f12c009ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12bf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c006e7e0 0x7f12c009ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39786/0 (socket says 172.21.15.3:39786) 2024-05-08T21:22:04.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12bf7fe700 1 -- 172.21.15.3:0/3368813291 learned_addr learned my addr 172.21.15.3:0/3368813291 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:04.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12beffd700 1 --2- 172.21.15.3:0/3368813291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12c006f8c0 0x7f12c01f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12bffff700 1 --2- 172.21.15.3:0/3368813291 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12c015d880 0x7f12c00ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12bf7fe700 1 -- 172.21.15.3:0/3368813291 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12c015d880 msgr2=0x7f12c00ca000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:04.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12bf7fe700 1 --2- 172.21.15.3:0/3368813291 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12c015d880 0x7f12c00ca000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12bf7fe700 1 -- 172.21.15.3:0/3368813291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12c006f8c0 msgr2=0x7f12c01f3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:04.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12bf7fe700 1 --2- 172.21.15.3:0/3368813291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12c006f8c0 0x7f12c01f3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:04.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12bf7fe700 1 -- 172.21.15.3:0/3368813291 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b00094a0 con 0x7f12c006e7e0 2024-05-08T21:22:04.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12bffff700 1 --2- 172.21.15.3:0/3368813291 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12c015d880 0x7f12c00ca000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:04.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12bf7fe700 1 --2- 172.21.15.3:0/3368813291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c006e7e0 0x7f12c009ba40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f12b0014040 tx=0x7f12b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:04.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12a7fff700 1 -- 172.21.15.3:0/3368813291 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b001a900 con 0x7f12c006e7e0 2024-05-08T21:22:04.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12a7fff700 1 -- 172.21.15.3:0/3368813291 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b001aa60 con 0x7f12c006e7e0 2024-05-08T21:22:04.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12c5a68700 1 -- 172.21.15.3:0/3368813291 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c009ac80 con 0x7f12c006e7e0 2024-05-08T21:22:04.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12a7fff700 1 -- 172.21.15.3:0/3368813291 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b001add0 con 0x7f12c006e7e0 2024-05-08T21:22:04.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.777+0000 7f12c5a68700 1 -- 172.21.15.3:0/3368813291 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c009b240 con 0x7f12c006e7e0 2024-05-08T21:22:04.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.778+0000 7f12c5a68700 1 -- 172.21.15.3:0/3368813291 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c009b890 con 0x7f12c006e7e0 2024-05-08T21:22:04.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.779+0000 7f12a7fff700 1 -- 172.21.15.3:0/3368813291 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f12b0022070 con 0x7f12c006e7e0 2024-05-08T21:22:04.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.779+0000 7f12a7fff700 1 --2- 172.21.15.3:0/3368813291 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f12ac06ac90 0x7f12ac06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:04.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.779+0000 7f12a7fff700 1 -- 172.21.15.3:0/3368813291 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12b008a6a0 con 0x7f12c006e7e0 2024-05-08T21:22:04.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.779+0000 7f12beffd700 1 --2- 172.21.15.3:0/3368813291 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f12ac06ac90 0x7f12ac06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:04.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.780+0000 7f12beffd700 1 --2- 172.21.15.3:0/3368813291 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f12ac06ac90 0x7f12ac06d150 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f12c015c810 tx=0x7f12b4007c40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:04.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:04.784+0000 7f12a7fff700 1 -- 172.21.15.3:0/3368813291 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f12b0055870 con 0x7f12c006e7e0 2024-05-08T21:22:05.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.001+0000 7f12c5a68700 1 -- 172.21.15.3:0/3368813291 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f12c009b3d0 con 0x7f12c006e7e0 2024-05-08T21:22:05.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.001+0000 7f12a7fff700 1 -- 172.21.15.3:0/3368813291 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f12b0058e90 con 0x7f12c006e7e0 2024-05-08T21:22:05.003 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:22:05.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.004+0000 7f12c5a68700 1 -- 172.21.15.3:0/3368813291 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f12ac06ac90 msgr2=0x7f12ac06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.004+0000 7f12c5a68700 1 --2- 172.21.15.3:0/3368813291 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f12ac06ac90 0x7f12ac06d150 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f12c015c810 tx=0x7f12b4007c40 comp rx=0 tx=0).stop 2024-05-08T21:22:05.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.004+0000 7f12c5a68700 1 -- 172.21.15.3:0/3368813291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c006e7e0 msgr2=0x7f12c009ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.004+0000 7f12c5a68700 1 --2- 172.21.15.3:0/3368813291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c006e7e0 0x7f12c009ba40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f12b0014040 tx=0x7f12b0004750 comp rx=0 tx=0).stop 2024-05-08T21:22:05.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.004+0000 7f12c5a68700 1 -- 172.21.15.3:0/3368813291 shutdown_connections 2024-05-08T21:22:05.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.004+0000 7f12c5a68700 1 --2- 172.21.15.3:0/3368813291 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12c015d880 0x7f12c00ca000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.004+0000 7f12c5a68700 1 --2- 172.21.15.3:0/3368813291 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f12ac06ac90 0x7f12ac06d150 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.004+0000 7f12c5a68700 1 --2- 172.21.15.3:0/3368813291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c006e7e0 0x7f12c009ba40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.004+0000 7f12c5a68700 1 --2- 172.21.15.3:0/3368813291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12c006f8c0 0x7f12c01f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.004+0000 7f12c5a68700 1 -- 172.21.15.3:0/3368813291 >> 172.21.15.3:0/3368813291 conn(0x7f12c00b5810 msgr2=0x7f12c015fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:05.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.004+0000 7f12c5a68700 1 -- 172.21.15.3:0/3368813291 shutdown_connections 2024-05-08T21:22:05.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.004+0000 7f12c5a68700 1 -- 172.21.15.3:0/3368813291 wait complete. 2024-05-08T21:22:05.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:22:05.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.157+0000 7f80668cd700 1 Processor -- start 2024-05-08T21:22:05.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.157+0000 7f80668cd700 1 -- start start 2024-05-08T21:22:05.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.158+0000 7f80668cd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f806015da60 0x7f80600d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.158+0000 7f80668cd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806009a790 0x7f8060099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.158+0000 7f80668cd700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f806006d010 con 0x7f8060099f90 2024-05-08T21:22:05.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.158+0000 7f80668cd700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f806009a420 con 0x7f806015da60 2024-05-08T21:22:05.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.158+0000 7f80668cd700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806015de40 con 0x7f806009a790 2024-05-08T21:22:05.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.158+0000 7f805f7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806009a790 0x7f8060099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.158+0000 7f805ffff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f806015da60 0x7f80600d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.158+0000 7f805f7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806009a790 0x7f8060099a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.3:54844/0 (socket says 172.21.15.3:54844) 2024-05-08T21:22:05.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.158+0000 7f8064e6a700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f8060099f90 0x7f806006bc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:51432/0 (socket says 172.21.15.3:51432) 2024-05-08T21:22:05.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.158+0000 7f805f7fe700 1 -- 172.21.15.3:0/3793100873 learned_addr learned my addr 172.21.15.3:0/3793100873 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:05.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.158+0000 7f805ffff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f806015da60 0x7f80600d0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:58010/0 (socket says 172.21.15.3:58010) 2024-05-08T21:22:05.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.159+0000 7f805ffff700 1 -- 172.21.15.3:0/3793100873 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806009a790 msgr2=0x7f8060099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:05.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.159+0000 7f805ffff700 1 --2- 172.21.15.3:0/3793100873 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806009a790 0x7f8060099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.159+0000 7f805ffff700 1 -- 172.21.15.3:0/3793100873 >> v1:172.21.15.186:6789/0 conn(0x7f8060099f90 legacy=0x7f806006bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.159+0000 7f805ffff700 1 -- 172.21.15.3:0/3793100873 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80600d0880 con 0x7f806015da60 2024-05-08T21:22:05.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.159+0000 7f805f7fe700 1 --2- 172.21.15.3:0/3793100873 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806009a790 0x7f8060099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:05.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.159+0000 7f805ffff700 1 --2- 172.21.15.3:0/3793100873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f806015da60 0x7f80600d0340 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8050009840 tx=0x7f8050009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=773f68c36af21593 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:05.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.159+0000 7f805effd700 1 -- 172.21.15.3:0/3793100873 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050011830 con 0x7f806015da60 2024-05-08T21:22:05.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.159+0000 7f805effd700 1 -- 172.21.15.3:0/3793100873 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8050011990 con 0x7f806015da60 2024-05-08T21:22:05.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.159+0000 7f805effd700 1 -- 172.21.15.3:0/3793100873 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050011d20 con 0x7f806015da60 2024-05-08T21:22:05.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.160+0000 7f80668cd700 1 -- 172.21.15.3:0/3793100873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f806015da60 msgr2=0x7f80600d0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.160+0000 7f80668cd700 1 --2- 172.21.15.3:0/3793100873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f806015da60 0x7f80600d0340 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8050009840 tx=0x7f8050009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:05.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.160+0000 7f80668cd700 1 -- 172.21.15.3:0/3793100873 shutdown_connections 2024-05-08T21:22:05.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.160+0000 7f80668cd700 1 --2- 172.21.15.3:0/3793100873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f806015da60 0x7f80600d0340 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.160+0000 7f80668cd700 1 --2- 172.21.15.3:0/3793100873 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806009a790 0x7f8060099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.160+0000 7f80668cd700 1 -- 172.21.15.3:0/3793100873 >> 172.21.15.3:0/3793100873 conn(0x7f80600b2f60 msgr2=0x7f80600b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:05.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.160+0000 7f80668cd700 1 -- 172.21.15.3:0/3793100873 shutdown_connections 2024-05-08T21:22:05.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.160+0000 7f80668cd700 1 -- 172.21.15.3:0/3793100873 wait complete. 2024-05-08T21:22:05.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.161+0000 7f80668cd700 1 Processor -- start 2024-05-08T21:22:05.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.161+0000 7f80668cd700 1 -- start start 2024-05-08T21:22:05.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.162+0000 7f80668cd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8060099f90 0x7f8060152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.162+0000 7f80668cd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f806009a790 0x7f80600b6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.162+0000 7f80668cd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806015da60 0x7f806015e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.162+0000 7f80668cd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80600b6830 con 0x7f8060099f90 2024-05-08T21:22:05.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.162+0000 7f80668cd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8060152e40 con 0x7f806009a790 2024-05-08T21:22:05.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.162+0000 7f80668cd700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80600b5bb0 con 0x7f806015da60 2024-05-08T21:22:05.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.162+0000 7f805ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8060099f90 0x7f8060152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.162+0000 7f805ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8060099f90 0x7f8060152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39790/0 (socket says 172.21.15.3:39790) 2024-05-08T21:22:05.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.162+0000 7f805ffff700 1 -- 172.21.15.3:0/1991462479 learned_addr learned my addr 172.21.15.3:0/1991462479 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:05.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.162+0000 7f805f7fe700 1 --2- 172.21.15.3:0/1991462479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f806009a790 0x7f80600b6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.162+0000 7f8064e6a700 1 --2- 172.21.15.3:0/1991462479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806015da60 0x7f806015e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.163+0000 7f805ffff700 1 -- 172.21.15.3:0/1991462479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806015da60 msgr2=0x7f806015e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.163+0000 7f805ffff700 1 --2- 172.21.15.3:0/1991462479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806015da60 0x7f806015e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.163+0000 7f805ffff700 1 -- 172.21.15.3:0/1991462479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f806009a790 msgr2=0x7f80600b6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.163+0000 7f805ffff700 1 --2- 172.21.15.3:0/1991462479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f806009a790 0x7f80600b6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.163+0000 7f805ffff700 1 -- 172.21.15.3:0/1991462479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8054009550 con 0x7f8060099f90 2024-05-08T21:22:05.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.163+0000 7f805ffff700 1 --2- 172.21.15.3:0/1991462479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8060099f90 0x7f8060152900 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8050009fd0 tx=0x7f8050004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:05.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.163+0000 7f805cff9700 1 -- 172.21.15.3:0/1991462479 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80500036a0 con 0x7f8060099f90 2024-05-08T21:22:05.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.163+0000 7f805cff9700 1 -- 172.21.15.3:0/1991462479 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805001e070 con 0x7f8060099f90 2024-05-08T21:22:05.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.163+0000 7f80668cd700 1 -- 172.21.15.3:0/1991462479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80500094a0 con 0x7f8060099f90 2024-05-08T21:22:05.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.163+0000 7f805cff9700 1 -- 172.21.15.3:0/1991462479 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050003a60 con 0x7f8060099f90 2024-05-08T21:22:05.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.163+0000 7f80668cd700 1 -- 172.21.15.3:0/1991462479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80600b6170 con 0x7f8060099f90 2024-05-08T21:22:05.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.164+0000 7f80668cd700 1 -- 172.21.15.3:0/1991462479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8060074bc0 con 0x7f8060099f90 2024-05-08T21:22:05.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.165+0000 7f805cff9700 1 -- 172.21.15.3:0/1991462479 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8050003bc0 con 0x7f8060099f90 2024-05-08T21:22:05.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.165+0000 7f805cff9700 1 --2- 172.21.15.3:0/1991462479 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f804c06ac90 0x7f804c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.165+0000 7f805cff9700 1 -- 172.21.15.3:0/1991462479 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f805008a230 con 0x7f8060099f90 2024-05-08T21:22:05.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.165+0000 7f805f7fe700 1 --2- 172.21.15.3:0/1991462479 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f804c06ac90 0x7f804c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.166+0000 7f805f7fe700 1 --2- 172.21.15.3:0/1991462479 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f804c06ac90 0x7f804c06d150 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f806015e130 tx=0x7f80540092c0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:05.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.170+0000 7f805cff9700 1 -- 172.21.15.3:0/1991462479 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f8050055420 con 0x7f8060099f90 2024-05-08T21:22:05.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.387+0000 7f80668cd700 1 -- 172.21.15.3:0/1991462479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f80600b6aa0 con 0x7f8060099f90 2024-05-08T21:22:05.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.388+0000 7f805cff9700 1 -- 172.21.15.3:0/1991462479 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8050058a40 con 0x7f8060099f90 2024-05-08T21:22:05.390 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:22:05.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.390+0000 7f80668cd700 1 -- 172.21.15.3:0/1991462479 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f804c06ac90 msgr2=0x7f804c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.390+0000 7f80668cd700 1 --2- 172.21.15.3:0/1991462479 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f804c06ac90 0x7f804c06d150 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f806015e130 tx=0x7f80540092c0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.390+0000 7f80668cd700 1 -- 172.21.15.3:0/1991462479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8060099f90 msgr2=0x7f8060152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.390+0000 7f80668cd700 1 --2- 172.21.15.3:0/1991462479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8060099f90 0x7f8060152900 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8050009fd0 tx=0x7f8050004750 comp rx=0 tx=0).stop 2024-05-08T21:22:05.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.390+0000 7f80668cd700 1 -- 172.21.15.3:0/1991462479 shutdown_connections 2024-05-08T21:22:05.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.390+0000 7f80668cd700 1 --2- 172.21.15.3:0/1991462479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806015da60 0x7f806015e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.390+0000 7f80668cd700 1 --2- 172.21.15.3:0/1991462479 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f804c06ac90 0x7f804c06d150 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.390+0000 7f80668cd700 1 --2- 172.21.15.3:0/1991462479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8060099f90 0x7f8060152900 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.390+0000 7f80668cd700 1 --2- 172.21.15.3:0/1991462479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f806009a790 0x7f80600b6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.390+0000 7f80668cd700 1 -- 172.21.15.3:0/1991462479 >> 172.21.15.3:0/1991462479 conn(0x7f80600b2f60 msgr2=0x7f80600cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:05.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.390+0000 7f80668cd700 1 -- 172.21.15.3:0/1991462479 shutdown_connections 2024-05-08T21:22:05.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.390+0000 7f80668cd700 1 -- 172.21.15.3:0/1991462479 wait complete. 2024-05-08T21:22:05.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:22:05.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.539+0000 7f3386dc0700 1 Processor -- start 2024-05-08T21:22:05.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.539+0000 7f3386dc0700 1 -- start start 2024-05-08T21:22:05.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f3386dc0700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f338015b810 0x7f33800cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f3386dc0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33800d0520 0x7f3380099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f3386dc0700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3380099f90 0x7f33800ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f3386dc0700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f338009a440 con 0x7f338015b810 2024-05-08T21:22:05.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f3386dc0700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f338009aca0 con 0x7f3380099f90 2024-05-08T21:22:05.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f3386dc0700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33800cd9e0 con 0x7f33800d0520 2024-05-08T21:22:05.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f3384b5c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f338015b810 0x7f33800cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f3384b5c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f338015b810 0x7f33800cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39792/0 (socket says 172.21.15.3:39792) 2024-05-08T21:22:05.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f3384b5c700 1 -- 172.21.15.3:0/2844880237 learned_addr learned my addr 172.21.15.3:0/2844880237 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:05.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f337ffff700 1 --2- 172.21.15.3:0/2844880237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33800d0520 0x7f3380099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f338535d700 1 --2- 172.21.15.3:0/2844880237 >> v2:172.21.15.186:3300/0 conn(0x7f3380099f90 0x7f33800ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f3384b5c700 1 -- 172.21.15.3:0/2844880237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33800d0520 msgr2=0x7f3380099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:05.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f3384b5c700 1 --2- 172.21.15.3:0/2844880237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33800d0520 0x7f3380099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f3384b5c700 1 -- 172.21.15.3:0/2844880237 >> v2:172.21.15.186:3300/0 conn(0x7f3380099f90 msgr2=0x7f33800ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:05.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f3384b5c700 1 --2- 172.21.15.3:0/2844880237 >> v2:172.21.15.186:3300/0 conn(0x7f3380099f90 0x7f33800ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f3384b5c700 1 -- 172.21.15.3:0/2844880237 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33800cf220 con 0x7f338015b810 2024-05-08T21:22:05.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f3384b5c700 1 --2- 172.21.15.3:0/2844880237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f338015b810 0x7f33800cffe0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f33800cbb00 tx=0x7f3370009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994769a459b61fd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:05.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f337f7fe700 1 -- 172.21.15.3:0/2844880237 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3370011830 con 0x7f338015b810 2024-05-08T21:22:05.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f337f7fe700 1 -- 172.21.15.3:0/2844880237 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3370011990 con 0x7f338015b810 2024-05-08T21:22:05.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.541+0000 7f337f7fe700 1 -- 172.21.15.3:0/2844880237 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3370011d20 con 0x7f338015b810 2024-05-08T21:22:05.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.542+0000 7f3386dc0700 1 -- 172.21.15.3:0/2844880237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f338015b810 msgr2=0x7f33800cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.542+0000 7f3386dc0700 1 --2- 172.21.15.3:0/2844880237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f338015b810 0x7f33800cffe0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f33800cbb00 tx=0x7f3370009c00 comp rx=0 tx=0).stop 2024-05-08T21:22:05.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.542+0000 7f3386dc0700 1 -- 172.21.15.3:0/2844880237 shutdown_connections 2024-05-08T21:22:05.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.542+0000 7f3386dc0700 1 --2- 172.21.15.3:0/2844880237 >> v2:172.21.15.186:3300/0 conn(0x7f3380099f90 0x7f33800ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.542+0000 7f3386dc0700 1 --2- 172.21.15.3:0/2844880237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f338015b810 0x7f33800cffe0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.542+0000 7f3386dc0700 1 --2- 172.21.15.3:0/2844880237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33800d0520 0x7f3380099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.542+0000 7f3386dc0700 1 -- 172.21.15.3:0/2844880237 >> 172.21.15.3:0/2844880237 conn(0x7f33800b3010 msgr2=0x7f33800b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:05.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.542+0000 7f3386dc0700 1 -- 172.21.15.3:0/2844880237 shutdown_connections 2024-05-08T21:22:05.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.542+0000 7f3386dc0700 1 -- 172.21.15.3:0/2844880237 wait complete. 2024-05-08T21:22:05.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.543+0000 7f3386dc0700 1 Processor -- start 2024-05-08T21:22:05.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.543+0000 7f3386dc0700 1 -- start start 2024-05-08T21:22:05.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.544+0000 7f3386dc0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3380099f90 0x7f33800c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.544+0000 7f3386dc0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33800d0520 0x7f338015be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.544+0000 7f3386dc0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f338015b810 0x7f3380149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.544+0000 7f3386dc0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338015c3a0 con 0x7f3380099f90 2024-05-08T21:22:05.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.544+0000 7f3386dc0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3380149e80 con 0x7f33800d0520 2024-05-08T21:22:05.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.544+0000 7f3386dc0700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f338015c590 con 0x7f338015b810 2024-05-08T21:22:05.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.544+0000 7f3384b5c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3380099f90 0x7f33800c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.545+0000 7f3384b5c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3380099f90 0x7f33800c9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39808/0 (socket says 172.21.15.3:39808) 2024-05-08T21:22:05.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.545+0000 7f3384b5c700 1 -- 172.21.15.3:0/3759358376 learned_addr learned my addr 172.21.15.3:0/3759358376 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:05.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.545+0000 7f337ffff700 1 --2- 172.21.15.3:0/3759358376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33800d0520 0x7f338015be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.545+0000 7f338535d700 1 --2- 172.21.15.3:0/3759358376 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f338015b810 0x7f3380149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.545+0000 7f3384b5c700 1 -- 172.21.15.3:0/3759358376 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f338015b810 msgr2=0x7f3380149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.545+0000 7f3384b5c700 1 --2- 172.21.15.3:0/3759358376 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f338015b810 0x7f3380149940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.545+0000 7f3384b5c700 1 -- 172.21.15.3:0/3759358376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33800d0520 msgr2=0x7f338015be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.545+0000 7f3384b5c700 1 --2- 172.21.15.3:0/3759358376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33800d0520 0x7f338015be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.545+0000 7f3384b5c700 1 -- 172.21.15.3:0/3759358376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3370009550 con 0x7f3380099f90 2024-05-08T21:22:05.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.545+0000 7f3384b5c700 1 --2- 172.21.15.3:0/3759358376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3380099f90 0x7f33800c9230 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f33700098f0 tx=0x7f3370004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:05.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.545+0000 7f337d7fa700 1 -- 172.21.15.3:0/3759358376 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3370004300 con 0x7f3380099f90 2024-05-08T21:22:05.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.545+0000 7f337d7fa700 1 -- 172.21.15.3:0/3759358376 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337001f070 con 0x7f3380099f90 2024-05-08T21:22:05.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.545+0000 7f3386dc0700 1 -- 172.21.15.3:0/3759358376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338015c870 con 0x7f3380099f90 2024-05-08T21:22:05.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.545+0000 7f337d7fa700 1 -- 172.21.15.3:0/3759358376 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33700036a0 con 0x7f3380099f90 2024-05-08T21:22:05.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.545+0000 7f3386dc0700 1 -- 172.21.15.3:0/3759358376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338015ce30 con 0x7f3380099f90 2024-05-08T21:22:05.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.546+0000 7f3386dc0700 1 -- 172.21.15.3:0/3759358376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3380068000 con 0x7f3380099f90 2024-05-08T21:22:05.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.547+0000 7f337d7fa700 1 -- 172.21.15.3:0/3759358376 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3370021030 con 0x7f3380099f90 2024-05-08T21:22:05.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.547+0000 7f337d7fa700 1 --2- 172.21.15.3:0/3759358376 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f336006ac90 0x7f336006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.547+0000 7f337d7fa700 1 -- 172.21.15.3:0/3759358376 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f337008a700 con 0x7f3380099f90 2024-05-08T21:22:05.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.548+0000 7f337ffff700 1 --2- 172.21.15.3:0/3759358376 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f336006ac90 0x7f336006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.548+0000 7f337ffff700 1 --2- 172.21.15.3:0/3759358376 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f336006ac90 0x7f336006d150 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f33800c7e80 tx=0x7f3368009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:05.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.552+0000 7f337d7fa700 1 -- 172.21.15.3:0/3759358376 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f3370055850 con 0x7f3380099f90 2024-05-08T21:22:05.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.819+0000 7f3386dc0700 1 -- 172.21.15.3:0/3759358376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f338015f0d0 con 0x7f3380099f90 2024-05-08T21:22:05.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.820+0000 7f337d7fa700 1 -- 172.21.15.3:0/3759358376 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f3370058e70 con 0x7f3380099f90 2024-05-08T21:22:05.821 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:05.822 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/2839854293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2839854293},{"type":"v1","addr":"172.21.15.3:6834","nonce":2839854293}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1878326567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":1878326567},{"type":"v1","addr":"172.21.15.3:6835","nonce":1878326567}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/3624849830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3624849830},{"type":"v1","addr":"172.21.15.186:6836","nonce":3624849830}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:21:52.062548+0000","modified":"2024-05-08T21:21:55.079806+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6837/1462886680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1462886680},{"type":"v1","addr":"172.21.15.186:6837","nonce":1462886680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-05-08T21:22:05.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.822+0000 7f3386dc0700 1 -- 172.21.15.3:0/3759358376 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f336006ac90 msgr2=0x7f336006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.822+0000 7f3386dc0700 1 --2- 172.21.15.3:0/3759358376 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f336006ac90 0x7f336006d150 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f33800c7e80 tx=0x7f3368009210 comp rx=0 tx=0).stop 2024-05-08T21:22:05.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.822+0000 7f3386dc0700 1 -- 172.21.15.3:0/3759358376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3380099f90 msgr2=0x7f33800c9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.822+0000 7f3386dc0700 1 --2- 172.21.15.3:0/3759358376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3380099f90 0x7f33800c9230 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f33700098f0 tx=0x7f3370004750 comp rx=0 tx=0).stop 2024-05-08T21:22:05.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.822+0000 7f3386dc0700 1 -- 172.21.15.3:0/3759358376 shutdown_connections 2024-05-08T21:22:05.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.822+0000 7f3386dc0700 1 --2- 172.21.15.3:0/3759358376 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f338015b810 0x7f3380149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.822+0000 7f3386dc0700 1 --2- 172.21.15.3:0/3759358376 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f336006ac90 0x7f336006d150 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.822+0000 7f3386dc0700 1 --2- 172.21.15.3:0/3759358376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3380099f90 0x7f33800c9230 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.822+0000 7f3386dc0700 1 --2- 172.21.15.3:0/3759358376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33800d0520 0x7f338015be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.822+0000 7f3386dc0700 1 -- 172.21.15.3:0/3759358376 >> 172.21.15.3:0/3759358376 conn(0x7f33800b3010 msgr2=0x7f33800b2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:05.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.822+0000 7f3386dc0700 1 -- 172.21.15.3:0/3759358376 shutdown_connections 2024-05-08T21:22:05.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.822+0000 7f3386dc0700 1 -- 172.21.15.3:0/3759358376 wait complete. 2024-05-08T21:22:05.825 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 8 2024-05-08T21:22:05.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:05.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.971+0000 7f2726def700 1 Processor -- start 2024-05-08T21:22:05.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.972+0000 7f2726def700 1 -- start start 2024-05-08T21:22:05.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.972+0000 7f2726def700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f27200ce120 0x7f272006e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.972+0000 7f2726def700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272006fbf0 con 0x7f272015db00 2024-05-08T21:22:05.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.972+0000 7f2726def700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272006edb0 con 0x7f27200cb2a0 2024-05-08T21:22:05.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.972+0000 7f2726def700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f272006f090 con 0x7f27200ce120 2024-05-08T21:22:05.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.973+0000 7f2724b8b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f272015db00 0x7f272006dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58050/0 (socket says 172.21.15.3:58050) 2024-05-08T21:22:05.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.973+0000 7f2724b8b700 1 -- 172.21.15.3:0/2082604538 learned_addr learned my addr 172.21.15.3:0/2082604538 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:05.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.973+0000 7f271ffff700 1 --2- 172.21.15.3:0/2082604538 >> v2:172.21.15.186:3300/0 conn(0x7f27200ce120 0x7f272006e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.973+0000 7f271f7fe700 1 -- 172.21.15.3:0/2082604538 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656403829 0 0) 0x7f272006fbf0 con 0x7f272015db00 2024-05-08T21:22:05.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.973+0000 7f271f7fe700 1 -- 172.21.15.3:0/2082604538 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f270c0035b0 con 0x7f272015db00 2024-05-08T21:22:05.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.973+0000 7f271f7fe700 1 -- 172.21.15.3:0/2082604538 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3234129763 0 0) 0x7f270c0035b0 con 0x7f272015db00 2024-05-08T21:22:05.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.973+0000 7f271ffff700 1 -- 172.21.15.3:0/2082604538 >> v1:172.21.15.186:6789/0 conn(0x7f27200cb2a0 legacy=0x7f27200cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.973+0000 7f271ffff700 1 -- 172.21.15.3:0/2082604538 >> v1:172.21.15.3:6789/0 conn(0x7f272015db00 legacy=0x7f272006dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.973+0000 7f271ffff700 1 -- 172.21.15.3:0/2082604538 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272006f370 con 0x7f27200ce120 2024-05-08T21:22:05.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.973+0000 7f271ffff700 1 --2- 172.21.15.3:0/2082604538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27200ce120 0x7f272006e390 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2708009840 tx=0x7f2708009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd603d6bfd42d4a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:05.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.974+0000 7f271f7fe700 1 -- 172.21.15.3:0/2082604538 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710001670 con 0x7f27200ce120 2024-05-08T21:22:05.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.974+0000 7f271f7fe700 1 -- 172.21.15.3:0/2082604538 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2708004030 con 0x7f27200ce120 2024-05-08T21:22:05.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.974+0000 7f271f7fe700 1 -- 172.21.15.3:0/2082604538 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27080043c0 con 0x7f27200ce120 2024-05-08T21:22:05.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.974+0000 7f2726def700 1 -- 172.21.15.3:0/2082604538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27200ce120 msgr2=0x7f272006e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.974+0000 7f2726def700 1 --2- 172.21.15.3:0/2082604538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27200ce120 0x7f272006e390 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2708009840 tx=0x7f2708009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:05.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.974+0000 7f2726def700 1 -- 172.21.15.3:0/2082604538 shutdown_connections 2024-05-08T21:22:05.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.974+0000 7f2726def700 1 --2- 172.21.15.3:0/2082604538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27200ce120 0x7f272006e390 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.974+0000 7f2726def700 1 -- 172.21.15.3:0/2082604538 >> 172.21.15.3:0/2082604538 conn(0x7f2720099a50 msgr2=0x7f27200d0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.974+0000 7f2726def700 1 -- 172.21.15.3:0/2082604538 shutdown_connections 2024-05-08T21:22:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.974+0000 7f2726def700 1 -- 172.21.15.3:0/2082604538 wait complete. 2024-05-08T21:22:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.975+0000 7f2726def700 1 Processor -- start 2024-05-08T21:22:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.976+0000 7f2726def700 1 -- start start 2024-05-08T21:22:05.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.976+0000 7f2726def700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27200cb2a0 0x7f272015e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.976+0000 7f2726def700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f27200ce120 0x7f27201513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.976+0000 7f2726def700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f272015db00 0x7f272014a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.976+0000 7f2726def700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2720153960 con 0x7f272015db00 2024-05-08T21:22:05.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.976+0000 7f2726def700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2720153c40 con 0x7f27200cb2a0 2024-05-08T21:22:05.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.976+0000 7f2726def700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2720153f20 con 0x7f27200ce120 2024-05-08T21:22:05.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.977+0000 7f272538c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f272015db00 0x7f272014a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.977+0000 7f272538c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f272015db00 0x7f272014a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39824/0 (socket says 172.21.15.3:39824) 2024-05-08T21:22:05.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.977+0000 7f272538c700 1 -- 172.21.15.3:0/3020857490 learned_addr learned my addr 172.21.15.3:0/3020857490 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:05.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.977+0000 7f2724b8b700 1 --2- 172.21.15.3:0/3020857490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27200cb2a0 0x7f272015e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.977+0000 7f271ffff700 1 --2- 172.21.15.3:0/3020857490 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f27200ce120 0x7f27201513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.977+0000 7f272538c700 1 -- 172.21.15.3:0/3020857490 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f27200ce120 msgr2=0x7f27201513d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.977+0000 7f272538c700 1 --2- 172.21.15.3:0/3020857490 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f27200ce120 0x7f27201513d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.977+0000 7f272538c700 1 -- 172.21.15.3:0/3020857490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27200cb2a0 msgr2=0x7f272015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:05.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.977+0000 7f272538c700 1 --2- 172.21.15.3:0/3020857490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27200cb2a0 0x7f272015e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:05.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.977+0000 7f272538c700 1 -- 172.21.15.3:0/3020857490 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27080094a0 con 0x7f272015db00 2024-05-08T21:22:05.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.977+0000 7f272538c700 1 --2- 172.21.15.3:0/3020857490 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f272015db00 0x7f272014a180 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f271800cb50 tx=0x7f271800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:05.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.977+0000 7f271d7fa700 1 -- 172.21.15.3:0/3020857490 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2718003c00 con 0x7f272015db00 2024-05-08T21:22:05.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.977+0000 7f271d7fa700 1 -- 172.21.15.3:0/3020857490 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2718003d60 con 0x7f272015db00 2024-05-08T21:22:05.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.977+0000 7f271d7fa700 1 -- 172.21.15.3:0/3020857490 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271800aac0 con 0x7f272015db00 2024-05-08T21:22:05.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.977+0000 7f2726def700 1 -- 172.21.15.3:0/3020857490 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2720154200 con 0x7f272015db00 2024-05-08T21:22:05.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.977+0000 7f2726def700 1 -- 172.21.15.3:0/3020857490 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2720151910 con 0x7f272015db00 2024-05-08T21:22:05.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.979+0000 7f2726def700 1 -- 172.21.15.3:0/3020857490 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27201666c0 con 0x7f272015db00 2024-05-08T21:22:05.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.979+0000 7f271d7fa700 1 -- 172.21.15.3:0/3020857490 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f271800ac20 con 0x7f272015db00 2024-05-08T21:22:05.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.979+0000 7f271d7fa700 1 --2- 172.21.15.3:0/3020857490 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f270006ac90 0x7f270006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:05.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.980+0000 7f271d7fa700 1 -- 172.21.15.3:0/3020857490 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2718089310 con 0x7f272015db00 2024-05-08T21:22:05.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.980+0000 7f2724b8b700 1 --2- 172.21.15.3:0/3020857490 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f270006ac90 0x7f270006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:05.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.980+0000 7f2724b8b700 1 --2- 172.21.15.3:0/3020857490 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f270006ac90 0x7f270006d150 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f27200b63e0 tx=0x7f2710001e40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:05.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:05.984+0000 7f271d7fa700 1 -- 172.21.15.3:0/3020857490 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f27180544f0 con 0x7f272015db00 2024-05-08T21:22:06.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.203+0000 7f2726def700 1 -- 172.21.15.3:0/3020857490 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2720151b00 con 0x7f272015db00 2024-05-08T21:22:06.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.204+0000 7f271d7fa700 1 -- 172.21.15.3:0/3020857490 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15989 (secure 0 0 0) 0x7f2718057b10 con 0x7f272015db00 2024-05-08T21:22:06.206 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:06.206 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:53.064342+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.207+0000 7f2726def700 1 -- 172.21.15.3:0/3020857490 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f270006ac90 msgr2=0x7f270006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.207+0000 7f2726def700 1 --2- 172.21.15.3:0/3020857490 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f270006ac90 0x7f270006d150 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f27200b63e0 tx=0x7f2710001e40 comp rx=0 tx=0).stop 2024-05-08T21:22:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.207+0000 7f2726def700 1 -- 172.21.15.3:0/3020857490 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f272015db00 msgr2=0x7f272014a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.207+0000 7f2726def700 1 --2- 172.21.15.3:0/3020857490 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f272015db00 0x7f272014a180 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f271800cb50 tx=0x7f271800ce60 comp rx=0 tx=0).stop 2024-05-08T21:22:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.207+0000 7f2726def700 1 -- 172.21.15.3:0/3020857490 shutdown_connections 2024-05-08T21:22:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.207+0000 7f2726def700 1 --2- 172.21.15.3:0/3020857490 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f270006ac90 0x7f270006d150 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:06.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.207+0000 7f2726def700 1 --2- 172.21.15.3:0/3020857490 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f272015db00 0x7f272014a180 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.207+0000 7f2726def700 1 --2- 172.21.15.3:0/3020857490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27200cb2a0 0x7f272015e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.207+0000 7f2726def700 1 --2- 172.21.15.3:0/3020857490 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f27200ce120 0x7f27201513d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.207+0000 7f2726def700 1 -- 172.21.15.3:0/3020857490 >> 172.21.15.3:0/3020857490 conn(0x7f2720099a50 msgr2=0x7f27200d0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.207+0000 7f2726def700 1 -- 172.21.15.3:0/3020857490 shutdown_connections 2024-05-08T21:22:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.207+0000 7f2726def700 1 -- 172.21.15.3:0/3020857490 wait complete. 2024-05-08T21:22:06.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:22:06.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.357+0000 7f862dcc9700 1 Processor -- start 2024-05-08T21:22:06.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.358+0000 7f862dcc9700 1 -- start start 2024-05-08T21:22:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.358+0000 7f862dcc9700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f862815db20 0x7f86280af640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.358+0000 7f862dcc9700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86280cf410 con 0x7f862806b070 2024-05-08T21:22:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.358+0000 7f862dcc9700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86280ad0f0 con 0x7f86280b2960 2024-05-08T21:22:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.358+0000 7f862dcc9700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862815df00 con 0x7f862815db20 2024-05-08T21:22:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.359+0000 7f86277fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f862815db20 0x7f86280af640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.359+0000 7f8627fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f862806b070 0x7f86280ad580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58064/0 (socket says 172.21.15.3:58064) 2024-05-08T21:22:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.359+0000 7f86277fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f862815db20 0x7f86280af640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39834/0 (socket says 172.21.15.3:39834) 2024-05-08T21:22:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.359+0000 7f8627fff700 1 -- 172.21.15.3:0/2362869661 learned_addr learned my addr 172.21.15.3:0/2362869661 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.359+0000 7f86277fe700 1 -- 172.21.15.3:0/2362869661 >> v1:172.21.15.186:6789/0 conn(0x7f86280b2960 legacy=0x7f862806a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.359+0000 7f86277fe700 1 -- 172.21.15.3:0/2362869661 >> v1:172.21.15.3:6789/0 conn(0x7f862806b070 legacy=0x7f86280ad580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.359+0000 7f86277fe700 1 -- 172.21.15.3:0/2362869661 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86280cf640 con 0x7f862815db20 2024-05-08T21:22:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.359+0000 7f86267fc700 1 -- 172.21.15.3:0/2362869661 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984299257 0 0) 0x7f86280cf410 con 0x7f862806b070 2024-05-08T21:22:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.359+0000 7f86277fe700 1 --2- 172.21.15.3:0/2362869661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f862815db20 0x7f86280af640 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8618009840 tx=0x7f8618009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591b9487defbe950 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.359+0000 7f86267fc700 1 -- 172.21.15.3:0/2362869661 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8618011830 con 0x7f862815db20 2024-05-08T21:22:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.359+0000 7f86267fc700 1 -- 172.21.15.3:0/2362869661 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8618011990 con 0x7f862815db20 2024-05-08T21:22:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.359+0000 7f86267fc700 1 -- 172.21.15.3:0/2362869661 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8618011d20 con 0x7f862815db20 2024-05-08T21:22:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.360+0000 7f862dcc9700 1 -- 172.21.15.3:0/2362869661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f862815db20 msgr2=0x7f86280af640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.360+0000 7f862dcc9700 1 --2- 172.21.15.3:0/2362869661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f862815db20 0x7f86280af640 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8618009840 tx=0x7f8618009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.360+0000 7f862dcc9700 1 -- 172.21.15.3:0/2362869661 shutdown_connections 2024-05-08T21:22:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.360+0000 7f862dcc9700 1 --2- 172.21.15.3:0/2362869661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f862815db20 0x7f86280af640 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.360+0000 7f862dcc9700 1 -- 172.21.15.3:0/2362869661 >> 172.21.15.3:0/2362869661 conn(0x7f8628099a50 msgr2=0x7f86280b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.360+0000 7f862dcc9700 1 -- 172.21.15.3:0/2362869661 shutdown_connections 2024-05-08T21:22:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.360+0000 7f862dcc9700 1 -- 172.21.15.3:0/2362869661 wait complete. 2024-05-08T21:22:06.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.361+0000 7f862dcc9700 1 Processor -- start 2024-05-08T21:22:06.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.361+0000 7f862dcc9700 1 -- start start 2024-05-08T21:22:06.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.362+0000 7f862dcc9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f862806b070 0x7f86280c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:06.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.362+0000 7f862dcc9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86280b2960 0x7f86280b1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:06.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.362+0000 7f862dcc9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f862815db20 0x7f86281f3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:06.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.362+0000 7f862dcc9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86280c7770 con 0x7f86280b2960 2024-05-08T21:22:06.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.362+0000 7f862dcc9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86280c7a50 con 0x7f862815db20 2024-05-08T21:22:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.362+0000 7f862dcc9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86280c7d30 con 0x7f862806b070 2024-05-08T21:22:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.362+0000 7f8626ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86280b2960 0x7f86280b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.362+0000 7f8626ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86280b2960 0x7f86280b1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39838/0 (socket says 172.21.15.3:39838) 2024-05-08T21:22:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.362+0000 7f8626ffd700 1 -- 172.21.15.3:0/2085511580 learned_addr learned my addr 172.21.15.3:0/2085511580 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.363+0000 7f8627fff700 1 --2- 172.21.15.3:0/2085511580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f862815db20 0x7f86281f3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.363+0000 7f8626ffd700 1 -- 172.21.15.3:0/2085511580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f862806b070 msgr2=0x7f86280c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.363+0000 7f86277fe700 1 --2- 172.21.15.3:0/2085511580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f862806b070 0x7f86280c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.363+0000 7f8626ffd700 1 --2- 172.21.15.3:0/2085511580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f862806b070 0x7f86280c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.363+0000 7f8626ffd700 1 -- 172.21.15.3:0/2085511580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f862815db20 msgr2=0x7f86281f3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.363+0000 7f8626ffd700 1 --2- 172.21.15.3:0/2085511580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f862815db20 0x7f86281f3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.363+0000 7f8626ffd700 1 -- 172.21.15.3:0/2085511580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86180094a0 con 0x7f86280b2960 2024-05-08T21:22:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.363+0000 7f8627fff700 1 --2- 172.21.15.3:0/2085511580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f862815db20 0x7f86281f3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.363+0000 7f8626ffd700 1 --2- 172.21.15.3:0/2085511580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86280b2960 0x7f86280b1040 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f861000c4e0 tx=0x7f861000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.363+0000 7f861ffff700 1 -- 172.21.15.3:0/2085511580 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86100052d0 con 0x7f86280b2960 2024-05-08T21:22:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.363+0000 7f861ffff700 1 -- 172.21.15.3:0/2085511580 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8610005430 con 0x7f86280b2960 2024-05-08T21:22:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.363+0000 7f862dcc9700 1 -- 172.21.15.3:0/2085511580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86280c8010 con 0x7f86280b2960 2024-05-08T21:22:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.363+0000 7f861ffff700 1 -- 172.21.15.3:0/2085511580 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86100038c0 con 0x7f86280b2960 2024-05-08T21:22:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.363+0000 7f862dcc9700 1 -- 172.21.15.3:0/2085511580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86280c73d0 con 0x7f86280b2960 2024-05-08T21:22:06.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.364+0000 7f862dcc9700 1 -- 172.21.15.3:0/2085511580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8628068000 con 0x7f86280b2960 2024-05-08T21:22:06.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.365+0000 7f861ffff700 1 -- 172.21.15.3:0/2085511580 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8610003a20 con 0x7f86280b2960 2024-05-08T21:22:06.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.365+0000 7f861ffff700 1 --2- 172.21.15.3:0/2085511580 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f860806ac90 0x7f860806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:06.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.365+0000 7f861ffff700 1 -- 172.21.15.3:0/2085511580 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86100887f0 con 0x7f86280b2960 2024-05-08T21:22:06.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.365+0000 7f86277fe700 1 --2- 172.21.15.3:0/2085511580 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f860806ac90 0x7f860806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:06.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.366+0000 7f86277fe700 1 --2- 172.21.15.3:0/2085511580 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f860806ac90 0x7f860806d150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8618009b90 tx=0x7f861801c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:06.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.370+0000 7f861ffff700 1 -- 172.21.15.3:0/2085511580 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f86100539d0 con 0x7f86280b2960 2024-05-08T21:22:06.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.633+0000 7f862dcc9700 1 -- 172.21.15.3:0/2085511580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86280ad900 con 0x7f86280b2960 2024-05-08T21:22:06.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.634+0000 7f861ffff700 1 -- 172.21.15.3:0/2085511580 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f8610017090 con 0x7f86280b2960 2024-05-08T21:22:06.636 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:06.636 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/2839854293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2839854293},{"type":"v1","addr":"172.21.15.3:6834","nonce":2839854293}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1878326567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":1878326567},{"type":"v1","addr":"172.21.15.3:6835","nonce":1878326567}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/3624849830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3624849830},{"type":"v1","addr":"172.21.15.186:6836","nonce":3624849830}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:21:52.062548+0000","modified":"2024-05-08T21:21:55.079806+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6837/1462886680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1462886680},{"type":"v1","addr":"172.21.15.186:6837","nonce":1462886680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-05-08T21:22:06.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.636+0000 7f862dcc9700 1 -- 172.21.15.3:0/2085511580 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f860806ac90 msgr2=0x7f860806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:06.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.636+0000 7f862dcc9700 1 --2- 172.21.15.3:0/2085511580 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f860806ac90 0x7f860806d150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8618009b90 tx=0x7f861801c040 comp rx=0 tx=0).stop 2024-05-08T21:22:06.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.636+0000 7f862dcc9700 1 -- 172.21.15.3:0/2085511580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86280b2960 msgr2=0x7f86280b1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:06.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.636+0000 7f862dcc9700 1 --2- 172.21.15.3:0/2085511580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86280b2960 0x7f86280b1040 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f861000c4e0 tx=0x7f861000c8a0 comp rx=0 tx=0).stop 2024-05-08T21:22:06.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.636+0000 7f862dcc9700 1 -- 172.21.15.3:0/2085511580 shutdown_connections 2024-05-08T21:22:06.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.636+0000 7f862dcc9700 1 --2- 172.21.15.3:0/2085511580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f862806b070 0x7f86280c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:06.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.636+0000 7f862dcc9700 1 --2- 172.21.15.3:0/2085511580 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f860806ac90 0x7f860806d150 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:06.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.636+0000 7f862dcc9700 1 --2- 172.21.15.3:0/2085511580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f862815db20 0x7f86281f3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:06.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.636+0000 7f862dcc9700 1 --2- 172.21.15.3:0/2085511580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86280b2960 0x7f86280b1040 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:06.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.636+0000 7f862dcc9700 1 -- 172.21.15.3:0/2085511580 >> 172.21.15.3:0/2085511580 conn(0x7f8628099a50 msgr2=0x7f86280ae6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:06.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.636+0000 7f862dcc9700 1 -- 172.21.15.3:0/2085511580 shutdown_connections 2024-05-08T21:22:06.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:06.636+0000 7f862dcc9700 1 -- 172.21.15.3:0/2085511580 wait complete. 2024-05-08T21:22:06.643 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 8 2024-05-08T21:22:07.054 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-08T21:21:52.062548+0000', 'modified': '2024-05-08T21:21:55.079806+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4294}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4294': {'gid': 4294, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.186:6837/1462886680', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6835', 'nonce': 1462886680}, {'type': 'v1', 'addr': '172.21.15.186:6837', 'nonce': 1462886680}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-05-08T21:22:07.054 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-08T21:22:07.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-05-08T21:22:07.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.200+0000 7f77f287d700 1 Processor -- start 2024-05-08T21:22:07.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.200+0000 7f77f287d700 1 -- start start 2024-05-08T21:22:07.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.201+0000 7f77f287d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77ec15a800 0x7f77ec0cba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.201+0000 7f77f287d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ec15abe0 con 0x7f77ec0cd380 2024-05-08T21:22:07.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.201+0000 7f77f287d700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ec06e8a0 con 0x7f77ec06ed30 2024-05-08T21:22:07.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.201+0000 7f77f287d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec06dc90 con 0x7f77ec15a800 2024-05-08T21:22:07.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.201+0000 7f77f0e1a700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f77ec0cd380 0x7f77ec0ce230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58066/0 (socket says 172.21.15.3:58066) 2024-05-08T21:22:07.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.201+0000 7f77f0e1a700 1 -- 172.21.15.3:0/3643291964 learned_addr learned my addr 172.21.15.3:0/3643291964 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:07.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.201+0000 7f77ebfff700 1 --2- 172.21.15.3:0/3643291964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77ec15a800 0x7f77ec0cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.201+0000 7f77eaffd700 1 -- 172.21.15.3:0/3643291964 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062242455 0 0) 0x7f77e4001490 con 0x7f77ec0cd380 2024-05-08T21:22:07.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.201+0000 7f77eaffd700 1 -- 172.21.15.3:0/3643291964 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77d40035b0 con 0x7f77ec0cd380 2024-05-08T21:22:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.201+0000 7f77eaffd700 1 -- 172.21.15.3:0/3643291964 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790037019 0 0) 0x7f77ec06e8a0 con 0x7f77ec06ed30 2024-05-08T21:22:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.201+0000 7f77eaffd700 1 -- 172.21.15.3:0/3643291964 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77e4001490 con 0x7f77ec06ed30 2024-05-08T21:22:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.201+0000 7f77ebfff700 1 -- 172.21.15.3:0/3643291964 >> v1:172.21.15.186:6789/0 conn(0x7f77ec06ed30 legacy=0x7f77ec0ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.201+0000 7f77eaffd700 1 -- 172.21.15.3:0/3643291964 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3094897237 0 0) 0x7f77d40035b0 con 0x7f77ec0cd380 2024-05-08T21:22:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.201+0000 7f77ebfff700 1 -- 172.21.15.3:0/3643291964 >> v1:172.21.15.3:6789/0 conn(0x7f77ec0cd380 legacy=0x7f77ec0ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.201+0000 7f77ebfff700 1 -- 172.21.15.3:0/3643291964 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ec06df70 con 0x7f77ec15a800 2024-05-08T21:22:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.202+0000 7f77ebfff700 1 --2- 172.21.15.3:0/3643291964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77ec15a800 0x7f77ec0cba50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009870 tx=0x7f77dc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ad015cfdb506d57 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.202+0000 7f77eaffd700 1 -- 172.21.15.3:0/3643291964 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e4001b80 con 0x7f77ec15a800 2024-05-08T21:22:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.202+0000 7f77eaffd700 1 -- 172.21.15.3:0/3643291964 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77dc004030 con 0x7f77ec15a800 2024-05-08T21:22:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.202+0000 7f77eaffd700 1 -- 172.21.15.3:0/3643291964 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc0043c0 con 0x7f77ec15a800 2024-05-08T21:22:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.202+0000 7f77f287d700 1 -- 172.21.15.3:0/3643291964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77ec15a800 msgr2=0x7f77ec0cba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.202+0000 7f77f287d700 1 --2- 172.21.15.3:0/3643291964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77ec15a800 0x7f77ec0cba50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009870 tx=0x7f77dc009b80 comp rx=0 tx=0).stop 2024-05-08T21:22:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.202+0000 7f77f287d700 1 -- 172.21.15.3:0/3643291964 shutdown_connections 2024-05-08T21:22:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.202+0000 7f77f287d700 1 --2- 172.21.15.3:0/3643291964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77ec15a800 0x7f77ec0cba50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.202+0000 7f77f287d700 1 -- 172.21.15.3:0/3643291964 >> 172.21.15.3:0/3643291964 conn(0x7f77ec099a50 msgr2=0x7f77ec0d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:07.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.202+0000 7f77f287d700 1 -- 172.21.15.3:0/3643291964 shutdown_connections 2024-05-08T21:22:07.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.202+0000 7f77f287d700 1 -- 172.21.15.3:0/3643291964 wait complete. 2024-05-08T21:22:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.204+0000 7f77f287d700 1 Processor -- start 2024-05-08T21:22:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.204+0000 7f77f287d700 1 -- start start 2024-05-08T21:22:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77f287d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ec06ed30 0x7f77ec154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77f287d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77ec0cd380 0x7f77ec0c2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77f287d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77ec15a800 0x7f77ec0c51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77f287d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec0c3430 con 0x7f77ec06ed30 2024-05-08T21:22:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77f287d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec0b2ca0 con 0x7f77ec0cd380 2024-05-08T21:22:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77f287d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec0b26e0 con 0x7f77ec15a800 2024-05-08T21:22:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77ebfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ec06ed30 0x7f77ec154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77ebfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ec06ed30 0x7f77ec154440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39850/0 (socket says 172.21.15.3:39850) 2024-05-08T21:22:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77ebfff700 1 -- 172.21.15.3:0/1321061782 learned_addr learned my addr 172.21.15.3:0/1321061782 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77eb7fe700 1 --2- 172.21.15.3:0/1321061782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77ec0cd380 0x7f77ec0c2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77f0e1a700 1 --2- 172.21.15.3:0/1321061782 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77ec15a800 0x7f77ec0c51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77ebfff700 1 -- 172.21.15.3:0/1321061782 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77ec15a800 msgr2=0x7f77ec0c51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:07.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77ebfff700 1 --2- 172.21.15.3:0/1321061782 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77ec15a800 0x7f77ec0c51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77ebfff700 1 -- 172.21.15.3:0/1321061782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77ec0cd380 msgr2=0x7f77ec0c2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:07.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77ebfff700 1 --2- 172.21.15.3:0/1321061782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77ec0cd380 0x7f77ec0c2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77ebfff700 1 -- 172.21.15.3:0/1321061782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77dc0094a0 con 0x7f77ec06ed30 2024-05-08T21:22:07.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77ebfff700 1 --2- 172.21.15.3:0/1321061782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ec06ed30 0x7f77ec154440 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f77dc000c00 tx=0x7f77dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:07.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77e8ff9700 1 -- 172.21.15.3:0/1321061782 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc003db0 con 0x7f77ec06ed30 2024-05-08T21:22:07.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77e8ff9700 1 -- 172.21.15.3:0/1321061782 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77dc01f070 con 0x7f77ec06ed30 2024-05-08T21:22:07.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77f287d700 1 -- 172.21.15.3:0/1321061782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ec0b29c0 con 0x7f77ec06ed30 2024-05-08T21:22:07.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.205+0000 7f77e8ff9700 1 -- 172.21.15.3:0/1321061782 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc011870 con 0x7f77ec06ed30 2024-05-08T21:22:07.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.206+0000 7f77f287d700 1 -- 172.21.15.3:0/1321061782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ec0c4c80 con 0x7f77ec06ed30 2024-05-08T21:22:07.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.207+0000 7f77e8ff9700 1 -- 172.21.15.3:0/1321061782 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f77dc021030 con 0x7f77ec06ed30 2024-05-08T21:22:07.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.207+0000 7f77e8ff9700 1 --2- 172.21.15.3:0/1321061782 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f77d806ac90 0x7f77d806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.208+0000 7f77e8ff9700 1 -- 172.21.15.3:0/1321061782 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77dc08a6f0 con 0x7f77ec06ed30 2024-05-08T21:22:07.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.208+0000 7f77eb7fe700 1 --2- 172.21.15.3:0/1321061782 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f77d806ac90 0x7f77d806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.208+0000 7f77eb7fe700 1 --2- 172.21.15.3:0/1321061782 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f77d806ac90 0x7f77d806d150 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f77e00011e0 tx=0x7f77e0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:07.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.209+0000 7f77fea37b80 1 Processor -- start 2024-05-08T21:22:07.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.209+0000 7f77fea37b80 1 -- start start 2024-05-08T21:22:07.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.210+0000 7f77fea37b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d2750 0x562b4e9ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.210+0000 7f77fea37b80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9ad320 0x562b4e9b0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.210+0000 7f77fea37b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d9d80 0x562b4e9da200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.210+0000 7f77fea37b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4e906100 con 0x562b4e9d9d80 2024-05-08T21:22:07.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.210+0000 7f77fea37b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4e903ac0 con 0x562b4e9d2750 2024-05-08T21:22:07.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.210+0000 7f77fea37b80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x562b4e8ff3b0 con 0x562b4e9ad320 2024-05-08T21:22:07.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.210+0000 7f77f0e1a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d9d80 0x562b4e9da200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.210+0000 7f77f0e1a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d9d80 0x562b4e9da200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39866/0 (socket says 172.21.15.3:39866) 2024-05-08T21:22:07.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.210+0000 7f77f0e1a700 1 -- 172.21.15.3:0/803211658 learned_addr learned my addr 172.21.15.3:0/803211658 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.211+0000 7f77ebfff700 1 --2- 172.21.15.3:0/803211658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d2750 0x562b4e9ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.211+0000 7f77eb7fe700 1 --2- 172.21.15.3:0/803211658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9ad320 0x562b4e9b0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.211+0000 7f77f0e1a700 1 -- 172.21.15.3:0/803211658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9ad320 msgr2=0x562b4e9b0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.211+0000 7f77f0e1a700 1 --2- 172.21.15.3:0/803211658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9ad320 0x562b4e9b0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.211+0000 7f77f0e1a700 1 -- 172.21.15.3:0/803211658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d2750 msgr2=0x562b4e9ab0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.211+0000 7f77f0e1a700 1 --2- 172.21.15.3:0/803211658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d2750 0x562b4e9ab0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.211+0000 7f77f0e1a700 1 -- 172.21.15.3:0/803211658 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b4e780360 con 0x562b4e9d9d80 2024-05-08T21:22:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.211+0000 7f77ebfff700 1 --2- 172.21.15.3:0/803211658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d2750 0x562b4e9ab0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.211+0000 7f77f0e1a700 1 --2- 172.21.15.3:0/803211658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d9d80 0x562b4e9da200 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f77e40071d0 tx=0x7f77e40074e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e72abdf3b45ec0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.211+0000 7f77d1ffb700 1 -- 172.21.15.3:0/803211658 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e40114f0 con 0x562b4e9d9d80 2024-05-08T21:22:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.211+0000 7f77d1ffb700 1 -- 172.21.15.3:0/803211658 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e4011650 con 0x562b4e9d9d80 2024-05-08T21:22:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.211+0000 7f77d1ffb700 1 -- 172.21.15.3:0/803211658 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e4011a00 con 0x562b4e9d9d80 2024-05-08T21:22:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.212+0000 7f77fea37b80 1 -- 172.21.15.3:0/803211658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d9d80 msgr2=0x562b4e9da200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.212+0000 7f77fea37b80 1 --2- 172.21.15.3:0/803211658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d9d80 0x562b4e9da200 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f77e40071d0 tx=0x7f77e40074e0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.212+0000 7f77fea37b80 1 -- 172.21.15.3:0/803211658 shutdown_connections 2024-05-08T21:22:07.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.212+0000 7f77fea37b80 1 --2- 172.21.15.3:0/803211658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d9d80 0x562b4e9da200 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.212+0000 7f77fea37b80 1 --2- 172.21.15.3:0/803211658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d2750 0x562b4e9ab0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.212+0000 7f77fea37b80 1 --2- 172.21.15.3:0/803211658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9ad320 0x562b4e9b0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.212+0000 7f77fea37b80 1 -- 172.21.15.3:0/803211658 >> 172.21.15.3:0/803211658 conn(0x562b4e90d1c0 msgr2=0x562b4e9f4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:07.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.212+0000 7f77fea37b80 1 -- 172.21.15.3:0/803211658 shutdown_connections 2024-05-08T21:22:07.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.212+0000 7f77fea37b80 1 -- 172.21.15.3:0/803211658 wait complete. 2024-05-08T21:22:07.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.212+0000 7f77fea37b80 1 -- start start 2024-05-08T21:22:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.213+0000 7f77fea37b80 1 Processor -- start 2024-05-08T21:22:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.213+0000 7f77fea37b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9ad320 0x562b4e9f8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.213+0000 7f77fea37b80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9d2750 0x562b4e9f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.213+0000 7f77fea37b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d9d80 0x562b4e9f6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.213+0000 7f77fea37b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4e89b510 con 0x562b4e9ad320 2024-05-08T21:22:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.213+0000 7f77fea37b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4e89b1b0 con 0x562b4e9d9d80 2024-05-08T21:22:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77fea37b80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x562b4e782650 con 0x562b4e9d2750 2024-05-08T21:22:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77fea37b80 10 client.0 _ll_register_callbacks cb 0x562b4e8a4ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-08T21:22:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77ebfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9ad320 0x562b4e9f8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77ebfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9ad320 0x562b4e9f8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39878/0 (socket says 172.21.15.3:39878) 2024-05-08T21:22:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77ebfff700 1 -- 172.21.15.3:0/2410950255 learned_addr learned my addr 172.21.15.3:0/2410950255 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77f0e1a700 1 --2- 172.21.15.3:0/2410950255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d9d80 0x562b4e9f6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77eb7fe700 1 --2- 172.21.15.3:0/2410950255 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9d2750 0x562b4e9f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77ebfff700 1 -- 172.21.15.3:0/2410950255 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9d2750 msgr2=0x562b4e9f69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:07.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77ebfff700 1 --2- 172.21.15.3:0/2410950255 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9d2750 0x562b4e9f69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77ebfff700 1 -- 172.21.15.3:0/2410950255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d9d80 msgr2=0x562b4e9f6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:07.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77ebfff700 1 --2- 172.21.15.3:0/2410950255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d9d80 0x562b4e9f6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77ebfff700 1 -- 172.21.15.3:0/2410950255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e400f040 con 0x562b4e9ad320 2024-05-08T21:22:07.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77ebfff700 1 --2- 172.21.15.3:0/2410950255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9ad320 0x562b4e9f8210 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f77dc061440 tx=0x7f77dc059630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ef3a2e8fef5f9f server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:07.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77caffd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-05-08T21:22:07.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77caffd700 1 -- 172.21.15.3:0/2410950255 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc05d030 con 0x562b4e9ad320 2024-05-08T21:22:07.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77caffd700 1 -- 172.21.15.3:0/2410950255 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77dc05d190 con 0x562b4e9ad320 2024-05-08T21:22:07.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77fea37b80 1 -- 172.21.15.3:0/2410950255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562b4e858220 con 0x562b4e9ad320 2024-05-08T21:22:07.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77caffd700 1 -- 172.21.15.3:0/2410950255 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77dc055c90 con 0x562b4e9ad320 2024-05-08T21:22:07.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77fea37b80 10 client.4575 fetch_fsmap learned FSMap version 8 2024-05-08T21:22:07.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.214+0000 7f77fea37b80 1 -- 172.21.15.3:0/2410950255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x562b4e72cb10 con 0x562b4e9ad320 2024-05-08T21:22:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.215+0000 7f77caffd700 1 -- 172.21.15.3:0/2410950255 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562b4e858220 con 0x562b4e9ad320 2024-05-08T21:22:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.215+0000 7f77caffd700 1 -- 172.21.15.3:0/2410950255 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 8) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f77dc074020 con 0x562b4e9ad320 2024-05-08T21:22:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.215+0000 7f77fea37b80 10 client.4575 fetch_fsmap finished waiting for FSMap version 8 2024-05-08T21:22:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.215+0000 7f77fea37b80 10 client.4575 resolve_mds: resolved name 'd' to daemon mds.d 2024-05-08T21:22:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.215+0000 7f77fea37b80 20 client.4575 populate_metadata read hostname 'smithi003' 2024-05-08T21:22:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.215+0000 7f77fea37b80 1 --2- 172.21.15.3:0/2410950255 >> [v2:172.21.15.186:6835/1462886680,v1:172.21.15.186:6837/1462886680] conn(0x562b4e9f38a0 0x562b4e9f3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.215+0000 7f77fea37b80 4 client.4575 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-08T21:22:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.215+0000 7f77fea37b80 1 -- 172.21.15.3:0/2410950255 --> [v2:172.21.15.186:6835/1462886680,v1:172.21.15.186:6837/1462886680] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562b4e97e790 con 0x562b4e9f38a0 2024-05-08T21:22:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.215+0000 7f77eb7fe700 1 --2- 172.21.15.3:0/2410950255 >> [v2:172.21.15.186:6835/1462886680,v1:172.21.15.186:6837/1462886680] conn(0x562b4e9f38a0 0x562b4e9f3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.215+0000 7f77caffd700 10 client.4575 ms_handle_connect on v2:172.21.15.186:6835/1462886680 2024-05-08T21:22:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.215+0000 7f77eb7fe700 1 --2- 172.21.15.3:0/2410950255 >> [v2:172.21.15.186:6835/1462886680,v1:172.21.15.186:6837/1462886680] conn(0x562b4e9f38a0 0x562b4e9f3c80 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24f681b1539fb960 server_cookie=7964ecc13c553933 in_seq=0 out_seq=0 2024-05-08T21:22:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.216+0000 7f77caffd700 1 -- 172.21.15.3:0/2410950255 <== mds.0 v2:172.21.15.186:6835/1462886680 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562b4e97e790 con 0x562b4e9f38a0 2024-05-08T21:22:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.216+0000 7f77caffd700 10 client.4575 handle_command_reply: tid=0 2024-05-08T21:22:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.216+0000 7f77fea37b80 1 client.4575 shutdown 2024-05-08T21:22:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.216+0000 7f77fea37b80 1 -- 172.21.15.3:0/2410950255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9ad320 msgr2=0x562b4e9f8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.216+0000 7f77fea37b80 1 --2- 172.21.15.3:0/2410950255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9ad320 0x562b4e9f8210 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f77dc061440 tx=0x7f77dc059630 comp rx=0 tx=0).stop 2024-05-08T21:22:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.216+0000 7f77fea37b80 1 -- 172.21.15.3:0/2410950255 shutdown_connections 2024-05-08T21:22:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.216+0000 7f77fea37b80 1 --2- 172.21.15.3:0/2410950255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d9d80 0x562b4e9f6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.216+0000 7f77fea37b80 1 --2- 172.21.15.3:0/2410950255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9ad320 0x562b4e9f8210 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.216+0000 7f77fea37b80 1 --2- 172.21.15.3:0/2410950255 >> [v2:172.21.15.186:6835/1462886680,v1:172.21.15.186:6837/1462886680] conn(0x562b4e9f38a0 0x562b4e9f3c80 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.216+0000 7f77fea37b80 1 --2- 172.21.15.3:0/2410950255 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9d2750 0x562b4e9f69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.216+0000 7f77caffd700 0 client.4575 ms_handle_reset on v2:172.21.15.186:6835/1462886680 2024-05-08T21:22:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.216+0000 7f77fea37b80 1 -- 172.21.15.3:0/2410950255 >> 172.21.15.3:0/2410950255 conn(0x562b4e90d1c0 msgr2=0x562b4e9ab080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.217+0000 7f77fea37b80 1 -- 172.21.15.3:0/2410950255 shutdown_connections 2024-05-08T21:22:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.217+0000 7f77fea37b80 1 -- 172.21.15.3:0/2410950255 wait complete. 2024-05-08T21:22:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.217+0000 7f77fea37b80 20 client.4575 trim_cache size 0 max 16384 2024-05-08T21:22:07.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.236+0000 7f77fea37b80 1 Processor -- start 2024-05-08T21:22:07.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.236+0000 7f77fea37b80 1 -- start start 2024-05-08T21:22:07.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77fea37b80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9ad320 0x562b4e9ac760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77fea37b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d2750 0x562b4e9aed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77fea37b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d9d80 0x562b4e9a8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77fea37b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4e9f4000 con 0x562b4e9d9d80 2024-05-08T21:22:07.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77fea37b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4ea093a0 con 0x562b4e9d2750 2024-05-08T21:22:07.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77fea37b80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x562b4e9abc00 con 0x562b4e9ad320 2024-05-08T21:22:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77f0e1a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d9d80 0x562b4e9a8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77f0e1a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d9d80 0x562b4e9a8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39892/0 (socket says 172.21.15.3:39892) 2024-05-08T21:22:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77f0e1a700 1 -- 172.21.15.3:0/1389417656 learned_addr learned my addr 172.21.15.3:0/1389417656 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77eb7fe700 1 --2- 172.21.15.3:0/1389417656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d2750 0x562b4e9aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77ebfff700 1 --2- 172.21.15.3:0/1389417656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9ad320 0x562b4e9ac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77f0e1a700 1 -- 172.21.15.3:0/1389417656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9ad320 msgr2=0x562b4e9ac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77f0e1a700 1 --2- 172.21.15.3:0/1389417656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9ad320 0x562b4e9ac760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77f0e1a700 1 -- 172.21.15.3:0/1389417656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d2750 msgr2=0x562b4e9aed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77f0e1a700 1 --2- 172.21.15.3:0/1389417656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d2750 0x562b4e9aed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77f0e1a700 1 -- 172.21.15.3:0/1389417656 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b4e8c5510 con 0x562b4e9d9d80 2024-05-08T21:22:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77eb7fe700 1 --2- 172.21.15.3:0/1389417656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d2750 0x562b4e9aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77f0e1a700 1 --2- 172.21.15.3:0/1389417656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d9d80 0x562b4e9a8c90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f77e4002230 tx=0x7f77e400c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=403fa122a1da83f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.237+0000 7f77d27fc700 1 -- 172.21.15.3:0/1389417656 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e4002090 con 0x562b4e9d9d80 2024-05-08T21:22:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.238+0000 7f77d27fc700 1 -- 172.21.15.3:0/1389417656 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e400b040 con 0x562b4e9d9d80 2024-05-08T21:22:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.238+0000 7f77d27fc700 1 -- 172.21.15.3:0/1389417656 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e40119e0 con 0x562b4e9d9d80 2024-05-08T21:22:07.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.238+0000 7f77fea37b80 1 -- 172.21.15.3:0/1389417656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d9d80 msgr2=0x562b4e9a8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:07.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.238+0000 7f77fea37b80 1 --2- 172.21.15.3:0/1389417656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d9d80 0x562b4e9a8c90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f77e4002230 tx=0x7f77e400c000 comp rx=0 tx=0).stop 2024-05-08T21:22:07.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.238+0000 7f77fea37b80 1 -- 172.21.15.3:0/1389417656 shutdown_connections 2024-05-08T21:22:07.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.238+0000 7f77fea37b80 1 --2- 172.21.15.3:0/1389417656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9ad320 0x562b4e9ac760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.238+0000 7f77fea37b80 1 --2- 172.21.15.3:0/1389417656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d9d80 0x562b4e9a8c90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.238+0000 7f77fea37b80 1 --2- 172.21.15.3:0/1389417656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9d2750 0x562b4e9aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.238+0000 7f77fea37b80 1 -- 172.21.15.3:0/1389417656 >> 172.21.15.3:0/1389417656 conn(0x562b4e90d1c0 msgr2=0x562b4e9d9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:07.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.238+0000 7f77fea37b80 1 -- 172.21.15.3:0/1389417656 shutdown_connections 2024-05-08T21:22:07.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.238+0000 7f77fea37b80 1 -- 172.21.15.3:0/1389417656 wait complete. 2024-05-08T21:22:07.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.239+0000 7f77fea37b80 1 -- start start 2024-05-08T21:22:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.239+0000 7f77fea37b80 1 Processor -- start 2024-05-08T21:22:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.240+0000 7f77fea37b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9ad320 0x562b4ea01470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.240+0000 7f77fea37b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d2750 0x562b4e9aec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.240+0000 7f77fea37b80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9d9d80 0x562b4e9af140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.240+0000 7f77fea37b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4ea129a0 con 0x562b4e9d2750 2024-05-08T21:22:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.240+0000 7f77fea37b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4e9aa7d0 con 0x562b4e9ad320 2024-05-08T21:22:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.240+0000 7f77fea37b80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x562b4ea01e10 con 0x562b4e9d9d80 2024-05-08T21:22:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77fea37b80 10 client.0 _ll_register_callbacks cb 0x562b4e8a4ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-08T21:22:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77eb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d2750 0x562b4e9aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77eb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d2750 0x562b4e9aec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39898/0 (socket says 172.21.15.3:39898) 2024-05-08T21:22:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77eb7fe700 1 -- 172.21.15.3:0/2248143364 learned_addr learned my addr 172.21.15.3:0/2248143364 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77ebfff700 1 --2- 172.21.15.3:0/2248143364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9ad320 0x562b4ea01470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77f0e1a700 1 --2- 172.21.15.3:0/2248143364 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9d9d80 0x562b4e9af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77eb7fe700 1 -- 172.21.15.3:0/2248143364 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9d9d80 msgr2=0x562b4e9af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77eb7fe700 1 --2- 172.21.15.3:0/2248143364 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9d9d80 0x562b4e9af140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77eb7fe700 1 -- 172.21.15.3:0/2248143364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9ad320 msgr2=0x562b4ea01470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77eb7fe700 1 --2- 172.21.15.3:0/2248143364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9ad320 0x562b4ea01470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77eb7fe700 1 -- 172.21.15.3:0/2248143364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562b4e780360 con 0x562b4e9d2750 2024-05-08T21:22:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77eb7fe700 1 --2- 172.21.15.3:0/2248143364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d2750 0x562b4e9aec00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f77e0009010 tx=0x7f77e0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec0542a6eaddad2 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77caffd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-05-08T21:22:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77caffd700 1 -- 172.21.15.3:0/2248143364 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e0003890 con 0x562b4e9d2750 2024-05-08T21:22:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77caffd700 1 -- 172.21.15.3:0/2248143364 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77e002eb20 con 0x562b4e9d2750 2024-05-08T21:22:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77fea37b80 1 -- 172.21.15.3:0/2248143364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f77e4011650 con 0x562b4e9d2750 2024-05-08T21:22:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77caffd700 1 -- 172.21.15.3:0/2248143364 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77e002ece0 con 0x562b4e9d2750 2024-05-08T21:22:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77fea37b80 10 client.4581 fetch_fsmap learned FSMap version 8 2024-05-08T21:22:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.241+0000 7f77fea37b80 1 -- 172.21.15.3:0/2248143364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x562b4e784c80 con 0x562b4e9d2750 2024-05-08T21:22:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.242+0000 7f77caffd700 1 -- 172.21.15.3:0/2248143364 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e4011650 con 0x562b4e9d2750 2024-05-08T21:22:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.242+0000 7f77caffd700 1 -- 172.21.15.3:0/2248143364 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 8) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f77e0020070 con 0x562b4e9d2750 2024-05-08T21:22:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.242+0000 7f77fea37b80 10 client.4581 fetch_fsmap finished waiting for FSMap version 8 2024-05-08T21:22:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.242+0000 7f77fea37b80 10 client.4581 resolve_mds: resolved name 'd' to daemon mds.d 2024-05-08T21:22:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.242+0000 7f77fea37b80 20 client.4581 populate_metadata read hostname 'smithi003' 2024-05-08T21:22:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.242+0000 7f77fea37b80 1 --2- 172.21.15.3:0/2248143364 >> [v2:172.21.15.186:6835/1462886680,v1:172.21.15.186:6837/1462886680] conn(0x562b4ea61950 0x562b4ea61d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.242+0000 7f77fea37b80 4 client.4581 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-08T21:22:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.242+0000 7f77fea37b80 1 -- 172.21.15.3:0/2248143364 --> [v2:172.21.15.186:6835/1462886680,v1:172.21.15.186:6837/1462886680] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562b4e8f7e70 con 0x562b4ea61950 2024-05-08T21:22:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.242+0000 7f77ebfff700 1 --2- 172.21.15.3:0/2248143364 >> [v2:172.21.15.186:6835/1462886680,v1:172.21.15.186:6837/1462886680] conn(0x562b4ea61950 0x562b4ea61d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.242+0000 7f77ebfff700 1 --2- 172.21.15.3:0/2248143364 >> [v2:172.21.15.186:6835/1462886680,v1:172.21.15.186:6837/1462886680] conn(0x562b4ea61950 0x562b4ea61d30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c6146d03b5beb01 server_cookie=917645eb11f9d7ae in_seq=0 out_seq=0 2024-05-08T21:22:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.242+0000 7f77caffd700 10 client.4581 ms_handle_connect on v2:172.21.15.186:6835/1462886680 2024-05-08T21:22:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.243+0000 7f77caffd700 1 -- 172.21.15.3:0/2248143364 <== mds.0 v2:172.21.15.186:6835/1462886680 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x562b4e8f7e70 con 0x562b4ea61950 2024-05-08T21:22:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.243+0000 7f77caffd700 10 client.4581 handle_command_reply: tid=0 2024-05-08T21:22:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.243+0000 7f77fea37b80 1 client.4581 shutdown 2024-05-08T21:22:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.243+0000 7f77fea37b80 1 -- 172.21.15.3:0/2248143364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d2750 msgr2=0x562b4e9aec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.243+0000 7f77fea37b80 1 --2- 172.21.15.3:0/2248143364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d2750 0x562b4e9aec00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f77e0009010 tx=0x7f77e0000ed0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.243+0000 7f77fea37b80 1 -- 172.21.15.3:0/2248143364 shutdown_connections 2024-05-08T21:22:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.243+0000 7f77fea37b80 1 --2- 172.21.15.3:0/2248143364 >> [v2:172.21.15.186:6835/1462886680,v1:172.21.15.186:6837/1462886680] conn(0x562b4ea61950 0x562b4ea61d30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.243+0000 7f77fea37b80 1 --2- 172.21.15.3:0/2248143364 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x562b4e9d9d80 0x562b4e9af140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.243+0000 7f77fea37b80 1 --2- 172.21.15.3:0/2248143364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562b4e9ad320 0x562b4ea01470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.243+0000 7f77caffd700 0 client.4581 ms_handle_reset on v2:172.21.15.186:6835/1462886680 2024-05-08T21:22:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.243+0000 7f77fea37b80 1 --2- 172.21.15.3:0/2248143364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4e9d2750 0x562b4e9aec00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.243+0000 7f77fea37b80 1 -- 172.21.15.3:0/2248143364 >> 172.21.15.3:0/2248143364 conn(0x562b4e90d1c0 msgr2=0x562b4e9d9660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.243+0000 7f77fea37b80 1 -- 172.21.15.3:0/2248143364 shutdown_connections 2024-05-08T21:22:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.243+0000 7f77fea37b80 1 -- 172.21.15.3:0/2248143364 wait complete. 2024-05-08T21:22:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.243+0000 7f77fea37b80 20 client.4581 trim_cache size 0 max 16384 2024-05-08T21:22:07.249 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:22:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "d0f58275-758f-4470-b003-84d6abe25489", 2024-05-08T21:22:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 0, 2024-05-08T21:22:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 4294, 2024-05-08T21:22:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2024-05-08T21:22:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2024-05-08T21:22:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2024-05-08T21:22:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 15.178912479999999, 2024-05-08T21:22:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 8, 2024-05-08T21:22:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 20, 2024-05-08T21:22:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 20, 2024-05-08T21:22:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 20.287073165999999 2024-05-08T21:22:07.254 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:22:07.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.244+0000 7f77f287d700 1 -- 172.21.15.3:0/1321061782 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f77d806ac90 msgr2=0x7f77d806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.244+0000 7f77f287d700 1 --2- 172.21.15.3:0/1321061782 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f77d806ac90 0x7f77d806d150 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f77e00011e0 tx=0x7f77e0009040 comp rx=0 tx=0).stop 2024-05-08T21:22:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.244+0000 7f77f287d700 1 -- 172.21.15.3:0/1321061782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ec06ed30 msgr2=0x7f77ec154440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.244+0000 7f77f287d700 1 --2- 172.21.15.3:0/1321061782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ec06ed30 0x7f77ec154440 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f77dc000c00 tx=0x7f77dc004750 comp rx=0 tx=0).stop 2024-05-08T21:22:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.244+0000 7f77f287d700 1 -- 172.21.15.3:0/1321061782 shutdown_connections 2024-05-08T21:22:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.244+0000 7f77f287d700 1 --2- 172.21.15.3:0/1321061782 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77ec15a800 0x7f77ec0c51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.244+0000 7f77f287d700 1 --2- 172.21.15.3:0/1321061782 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f77d806ac90 0x7f77d806d150 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.244+0000 7f77f287d700 1 --2- 172.21.15.3:0/1321061782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ec06ed30 0x7f77ec154440 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.244+0000 7f77f287d700 1 --2- 172.21.15.3:0/1321061782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77ec0cd380 0x7f77ec0c2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.244+0000 7f77f287d700 1 -- 172.21.15.3:0/1321061782 >> 172.21.15.3:0/1321061782 conn(0x7f77ec099a50 msgr2=0x7f77ec0ade50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.244+0000 7f77f287d700 1 -- 172.21.15.3:0/1321061782 shutdown_connections 2024-05-08T21:22:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.244+0000 7f77f287d700 1 -- 172.21.15.3:0/1321061782 wait complete. 2024-05-08T21:22:07.259 INFO:teuthology.run_tasks:Running task sequential... 2024-05-08T21:22:07.271 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-08T21:22:07.271 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-08T21:22:07.272 INFO:teuthology.task.print:Enabling mgr modules 2024-05-08T21:22:07.272 INFO:teuthology.run_tasks:Running task kclient... 2024-05-08T21:22:07.308 INFO:tasks.kclient:Mounting kernel clients... 2024-05-08T21:22:07.308 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-05-08T21:22:07.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:07.308 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2024-05-08T21:22:07.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:07.328 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link delete ceph-brx 2024-05-08T21:22:07.405 INFO:teuthology.orchestra.run.smithi003.stderr:Cannot find device "ceph-brx" 2024-05-08T21:22:07.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T21:22:07.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:07.408 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2024-05-08T21:22:07.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:07.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link delete ceph-brx 2024-05-08T21:22:07.547 INFO:teuthology.orchestra.run.smithi003.stderr:Cannot find device "ceph-brx" 2024-05-08T21:22:07.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T21:22:07.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:07.549 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2024-05-08T21:22:07.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:07.571 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link delete ceph-brx 2024-05-08T21:22:07.646 INFO:teuthology.orchestra.run.smithi186.stderr:Cannot find device "ceph-brx" 2024-05-08T21:22:07.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T21:22:07.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:07.648 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2024-05-08T21:22:07.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:07.708 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link delete ceph-brx 2024-05-08T21:22:07.779 INFO:teuthology.orchestra.run.smithi186.stderr:Cannot find device "ceph-brx" 2024-05-08T21:22:07.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T21:22:07.780 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-05-08T21:22:07.780 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-08T21:22:07.781 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-08T21:22:07.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-08T21:22:07.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.923+0000 7f730d568700 1 Processor -- start 2024-05-08T21:22:07.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.923+0000 7f730d568700 1 -- start start 2024-05-08T21:22:07.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.924+0000 7f730d568700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f73080ac670 0x7f73080b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.924+0000 7f730d568700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73080acac0 con 0x7f730815b7b0 2024-05-08T21:22:07.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.924+0000 7f730d568700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730815f190 con 0x7f73080ac1c0 2024-05-08T21:22:07.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.924+0000 7f730d568700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73081600f0 con 0x7f73080ac670 2024-05-08T21:22:07.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.924+0000 7f73077fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f73080ac670 0x7f73080b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.924+0000 7f7306ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f730815b7b0 0x7f73080cb3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58074/0 (socket says 172.21.15.3:58074) 2024-05-08T21:22:07.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.924+0000 7f7306ffd700 1 -- 172.21.15.3:0/2688341557 learned_addr learned my addr 172.21.15.3:0/2688341557 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:07.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.924+0000 7f73077fe700 1 --2- 172.21.15.3:0/2688341557 >> v2:172.21.15.3:3300/0 conn(0x7f73080ac670 0x7f73080b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39910/0 (socket says 172.21.15.3:39910) 2024-05-08T21:22:07.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.924+0000 7f73077fe700 1 -- 172.21.15.3:0/2688341557 >> v1:172.21.15.186:6789/0 conn(0x7f73080ac1c0 legacy=0x7f73080b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:07.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.924+0000 7f73077fe700 1 -- 172.21.15.3:0/2688341557 >> v1:172.21.15.3:6789/0 conn(0x7f730815b7b0 legacy=0x7f73080cb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:07.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.924+0000 7f73077fe700 1 -- 172.21.15.3:0/2688341557 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730815f470 con 0x7f73080ac670 2024-05-08T21:22:07.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.924+0000 7f7305ffb700 1 -- 172.21.15.3:0/2688341557 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921859680 0 0) 0x7f73080acac0 con 0x7f730815b7b0 2024-05-08T21:22:07.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.925+0000 7f73077fe700 1 --2- 172.21.15.3:0/2688341557 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73080ac670 0x7f73080b2b30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7300009d90 tx=0x7f730000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=819bbfb12a1f4ca7 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:07.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.925+0000 7f7305ffb700 1 -- 172.21.15.3:0/2688341557 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730000e070 con 0x7f73080ac670 2024-05-08T21:22:07.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.925+0000 7f7305ffb700 1 -- 172.21.15.3:0/2688341557 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730000bbd0 con 0x7f73080ac670 2024-05-08T21:22:07.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.925+0000 7f7305ffb700 1 -- 172.21.15.3:0/2688341557 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7300012430 con 0x7f73080ac670 2024-05-08T21:22:07.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.925+0000 7f730d568700 1 -- 172.21.15.3:0/2688341557 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73080ac670 msgr2=0x7f73080b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:07.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.925+0000 7f730d568700 1 --2- 172.21.15.3:0/2688341557 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73080ac670 0x7f73080b2b30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7300009d90 tx=0x7f730000b330 comp rx=0 tx=0).stop 2024-05-08T21:22:07.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.925+0000 7f730d568700 1 -- 172.21.15.3:0/2688341557 shutdown_connections 2024-05-08T21:22:07.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.925+0000 7f730d568700 1 --2- 172.21.15.3:0/2688341557 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73080ac670 0x7f73080b2b30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.925+0000 7f730d568700 1 -- 172.21.15.3:0/2688341557 >> 172.21.15.3:0/2688341557 conn(0x7f73080cf270 msgr2=0x7f73080cf650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:07.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.925+0000 7f730d568700 1 -- 172.21.15.3:0/2688341557 shutdown_connections 2024-05-08T21:22:07.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.925+0000 7f730d568700 1 -- 172.21.15.3:0/2688341557 wait complete. 2024-05-08T21:22:07.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.927+0000 7f730d568700 1 Processor -- start 2024-05-08T21:22:07.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.927+0000 7f730d568700 1 -- start start 2024-05-08T21:22:07.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f730d568700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73080ac1c0 0x7f7308151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f730d568700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73080ac670 0x7f73080cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f730d568700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73080cd210 0x7f730815bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f730d568700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73080cd6c0 con 0x7f73080ac670 2024-05-08T21:22:07.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f730d568700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73081524a0 con 0x7f73080cd210 2024-05-08T21:22:07.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f730d568700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730815ca20 con 0x7f73080ac1c0 2024-05-08T21:22:07.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f73067fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73080ac670 0x7f73080cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f73067fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73080ac670 0x7f73080cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39926/0 (socket says 172.21.15.3:39926) 2024-05-08T21:22:07.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f73067fc700 1 -- 172.21.15.3:0/2223985059 learned_addr learned my addr 172.21.15.3:0/2223985059 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:07.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f73077fe700 1 --2- 172.21.15.3:0/2223985059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73080cd210 0x7f730815bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f7306ffd700 1 --2- 172.21.15.3:0/2223985059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73080ac1c0 0x7f7308151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f73067fc700 1 -- 172.21.15.3:0/2223985059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73080ac1c0 msgr2=0x7f7308151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:07.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f73067fc700 1 --2- 172.21.15.3:0/2223985059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73080ac1c0 0x7f7308151f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f73067fc700 1 -- 172.21.15.3:0/2223985059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73080cd210 msgr2=0x7f730815bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:07.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f73067fc700 1 --2- 172.21.15.3:0/2223985059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73080cd210 0x7f730815bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:07.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f73067fc700 1 -- 172.21.15.3:0/2223985059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7300009a70 con 0x7f73080ac670 2024-05-08T21:22:07.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.928+0000 7f73067fc700 1 --2- 172.21.15.3:0/2223985059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73080ac670 0x7f73080cccd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f72f000c480 tx=0x7f72f000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:07.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.929+0000 7f72ff7fe700 1 -- 172.21.15.3:0/2223985059 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f00052d0 con 0x7f73080ac670 2024-05-08T21:22:07.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.929+0000 7f72ff7fe700 1 -- 172.21.15.3:0/2223985059 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f0005430 con 0x7f73080ac670 2024-05-08T21:22:07.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.929+0000 7f730d568700 1 -- 172.21.15.3:0/2223985059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7308141aa0 con 0x7f73080ac670 2024-05-08T21:22:07.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.929+0000 7f72ff7fe700 1 -- 172.21.15.3:0/2223985059 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f00038c0 con 0x7f73080ac670 2024-05-08T21:22:07.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.929+0000 7f730d568700 1 -- 172.21.15.3:0/2223985059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7308142060 con 0x7f73080ac670 2024-05-08T21:22:07.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.930+0000 7f730d568700 1 -- 172.21.15.3:0/2223985059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73080678f0 con 0x7f73080ac670 2024-05-08T21:22:07.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.930+0000 7f72ff7fe700 1 -- 172.21.15.3:0/2223985059 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f72f0003a20 con 0x7f73080ac670 2024-05-08T21:22:07.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.931+0000 7f72ff7fe700 1 --2- 172.21.15.3:0/2223985059 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f72e806ac90 0x7f72e806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:07.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.931+0000 7f72ff7fe700 1 -- 172.21.15.3:0/2223985059 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72f00886b0 con 0x7f73080ac670 2024-05-08T21:22:07.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.931+0000 7f7306ffd700 1 --2- 172.21.15.3:0/2223985059 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f72e806ac90 0x7f72e806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:07.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.931+0000 7f7306ffd700 1 --2- 172.21.15.3:0/2223985059 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f72e806ac90 0x7f72e806d150 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f72f8007fd0 tx=0x7f72f8007c20 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:07.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:07.935+0000 7f72ff7fe700 1 -- 172.21.15.3:0/2223985059 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f72f0053920 con 0x7f73080ac670 2024-05-08T21:22:08.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:08.203+0000 7f730d568700 1 -- 172.21.15.3:0/2223985059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f73080cfed0 con 0x7f73080ac670 2024-05-08T21:22:08.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:08.203+0000 7f72ff7fe700 1 -- 172.21.15.3:0/2223985059 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f72f0056f40 con 0x7f73080ac670 2024-05-08T21:22:08.205 INFO:teuthology.orchestra.run.smithi003.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-08T21:22:08.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:08.205+0000 7f730d568700 1 -- 172.21.15.3:0/2223985059 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f72e806ac90 msgr2=0x7f72e806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:08.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:08.205+0000 7f730d568700 1 --2- 172.21.15.3:0/2223985059 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f72e806ac90 0x7f72e806d150 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f72f8007fd0 tx=0x7f72f8007c20 comp rx=0 tx=0).stop 2024-05-08T21:22:08.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:08.205+0000 7f730d568700 1 -- 172.21.15.3:0/2223985059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73080ac670 msgr2=0x7f73080cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:08.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:08.205+0000 7f730d568700 1 --2- 172.21.15.3:0/2223985059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73080ac670 0x7f73080cccd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f72f000c480 tx=0x7f72f000c840 comp rx=0 tx=0).stop 2024-05-08T21:22:08.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:08.205+0000 7f730d568700 1 -- 172.21.15.3:0/2223985059 shutdown_connections 2024-05-08T21:22:08.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:08.205+0000 7f730d568700 1 --2- 172.21.15.3:0/2223985059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73080ac1c0 0x7f7308151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:08.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:08.205+0000 7f730d568700 1 --2- 172.21.15.3:0/2223985059 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f72e806ac90 0x7f72e806d150 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:08.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:08.205+0000 7f730d568700 1 --2- 172.21.15.3:0/2223985059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73080cd210 0x7f730815bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:08.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:08.205+0000 7f730d568700 1 --2- 172.21.15.3:0/2223985059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73080ac670 0x7f73080cccd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:08.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:08.205+0000 7f730d568700 1 -- 172.21.15.3:0/2223985059 >> 172.21.15.3:0/2223985059 conn(0x7f73080cf270 msgr2=0x7f73080af6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:08.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:08.205+0000 7f730d568700 1 -- 172.21.15.3:0/2223985059 shutdown_connections 2024-05-08T21:22:08.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:08.205+0000 7f730d568700 1 -- 172.21.15.3:0/2223985059 wait complete. 2024-05-08T21:22:08.221 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-08T21:22:08.221 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-08T21:22:08.221 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi003.front.sepia.ceph.com 2024-05-08T21:22:08.221 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-08T21:22:08.221 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-08T21:22:08.221 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-08T21:22:08.221 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-08T21:22:08.221 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-08T21:22:08.221 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-08T21:22:08.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:08.221 DEBUG:teuthology.orchestra.run.smithi003:> ip addr 2024-05-08T21:22:08.240 INFO:teuthology.orchestra.run.smithi003.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-08T21:22:08.240 INFO:teuthology.orchestra.run.smithi003.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-08T21:22:08.241 INFO:teuthology.orchestra.run.smithi003.stdout: inet 127.0.0.1/8 scope host lo 2024-05-08T21:22:08.241 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:08.241 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 ::1/128 scope host 2024-05-08T21:22:08.241 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:08.241 INFO:teuthology.orchestra.run.smithi003.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:22:08.241 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ac brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:08.241 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f0 2024-05-08T21:22:08.241 INFO:teuthology.orchestra.run.smithi003.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:22:08.241 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ad brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:08.241 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f1 2024-05-08T21:22:08.241 INFO:teuthology.orchestra.run.smithi003.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-08T21:22:08.241 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dc brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:08.241 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f0 2024-05-08T21:22:08.241 INFO:teuthology.orchestra.run.smithi003.stdout: inet 172.21.15.3/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-08T21:22:08.242 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft 42204sec preferred_lft 42204sec 2024-05-08T21:22:08.242 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::ec4:7aff:febd:15dc/64 scope link 2024-05-08T21:22:08.242 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:08.242 INFO:teuthology.orchestra.run.smithi003.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:22:08.242 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dd brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:08.242 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f1 2024-05-08T21:22:08.242 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-08T21:22:08.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:08.242 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2024-05-08T21:22:08.243 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-05-08T21:22:08.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link add name ceph-brx type bridge 2024-05-08T21:22:08.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip addr flush dev ceph-brx 2024-05-08T21:22:08.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set ceph-brx up 2024-05-08T21:22:08.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-08T21:22:08.243 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-05-08T21:22:08.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:08.416 DEBUG:teuthology.orchestra.run.smithi003:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-08T21:22:08.449 INFO:teuthology.orchestra.run.smithi003.stdout:1 2024-05-08T21:22:08.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:08.450 DEBUG:teuthology.orchestra.run.smithi003:> route 2024-05-08T21:22:08.522 INFO:teuthology.orchestra.run.smithi003.stdout:Kernel IP routing table 2024-05-08T21:22:08.522 INFO:teuthology.orchestra.run.smithi003.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-08T21:22:08.522 INFO:teuthology.orchestra.run.smithi003.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-08T21:22:08.522 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-08T21:22:08.522 INFO:teuthology.orchestra.run.smithi003.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-08T21:22:08.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:08.524 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2024-05-08T21:22:08.524 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-05-08T21:22:08.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-08T21:22:08.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-08T21:22:08.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-08T21:22:08.524 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-05-08T21:22:08.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:08.797 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2024-05-08T21:22:08.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:08.818 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list-id 2024-05-08T21:22:08.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:08.880 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2024-05-08T21:22:08.880 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-05-08T21:22:08.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-08T21:22:08.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-08T21:22:08.880 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-05-08T21:22:08.990 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-08T21:22:08.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:08.991 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2024-05-08T21:22:08.991 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-05-08T21:22:08.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-08T21:22:08.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-08T21:22:08.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-08T21:22:08.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-08T21:22:08.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-08T21:22:08.991 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-05-08T21:22:09.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:09.231 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2024-05-08T21:22:09.231 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-05-08T21:22:09.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set brx.0 up 2024-05-08T21:22:09.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set dev brx.0 master ceph-brx 2024-05-08T21:22:09.231 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-05-08T21:22:09.343 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-08T21:22:09.343 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-08T21:22:09.362 INFO:teuthology.orchestra.run.smithi003.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-08T21:22:09.362 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-08T21:22:09.362 DEBUG:teuthology.orchestra.run.smithi003:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-08T21:22:09.422 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-05-08T21:22:09.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:09.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-08T21:22:09.521 INFO:teuthology.orchestra.run.smithi003.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-08T21:22:09.521 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-05-08T21:22:09.521 INFO:teuthology.orchestra.run.smithi003.stdout:invalid new device string format 2024-05-08T21:22:09.597 INFO:teuthology.orchestra.run.smithi003.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-08T21:22:09.597 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-05-08T21:22:09.597 INFO:teuthology.orchestra.run.smithi003.stdout:invalid new device string format 2024-05-08T21:22:09.597 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: resolved to: "172.21.15.3:3300,172.21.15.186:3300,172.21.15.186:3301" 2024-05-08T21:22:09.597 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-08T21:22:09.617 INFO:teuthology.orchestra.run.smithi003.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-08T21:22:09.617 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-05-08T21:22:09.617 INFO:teuthology.orchestra.run.smithi003.stdout:invalid new device string format 2024-05-08T21:22:09.617 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: resolved to: "172.21.15.3:3300,172.21.15.186:3300,172.21.15.186:3301" 2024-05-08T21:22:09.617 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-08T21:22:09.617 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: resolved to: "172.21.15.3:6789,172.21.15.186:6789,172.21.15.186:6790" 2024-05-08T21:22:09.618 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.3:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2024-05-08T21:22:09.618 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=d0f58275-758f-4470-b003-84d6abe25489" will pass to kernel 2024-05-08T21:22:09.620 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-08T21:22:09.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:09.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-08T21:22:09.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:09.653 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-08T21:22:09.748 INFO:teuthology.orchestra.run.smithi003.stdout:client4481 2024-05-08T21:22:09.748 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:22:09.748 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:09.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:09.769 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-08T21:22:09.867 INFO:teuthology.orchestra.run.smithi003.stdout:client4481 2024-05-08T21:22:09.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:09.867 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-08T21:22:09.924 INFO:teuthology.orchestra.run.smithi003.stdout:client4481 2024-05-08T21:22:09.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:09.925 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/d0f58275-758f-4470-b003-84d6abe25489.client4481/status') 2024-05-08T21:22:09.991 INFO:teuthology.orchestra.run.smithi003.stdout:instance: client.4481 (3)172.21.15.3:0/351518333 2024-05-08T21:22:09.991 INFO:teuthology.orchestra.run.smithi003.stdout:blocklisted: false 2024-05-08T21:22:09.991 INFO:teuthology.orchestra.run.smithi003.stderr:0+1 records in 2024-05-08T21:22:09.991 INFO:teuthology.orchestra.run.smithi003.stderr:0+1 records out 2024-05-08T21:22:09.991 INFO:teuthology.orchestra.run.smithi003.stderr:68 bytes copied, 2.9125e-05 s, 2.3 MB/s 2024-05-08T21:22:09.992 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-05-08T21:22:09.993 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-08T21:22:09.993 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-05-08T21:22:09.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-08T21:22:10.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.135+0000 7fdcc03c3700 1 Processor -- start 2024-05-08T21:22:10.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.135+0000 7fdcc03c3700 1 -- start start 2024-05-08T21:22:10.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.136+0000 7fdcc03c3700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdcb815a680 0x7fdcb815b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:10.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.136+0000 7fdcc03c3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdcb806dd40 0x7fdcb80d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:10.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.136+0000 7fdcc03c3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdcb80d0830 0x7fdcb80cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:10.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.136+0000 7fdcc03c3700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb80d0c80 con 0x7fdcb806dd40 2024-05-08T21:22:10.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.136+0000 7fdcc03c3700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb80ce670 con 0x7fdcb815a680 2024-05-08T21:22:10.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.136+0000 7fdcc03c3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb815bca0 con 0x7fdcb80d0830 2024-05-08T21:22:10.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.136+0000 7fdcbd95e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdcb806dd40 0x7fdcb80d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:10.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.136+0000 7fdcbd95e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdcb806dd40 0x7fdcb80d02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39942/0 (socket says 172.21.15.3:39942) 2024-05-08T21:22:10.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.136+0000 7fdcbd95e700 1 -- 172.21.15.3:0/650192142 learned_addr learned my addr 172.21.15.3:0/650192142 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:10.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.136+0000 7fdcbe960700 1 --2- 172.21.15.3:0/650192142 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdcb80d0830 0x7fdcb80cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:10.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.136+0000 7fdcbe15f700 1 --2- 172.21.15.3:0/650192142 >> v2:172.21.15.186:3300/0 conn(0x7fdcb815a680 0x7fdcb815b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:10.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.136+0000 7fdcbd95e700 1 -- 172.21.15.3:0/650192142 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdcb80d0830 msgr2=0x7fdcb80cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:10.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.136+0000 7fdcbd95e700 1 --2- 172.21.15.3:0/650192142 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdcb80d0830 0x7fdcb80cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:10.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.136+0000 7fdcbd95e700 1 -- 172.21.15.3:0/650192142 >> v2:172.21.15.186:3300/0 conn(0x7fdcb815a680 msgr2=0x7fdcb815b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:10.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.136+0000 7fdcbd95e700 1 --2- 172.21.15.3:0/650192142 >> v2:172.21.15.186:3300/0 conn(0x7fdcb815a680 0x7fdcb815b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:10.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.136+0000 7fdcbd95e700 1 -- 172.21.15.3:0/650192142 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb806fd90 con 0x7fdcb806dd40 2024-05-08T21:22:10.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.137+0000 7fdcbd95e700 1 --2- 172.21.15.3:0/650192142 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb806dd40 0x7fdcb80d02f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8071920 tx=0x7fdca8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ffac81492ed885 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:10.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.137+0000 7fdcbd15d700 1 -- 172.21.15.3:0/650192142 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca8011830 con 0x7fdcb806dd40 2024-05-08T21:22:10.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.137+0000 7fdcbd15d700 1 -- 172.21.15.3:0/650192142 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca8011990 con 0x7fdcb806dd40 2024-05-08T21:22:10.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.137+0000 7fdcbd15d700 1 -- 172.21.15.3:0/650192142 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca8011d20 con 0x7fdcb806dd40 2024-05-08T21:22:10.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.137+0000 7fdcc03c3700 1 -- 172.21.15.3:0/650192142 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb806dd40 msgr2=0x7fdcb80d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:10.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.137+0000 7fdcc03c3700 1 --2- 172.21.15.3:0/650192142 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb806dd40 0x7fdcb80d02f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8071920 tx=0x7fdca8009bd0 comp rx=0 tx=0).stop 2024-05-08T21:22:10.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.137+0000 7fdcc03c3700 1 -- 172.21.15.3:0/650192142 shutdown_connections 2024-05-08T21:22:10.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.138+0000 7fdcc03c3700 1 --2- 172.21.15.3:0/650192142 >> v2:172.21.15.186:3300/0 conn(0x7fdcb815a680 0x7fdcb815b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:10.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.138+0000 7fdcc03c3700 1 --2- 172.21.15.3:0/650192142 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdcb80d0830 0x7fdcb80cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:10.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.138+0000 7fdcc03c3700 1 --2- 172.21.15.3:0/650192142 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb806dd40 0x7fdcb80d02f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:10.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.138+0000 7fdcc03c3700 1 -- 172.21.15.3:0/650192142 >> 172.21.15.3:0/650192142 conn(0x7fdcb80b5810 msgr2=0x7fdcb80b5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:10.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.138+0000 7fdcc03c3700 1 -- 172.21.15.3:0/650192142 shutdown_connections 2024-05-08T21:22:10.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.138+0000 7fdcc03c3700 1 -- 172.21.15.3:0/650192142 wait complete. 2024-05-08T21:22:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.139+0000 7fdcc03c3700 1 Processor -- start 2024-05-08T21:22:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.139+0000 7fdcc03c3700 1 -- start start 2024-05-08T21:22:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.140+0000 7fdcc03c3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb806dd40 0x7fdcb80b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.140+0000 7fdcc03c3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdcb80d0830 0x7fdcb8146160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.140+0000 7fdcc03c3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdcb815a680 0x7fdcb80b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.140+0000 7fdcc03c3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb80b3a40 con 0x7fdcb806dd40 2024-05-08T21:22:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.140+0000 7fdcc03c3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb80b2460 con 0x7fdcb80d0830 2024-05-08T21:22:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.140+0000 7fdcc03c3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb80b2740 con 0x7fdcb815a680 2024-05-08T21:22:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.140+0000 7fdcbe15f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb806dd40 0x7fdcb80b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.140+0000 7fdcbe15f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb806dd40 0x7fdcb80b3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39954/0 (socket says 172.21.15.3:39954) 2024-05-08T21:22:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.140+0000 7fdcbe15f700 1 -- 172.21.15.3:0/882523156 learned_addr learned my addr 172.21.15.3:0/882523156 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.140+0000 7fdcbd95e700 1 --2- 172.21.15.3:0/882523156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdcb80d0830 0x7fdcb8146160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.140+0000 7fdcbe960700 1 --2- 172.21.15.3:0/882523156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdcb815a680 0x7fdcb80b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.140+0000 7fdcbe15f700 1 -- 172.21.15.3:0/882523156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdcb815a680 msgr2=0x7fdcb80b4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.140+0000 7fdcbe15f700 1 --2- 172.21.15.3:0/882523156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdcb815a680 0x7fdcb80b4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.140+0000 7fdcbe15f700 1 -- 172.21.15.3:0/882523156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdcb80d0830 msgr2=0x7fdcb8146160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.140+0000 7fdcbe15f700 1 --2- 172.21.15.3:0/882523156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdcb80d0830 0x7fdcb8146160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.140+0000 7fdcbe15f700 1 -- 172.21.15.3:0/882523156 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca8009550 con 0x7fdcb806dd40 2024-05-08T21:22:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.141+0000 7fdcbe15f700 1 --2- 172.21.15.3:0/882523156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb806dd40 0x7fdcb80b3500 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400e870 tx=0x7fdcb400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.141+0000 7fdcaeffd700 1 -- 172.21.15.3:0/882523156 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4009590 con 0x7fdcb806dd40 2024-05-08T21:22:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.141+0000 7fdcaeffd700 1 -- 172.21.15.3:0/882523156 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb4013680 con 0x7fdcb806dd40 2024-05-08T21:22:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.141+0000 7fdcaeffd700 1 -- 172.21.15.3:0/882523156 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb40139f0 con 0x7fdcb806dd40 2024-05-08T21:22:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.141+0000 7fdcc03c3700 1 -- 172.21.15.3:0/882523156 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb80b2a20 con 0x7fdcb806dd40 2024-05-08T21:22:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.141+0000 7fdcc03c3700 1 -- 172.21.15.3:0/882523156 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb80b2fe0 con 0x7fdcb806dd40 2024-05-08T21:22:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.142+0000 7fdcc03c3700 1 -- 172.21.15.3:0/882523156 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb80b3350 con 0x7fdcb806dd40 2024-05-08T21:22:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.142+0000 7fdcaeffd700 1 -- 172.21.15.3:0/882523156 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdcb4013b50 con 0x7fdcb806dd40 2024-05-08T21:22:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.143+0000 7fdcaeffd700 1 --2- 172.21.15.3:0/882523156 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdc9c06ac90 0x7fdc9c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.143+0000 7fdcaeffd700 1 -- 172.21.15.3:0/882523156 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdcb40890f0 con 0x7fdcb806dd40 2024-05-08T21:22:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.143+0000 7fdcbd95e700 1 --2- 172.21.15.3:0/882523156 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdc9c06ac90 0x7fdc9c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.144+0000 7fdcbd95e700 1 --2- 172.21.15.3:0/882523156 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdc9c06ac90 0x7fdc9c06d150 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdca8009c10 tx=0x7fdca8020040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:10.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.148+0000 7fdcaeffd700 1 -- 172.21.15.3:0/882523156 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fdcb4054360 con 0x7fdcb806dd40 2024-05-08T21:22:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.418+0000 7fdcc03c3700 1 -- 172.21.15.3:0/882523156 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdcb80d0160 con 0x7fdcb806dd40 2024-05-08T21:22:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.419+0000 7fdcaeffd700 1 -- 172.21.15.3:0/882523156 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fdcb4057980 con 0x7fdcb806dd40 2024-05-08T21:22:10.420 INFO:teuthology.orchestra.run.smithi003.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-08T21:22:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.421+0000 7fdcc03c3700 1 -- 172.21.15.3:0/882523156 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdc9c06ac90 msgr2=0x7fdc9c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.421+0000 7fdcc03c3700 1 --2- 172.21.15.3:0/882523156 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdc9c06ac90 0x7fdc9c06d150 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdca8009c10 tx=0x7fdca8020040 comp rx=0 tx=0).stop 2024-05-08T21:22:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.421+0000 7fdcc03c3700 1 -- 172.21.15.3:0/882523156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb806dd40 msgr2=0x7fdcb80b3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.421+0000 7fdcc03c3700 1 --2- 172.21.15.3:0/882523156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb806dd40 0x7fdcb80b3500 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400e870 tx=0x7fdcb400eb80 comp rx=0 tx=0).stop 2024-05-08T21:22:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.421+0000 7fdcc03c3700 1 -- 172.21.15.3:0/882523156 shutdown_connections 2024-05-08T21:22:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.421+0000 7fdcc03c3700 1 --2- 172.21.15.3:0/882523156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdcb815a680 0x7fdcb80b4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.421+0000 7fdcc03c3700 1 --2- 172.21.15.3:0/882523156 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdc9c06ac90 0x7fdc9c06d150 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.421+0000 7fdcc03c3700 1 --2- 172.21.15.3:0/882523156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb806dd40 0x7fdcb80b3500 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.421+0000 7fdcc03c3700 1 --2- 172.21.15.3:0/882523156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdcb80d0830 0x7fdcb8146160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.421+0000 7fdcc03c3700 1 -- 172.21.15.3:0/882523156 >> 172.21.15.3:0/882523156 conn(0x7fdcb80b5810 msgr2=0x7fdcb80aff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.421+0000 7fdcc03c3700 1 -- 172.21.15.3:0/882523156 shutdown_connections 2024-05-08T21:22:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:10.421+0000 7fdcc03c3700 1 -- 172.21.15.3:0/882523156 wait complete. 2024-05-08T21:22:10.436 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-08T21:22:10.436 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-08T21:22:10.436 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi003.front.sepia.ceph.com 2024-05-08T21:22:10.436 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-05-08T21:22:10.436 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-05-08T21:22:10.436 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-08T21:22:10.437 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-08T21:22:10.437 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-08T21:22:10.437 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-05-08T21:22:10.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:10.437 DEBUG:teuthology.orchestra.run.smithi003:> ip addr 2024-05-08T21:22:10.457 INFO:teuthology.orchestra.run.smithi003.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-08T21:22:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-08T21:22:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: inet 127.0.0.1/8 scope host lo 2024-05-08T21:22:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 ::1/128 scope host 2024-05-08T21:22:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:10.457 INFO:teuthology.orchestra.run.smithi003.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:22:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ac brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f0 2024-05-08T21:22:10.457 INFO:teuthology.orchestra.run.smithi003.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:22:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ad brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f1 2024-05-08T21:22:10.458 INFO:teuthology.orchestra.run.smithi003.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-08T21:22:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dc brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f0 2024-05-08T21:22:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: inet 172.21.15.3/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-08T21:22:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft 42201sec preferred_lft 42201sec 2024-05-08T21:22:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::ec4:7aff:febd:15dc/64 scope link 2024-05-08T21:22:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:10.458 INFO:teuthology.orchestra.run.smithi003.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:22:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dd brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f1 2024-05-08T21:22:10.458 INFO:teuthology.orchestra.run.smithi003.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-08T21:22:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether ee:e2:a2:93:f0:94 brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-08T21:22:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:10.459 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::a89e:98ff:fec6:3929/64 scope link 2024-05-08T21:22:10.459 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:10.459 INFO:teuthology.orchestra.run.smithi003.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-08T21:22:10.459 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether ee:e2:a2:93:f0:94 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-08T21:22:10.459 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::ece2:a2ff:fe93:f094/64 scope link tentative 2024-05-08T21:22:10.459 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:10.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:10.459 DEBUG:teuthology.orchestra.run.smithi003:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-08T21:22:10.528 INFO:teuthology.orchestra.run.smithi003.stdout:1 2024-05-08T21:22:10.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:10.530 DEBUG:teuthology.orchestra.run.smithi003:> route 2024-05-08T21:22:10.595 INFO:teuthology.orchestra.run.smithi003.stdout:Kernel IP routing table 2024-05-08T21:22:10.595 INFO:teuthology.orchestra.run.smithi003.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-08T21:22:10.595 INFO:teuthology.orchestra.run.smithi003.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-08T21:22:10.595 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-08T21:22:10.596 INFO:teuthology.orchestra.run.smithi003.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-08T21:22:10.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:10.596 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2024-05-08T21:22:10.596 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-05-08T21:22:10.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-08T21:22:10.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-08T21:22:10.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-08T21:22:10.596 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-05-08T21:22:10.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:10.720 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2024-05-08T21:22:10.741 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-08T21:22:10.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:10.741 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list-id 2024-05-08T21:22:10.803 INFO:teuthology.orchestra.run.smithi003.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-05-08T21:22:10.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:10.804 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2024-05-08T21:22:10.804 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-05-08T21:22:10.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-08T21:22:10.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-05-08T21:22:10.804 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-05-08T21:22:10.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:10.914 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-05-08T21:22:10.955 INFO:teuthology.orchestra.run.smithi003.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-08T21:22:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-08T21:22:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: inet 127.0.0.1/8 scope host lo 2024-05-08T21:22:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 ::1/128 scope host 2024-05-08T21:22:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:10.956 INFO:teuthology.orchestra.run.smithi003.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-08T21:22:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether ce:ad:87:dd:e1:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-08T21:22:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-08T21:22:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::ccad:87ff:fedd:e157/64 scope link tentative 2024-05-08T21:22:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:10.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:10.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-05-08T21:22:11.055 INFO:teuthology.orchestra.run.smithi003.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-08T21:22:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-08T21:22:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: inet 127.0.0.1/8 scope host lo 2024-05-08T21:22:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 ::1/128 scope host 2024-05-08T21:22:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:11.056 INFO:teuthology.orchestra.run.smithi003.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-08T21:22:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether ce:ad:87:dd:e1:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-08T21:22:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-08T21:22:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::ccad:87ff:fedd:e157/64 scope link 2024-05-08T21:22:11.057 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:11.068 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-05-08T21:22:11.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:11.068 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2024-05-08T21:22:11.068 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-05-08T21:22:11.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-05-08T21:22:11.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-05-08T21:22:11.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-05-08T21:22:11.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-05-08T21:22:11.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-05-08T21:22:11.069 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-05-08T21:22:11.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:11.294 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2024-05-08T21:22:11.294 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-05-08T21:22:11.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set brx.1 up 2024-05-08T21:22:11.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set dev brx.1 master ceph-brx 2024-05-08T21:22:11.294 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-05-08T21:22:11.406 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-08T21:22:11.406 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-05-08T21:22:11.425 INFO:teuthology.orchestra.run.smithi003.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-05-08T21:22:11.425 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-08T21:22:11.426 DEBUG:teuthology.orchestra.run.smithi003:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-05-08T21:22:11.486 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-05-08T21:22:11.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:11.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-05-08T21:22:11.584 INFO:teuthology.orchestra.run.smithi003.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-05-08T21:22:11.584 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-05-08T21:22:11.584 INFO:teuthology.orchestra.run.smithi003.stdout:invalid new device string format 2024-05-08T21:22:11.594 INFO:teuthology.orchestra.run.smithi003.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-05-08T21:22:11.595 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-05-08T21:22:11.595 INFO:teuthology.orchestra.run.smithi003.stdout:invalid new device string format 2024-05-08T21:22:11.595 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: resolved to: "172.21.15.3:3300,172.21.15.186:3300,172.21.15.186:3301" 2024-05-08T21:22:11.595 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-08T21:22:11.614 INFO:teuthology.orchestra.run.smithi003.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-05-08T21:22:11.614 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-05-08T21:22:11.614 INFO:teuthology.orchestra.run.smithi003.stdout:invalid new device string format 2024-05-08T21:22:11.615 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: resolved to: "172.21.15.3:3300,172.21.15.186:3300,172.21.15.186:3301" 2024-05-08T21:22:11.615 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-08T21:22:11.615 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: resolved to: "172.21.15.3:6789,172.21.15.186:6789,172.21.15.186:6790" 2024-05-08T21:22:11.615 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.3:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2024-05-08T21:22:11.615 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=d0f58275-758f-4470-b003-84d6abe25489" will pass to kernel 2024-05-08T21:22:11.617 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-08T21:22:11.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:11.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-05-08T21:22:11.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:11.649 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-08T21:22:11.730 INFO:teuthology.orchestra.run.smithi003.stdout:client4490 2024-05-08T21:22:11.730 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:22:11.730 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:11.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:11.792 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-08T21:22:11.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-08T21:22:11.874 INFO:teuthology.orchestra.run.smithi003.stdout:client4490 2024-05-08T21:22:11.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:11.875 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-08T21:22:11.876 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-08T21:22:11.954 INFO:teuthology.orchestra.run.smithi003.stdout:client4490 2024-05-08T21:22:11.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:11.954 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/d0f58275-758f-4470-b003-84d6abe25489.client4490/status') 2024-05-08T21:22:12.050 INFO:teuthology.orchestra.run.smithi003.stdout:instance: client.4490 (3)172.21.15.3:0/2420810529 2024-05-08T21:22:12.050 INFO:teuthology.orchestra.run.smithi003.stdout:blocklisted: false 2024-05-08T21:22:12.050 INFO:teuthology.orchestra.run.smithi003.stderr:0+1 records in 2024-05-08T21:22:12.051 INFO:teuthology.orchestra.run.smithi003.stderr:0+1 records out 2024-05-08T21:22:12.051 INFO:teuthology.orchestra.run.smithi003.stderr:69 bytes copied, 3.0132e-05 s, 2.3 MB/s 2024-05-08T21:22:12.052 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-05-08T21:22:12.052 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-08T21:22:12.052 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-08T21:22:12.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-08T21:22:12.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.193+0000 7f432a40b700 1 Processor -- start 2024-05-08T21:22:12.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.193+0000 7f432a40b700 1 -- start start 2024-05-08T21:22:12.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.194+0000 7f432a40b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43240af030 0x7f432415e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:12.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.194+0000 7f432a40b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f432406e4d0 0x7f432406a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:12.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.194+0000 7f432a40b700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43240b4a80 con 0x7f43240b3990 2024-05-08T21:22:12.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.194+0000 7f432a40b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f432415ede0 con 0x7f432406e4d0 2024-05-08T21:22:12.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.194+0000 7f432a40b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432406d800 con 0x7f43240af030 2024-05-08T21:22:12.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.194+0000 7f43289a8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f432406e4d0 0x7f432406a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:12.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.194+0000 7f43289a8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f432406e4d0 0x7f432406a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39886/0 (socket says 172.21.15.3:39886) 2024-05-08T21:22:12.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.194+0000 7f43289a8700 1 -- 172.21.15.3:0/1909584530 learned_addr learned my addr 172.21.15.3:0/1909584530 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:12.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.194+0000 7f4323fff700 1 --2- 172.21.15.3:0/1909584530 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43240af030 0x7f432415e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:12.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.194+0000 7f43289a8700 1 -- 172.21.15.3:0/1909584530 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43240af030 msgr2=0x7f432415e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:12.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.194+0000 7f43289a8700 1 --2- 172.21.15.3:0/1909584530 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43240af030 0x7f432415e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:12.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.194+0000 7f43289a8700 1 -- 172.21.15.3:0/1909584530 >> v1:172.21.15.186:6789/0 conn(0x7f43240b3990 legacy=0x7f432406ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:12.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.194+0000 7f43289a8700 1 -- 172.21.15.3:0/1909584530 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43240b3e70 con 0x7f432406e4d0 2024-05-08T21:22:12.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.194+0000 7f4323fff700 1 --2- 172.21.15.3:0/1909584530 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43240af030 0x7f432415e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:12.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.195+0000 7f43289a8700 1 --2- 172.21.15.3:0/1909584530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f432406e4d0 0x7f432406a960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f431c009e50 tx=0x7f431c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994b2767c4e604d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:12.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.195+0000 7f4322ffd700 1 -- 172.21.15.3:0/1909584530 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c00e070 con 0x7f432406e4d0 2024-05-08T21:22:12.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.195+0000 7f4322ffd700 1 -- 172.21.15.3:0/1909584530 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431c00b9c0 con 0x7f432406e4d0 2024-05-08T21:22:12.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.195+0000 7f4322ffd700 1 -- 172.21.15.3:0/1909584530 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c00bd70 con 0x7f432406e4d0 2024-05-08T21:22:12.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.195+0000 7f432a40b700 1 -- 172.21.15.3:0/1909584530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f432406e4d0 msgr2=0x7f432406a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:12.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.195+0000 7f432a40b700 1 --2- 172.21.15.3:0/1909584530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f432406e4d0 0x7f432406a960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f431c009e50 tx=0x7f431c00b330 comp rx=0 tx=0).stop 2024-05-08T21:22:12.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.195+0000 7f432a40b700 1 -- 172.21.15.3:0/1909584530 shutdown_connections 2024-05-08T21:22:12.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.195+0000 7f432a40b700 1 --2- 172.21.15.3:0/1909584530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f432406e4d0 0x7f432406a960 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:12.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.195+0000 7f432a40b700 1 --2- 172.21.15.3:0/1909584530 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43240af030 0x7f432415e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:12.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.195+0000 7f432a40b700 1 -- 172.21.15.3:0/1909584530 >> 172.21.15.3:0/1909584530 conn(0x7f43240cfce0 msgr2=0x7f43240d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:12.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.195+0000 7f432a40b700 1 -- 172.21.15.3:0/1909584530 shutdown_connections 2024-05-08T21:22:12.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.195+0000 7f432a40b700 1 -- 172.21.15.3:0/1909584530 wait complete. 2024-05-08T21:22:12.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.197+0000 7f432a40b700 1 Processor -- start 2024-05-08T21:22:12.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.197+0000 7f432a40b700 1 -- start start 2024-05-08T21:22:12.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.198+0000 7f432a40b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432406e4d0 0x7f43240ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:12.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.198+0000 7f432a40b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43240af030 0x7f43240b1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:12.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.198+0000 7f432a40b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43240b3990 0x7f432406b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:12.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.198+0000 7f432a40b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43240b0d20 con 0x7f43240af030 2024-05-08T21:22:12.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.198+0000 7f432a40b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43240cc0e0 con 0x7f432406e4d0 2024-05-08T21:22:12.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.198+0000 7f432a40b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43240cc3c0 con 0x7f43240b3990 2024-05-08T21:22:12.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.198+0000 7f43237fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43240af030 0x7f43240b1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:12.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.198+0000 7f43237fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43240af030 0x7f43240b1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39888/0 (socket says 172.21.15.3:39888) 2024-05-08T21:22:12.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.198+0000 7f43237fe700 1 -- 172.21.15.3:0/3731558966 learned_addr learned my addr 172.21.15.3:0/3731558966 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:12.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.198+0000 7f4323fff700 1 --2- 172.21.15.3:0/3731558966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432406e4d0 0x7f43240ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:12.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.198+0000 7f43289a8700 1 --2- 172.21.15.3:0/3731558966 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43240b3990 0x7f432406b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:12.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.198+0000 7f43237fe700 1 -- 172.21.15.3:0/3731558966 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43240b3990 msgr2=0x7f432406b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:12.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.198+0000 7f43237fe700 1 --2- 172.21.15.3:0/3731558966 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43240b3990 0x7f432406b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:12.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.198+0000 7f43237fe700 1 -- 172.21.15.3:0/3731558966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432406e4d0 msgr2=0x7f43240ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:12.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.198+0000 7f43237fe700 1 --2- 172.21.15.3:0/3731558966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432406e4d0 0x7f43240ccbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:12.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.198+0000 7f43237fe700 1 -- 172.21.15.3:0/3731558966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c009b30 con 0x7f43240af030 2024-05-08T21:22:12.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.199+0000 7f43237fe700 1 --2- 172.21.15.3:0/3731558966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43240af030 0x7f43240b1210 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f430c00c4e0 tx=0x7f430c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:12.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.199+0000 7f4320ff9700 1 -- 172.21.15.3:0/3731558966 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430c0052d0 con 0x7f43240af030 2024-05-08T21:22:12.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.199+0000 7f4320ff9700 1 -- 172.21.15.3:0/3731558966 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430c005430 con 0x7f43240af030 2024-05-08T21:22:12.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.199+0000 7f432a40b700 1 -- 172.21.15.3:0/3731558966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43240cc6a0 con 0x7f43240af030 2024-05-08T21:22:12.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.199+0000 7f4320ff9700 1 -- 172.21.15.3:0/3731558966 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430c0038c0 con 0x7f43240af030 2024-05-08T21:22:12.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.199+0000 7f432a40b700 1 -- 172.21.15.3:0/3731558966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432406b6f0 con 0x7f43240af030 2024-05-08T21:22:12.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.200+0000 7f432a40b700 1 -- 172.21.15.3:0/3731558966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43240cbf40 con 0x7f43240af030 2024-05-08T21:22:12.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.200+0000 7f4320ff9700 1 -- 172.21.15.3:0/3731558966 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f430c003a20 con 0x7f43240af030 2024-05-08T21:22:12.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.201+0000 7f4320ff9700 1 --2- 172.21.15.3:0/3731558966 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f430406ac90 0x7f430406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:12.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.201+0000 7f4320ff9700 1 -- 172.21.15.3:0/3731558966 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f430c0886c0 con 0x7f43240af030 2024-05-08T21:22:12.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.201+0000 7f4323fff700 1 --2- 172.21.15.3:0/3731558966 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f430406ac90 0x7f430406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:12.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.202+0000 7f4323fff700 1 --2- 172.21.15.3:0/3731558966 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f430406ac90 0x7f430406d150 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f43140099d0 tx=0x7f4314008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:12.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.206+0000 7f4320ff9700 1 -- 172.21.15.3:0/3731558966 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f43240cbf40 con 0x7f43240af030 2024-05-08T21:22:12.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.471+0000 7f432a40b700 1 -- 172.21.15.3:0/3731558966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f43240cbf40 con 0x7f43240af030 2024-05-08T21:22:12.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.472+0000 7f4320ff9700 1 -- 172.21.15.3:0/3731558966 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f430c017060 con 0x7f43240af030 2024-05-08T21:22:12.474 INFO:teuthology.orchestra.run.smithi003.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-08T21:22:12.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.474+0000 7f432a40b700 1 -- 172.21.15.3:0/3731558966 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f430406ac90 msgr2=0x7f430406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:12.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.474+0000 7f432a40b700 1 --2- 172.21.15.3:0/3731558966 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f430406ac90 0x7f430406d150 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f43140099d0 tx=0x7f4314008040 comp rx=0 tx=0).stop 2024-05-08T21:22:12.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.474+0000 7f432a40b700 1 -- 172.21.15.3:0/3731558966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43240af030 msgr2=0x7f43240b1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:12.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.474+0000 7f432a40b700 1 --2- 172.21.15.3:0/3731558966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43240af030 0x7f43240b1210 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f430c00c4e0 tx=0x7f430c00c7f0 comp rx=0 tx=0).stop 2024-05-08T21:22:12.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.474+0000 7f432a40b700 1 -- 172.21.15.3:0/3731558966 shutdown_connections 2024-05-08T21:22:12.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.474+0000 7f432a40b700 1 --2- 172.21.15.3:0/3731558966 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43240b3990 0x7f432406b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:12.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.474+0000 7f432a40b700 1 --2- 172.21.15.3:0/3731558966 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f430406ac90 0x7f430406d150 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:12.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.474+0000 7f432a40b700 1 --2- 172.21.15.3:0/3731558966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432406e4d0 0x7f43240ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:12.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.474+0000 7f432a40b700 1 --2- 172.21.15.3:0/3731558966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43240af030 0x7f43240b1210 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:12.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.474+0000 7f432a40b700 1 -- 172.21.15.3:0/3731558966 >> 172.21.15.3:0/3731558966 conn(0x7f43240cfce0 msgr2=0x7f43240d00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:12.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.474+0000 7f432a40b700 1 -- 172.21.15.3:0/3731558966 shutdown_connections 2024-05-08T21:22:12.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:12.474+0000 7f432a40b700 1 -- 172.21.15.3:0/3731558966 wait complete. 2024-05-08T21:22:12.490 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-08T21:22:12.490 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-08T21:22:12.490 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi186.front.sepia.ceph.com 2024-05-08T21:22:12.490 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-05-08T21:22:12.490 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-08T21:22:12.490 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-08T21:22:12.490 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-08T21:22:12.491 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-08T21:22:12.491 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-05-08T21:22:12.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:12.491 DEBUG:teuthology.orchestra.run.smithi186:> ip addr 2024-05-08T21:22:12.512 INFO:teuthology.orchestra.run.smithi186.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-08T21:22:12.512 INFO:teuthology.orchestra.run.smithi186.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-08T21:22:12.512 INFO:teuthology.orchestra.run.smithi186.stdout: inet 127.0.0.1/8 scope host lo 2024-05-08T21:22:12.512 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:12.512 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 ::1/128 scope host 2024-05-08T21:22:12.512 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:12.512 INFO:teuthology.orchestra.run.smithi186.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:22:12.512 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:d9:38:2c brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:12.512 INFO:teuthology.orchestra.run.smithi186.stdout: altname enp7s0f0 2024-05-08T21:22:12.512 INFO:teuthology.orchestra.run.smithi186.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:22:12.513 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:d9:38:2d brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:12.513 INFO:teuthology.orchestra.run.smithi186.stdout: altname enp7s0f1 2024-05-08T21:22:12.513 INFO:teuthology.orchestra.run.smithi186.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:22:12.513 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:88:6d:f4 brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:12.513 INFO:teuthology.orchestra.run.smithi186.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-08T21:22:12.513 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:88:6d:f5 brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:12.513 INFO:teuthology.orchestra.run.smithi186.stdout: inet 172.21.15.186/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-08T21:22:12.513 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft 42232sec preferred_lft 42232sec 2024-05-08T21:22:12.513 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 fe80::d996:13e2:ca9:325/64 scope link noprefixroute 2024-05-08T21:22:12.513 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:12.513 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-08T21:22:12.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:12.514 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2024-05-08T21:22:12.514 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-05-08T21:22:12.514 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link add name ceph-brx type bridge 2024-05-08T21:22:12.514 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip addr flush dev ceph-brx 2024-05-08T21:22:12.514 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link set ceph-brx up 2024-05-08T21:22:12.514 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-08T21:22:12.514 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-05-08T21:22:12.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:12.723 DEBUG:teuthology.orchestra.run.smithi186:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-08T21:22:12.752 INFO:teuthology.orchestra.run.smithi186.stdout:1 2024-05-08T21:22:12.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:12.753 DEBUG:teuthology.orchestra.run.smithi186:> route 2024-05-08T21:22:12.832 INFO:teuthology.orchestra.run.smithi186.stdout:Kernel IP routing table 2024-05-08T21:22:12.832 INFO:teuthology.orchestra.run.smithi186.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-08T21:22:12.833 INFO:teuthology.orchestra.run.smithi186.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-08T21:22:12.833 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-08T21:22:12.833 INFO:teuthology.orchestra.run.smithi186.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-08T21:22:12.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:12.833 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2024-05-08T21:22:12.833 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-05-08T21:22:12.833 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-08T21:22:12.833 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-08T21:22:12.833 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-08T21:22:12.833 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-05-08T21:22:13.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:13.100 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2024-05-08T21:22:13.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:13.118 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list-id 2024-05-08T21:22:13.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:13.176 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2024-05-08T21:22:13.176 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-05-08T21:22:13.176 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-08T21:22:13.176 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-05-08T21:22:13.176 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-05-08T21:22:13.272 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-05-08T21:22:13.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:13.272 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2024-05-08T21:22:13.272 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-05-08T21:22:13.272 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-05-08T21:22:13.272 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-08T21:22:13.272 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-05-08T21:22:13.272 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-05-08T21:22:13.272 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-05-08T21:22:13.272 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-05-08T21:22:13.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:13.491 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2024-05-08T21:22:13.491 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-05-08T21:22:13.491 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link set brx.0 up 2024-05-08T21:22:13.491 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link set dev brx.0 master ceph-brx 2024-05-08T21:22:13.492 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-05-08T21:22:13.589 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-08T21:22:13.589 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-05-08T21:22:13.606 INFO:teuthology.orchestra.run.smithi186.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-05-08T21:22:13.606 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-08T21:22:13.606 DEBUG:teuthology.orchestra.run.smithi186:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-05-08T21:22:13.663 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-05-08T21:22:13.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:13.663 DEBUG:teuthology.orchestra.run.smithi186:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-05-08T21:22:13.749 INFO:teuthology.orchestra.run.smithi186.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-05-08T21:22:13.750 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-05-08T21:22:13.750 INFO:teuthology.orchestra.run.smithi186.stdout:invalid new device string format 2024-05-08T21:22:13.826 INFO:teuthology.orchestra.run.smithi186.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-05-08T21:22:13.826 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-05-08T21:22:13.827 INFO:teuthology.orchestra.run.smithi186.stdout:invalid new device string format 2024-05-08T21:22:13.827 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: resolved to: "172.21.15.3:3300,172.21.15.186:3300,172.21.15.186:3301" 2024-05-08T21:22:13.827 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-08T21:22:13.846 INFO:teuthology.orchestra.run.smithi186.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-05-08T21:22:13.846 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-05-08T21:22:13.846 INFO:teuthology.orchestra.run.smithi186.stdout:invalid new device string format 2024-05-08T21:22:13.846 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: resolved to: "172.21.15.3:3300,172.21.15.186:3300,172.21.15.186:3301" 2024-05-08T21:22:13.846 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-08T21:22:13.847 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: resolved to: "172.21.15.3:6789,172.21.15.186:6789,172.21.15.186:6790" 2024-05-08T21:22:13.847 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.3:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2024-05-08T21:22:13.847 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=d0f58275-758f-4470-b003-84d6abe25489" will pass to kernel 2024-05-08T21:22:13.849 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-08T21:22:13.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:13.849 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-05-08T21:22:13.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:13.876 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-08T21:22:13.948 INFO:teuthology.orchestra.run.smithi186.stdout:client4605 2024-05-08T21:22:13.949 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:22:13.949 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:14.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:14.006 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-08T21:22:14.078 INFO:teuthology.orchestra.run.smithi186.stdout:client4605 2024-05-08T21:22:14.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:14.079 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-08T21:22:14.151 INFO:teuthology.orchestra.run.smithi186.stdout:client4605 2024-05-08T21:22:14.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:14.151 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/d0f58275-758f-4470-b003-84d6abe25489.client4605/status') 2024-05-08T21:22:14.232 INFO:teuthology.orchestra.run.smithi186.stdout:instance: client.4605 (3)172.21.15.186:0/1926780937 2024-05-08T21:22:14.232 INFO:teuthology.orchestra.run.smithi186.stdout:blocklisted: false 2024-05-08T21:22:14.232 INFO:teuthology.orchestra.run.smithi186.stderr:0+1 records in 2024-05-08T21:22:14.232 INFO:teuthology.orchestra.run.smithi186.stderr:0+1 records out 2024-05-08T21:22:14.232 INFO:teuthology.orchestra.run.smithi186.stderr:71 bytes copied, 2.2237e-05 s, 3.2 MB/s 2024-05-08T21:22:14.233 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-05-08T21:22:14.233 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-08T21:22:14.234 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-05-08T21:22:14.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-08T21:22:14.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.377+0000 7fc8cd193700 1 Processor -- start 2024-05-08T21:22:14.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.378+0000 7fc8cd193700 1 -- start start 2024-05-08T21:22:14.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.378+0000 7fc8cd193700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8c806bd80 0x7fc8c80d0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:14.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.378+0000 7fc8cd193700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c80cf990 con 0x7fc8c806c3d0 2024-05-08T21:22:14.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.378+0000 7fc8cd193700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c80d04b0 con 0x7fc8c815da10 2024-05-08T21:22:14.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.378+0000 7fc8cd193700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c809ab40 con 0x7fc8c806bd80 2024-05-08T21:22:14.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.379+0000 7fc8c759e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc8c806c3d0 0x7fc8c80cbb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55238/0 (socket says 172.21.15.3:55238) 2024-05-08T21:22:14.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.379+0000 7fc8c759e700 1 -- 172.21.15.3:0/2403297564 learned_addr learned my addr 172.21.15.3:0/2403297564 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:14.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.379+0000 7fc8c659c700 1 --2- 172.21.15.3:0/2403297564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8c806bd80 0x7fc8c80d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:14.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.379+0000 7fc8c5d9b700 1 -- 172.21.15.3:0/2403297564 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567599309 0 0) 0x7fc8c80cf990 con 0x7fc8c806c3d0 2024-05-08T21:22:14.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.379+0000 7fc8c5d9b700 1 -- 172.21.15.3:0/2403297564 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b00035b0 con 0x7fc8c806c3d0 2024-05-08T21:22:14.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.379+0000 7fc8c5d9b700 1 -- 172.21.15.3:0/2403297564 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341398942 0 0) 0x7fc8c80d04b0 con 0x7fc8c815da10 2024-05-08T21:22:14.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.379+0000 7fc8c5d9b700 1 -- 172.21.15.3:0/2403297564 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8c80cf990 con 0x7fc8c815da10 2024-05-08T21:22:14.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.379+0000 7fc8c659c700 1 -- 172.21.15.3:0/2403297564 >> v1:172.21.15.186:6789/0 conn(0x7fc8c815da10 legacy=0x7fc8c809a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:14.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.379+0000 7fc8c659c700 1 -- 172.21.15.3:0/2403297564 >> v1:172.21.15.3:6789/0 conn(0x7fc8c806c3d0 legacy=0x7fc8c80cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:14.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.379+0000 7fc8c659c700 1 -- 172.21.15.3:0/2403297564 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c809ad60 con 0x7fc8c806bd80 2024-05-08T21:22:14.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.379+0000 7fc8c659c700 1 --2- 172.21.15.3:0/2403297564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8c806bd80 0x7fc8c80d0940 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc009870 tx=0x7fc8bc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a09ebbd9a8054b51 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:14.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.380+0000 7fc8c5d9b700 1 -- 172.21.15.3:0/2403297564 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc011830 con 0x7fc8c806bd80 2024-05-08T21:22:14.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.380+0000 7fc8c5d9b700 1 -- 172.21.15.3:0/2403297564 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8bc011990 con 0x7fc8c806bd80 2024-05-08T21:22:14.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.380+0000 7fc8c5d9b700 1 -- 172.21.15.3:0/2403297564 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc011d20 con 0x7fc8c806bd80 2024-05-08T21:22:14.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.380+0000 7fc8cd193700 1 -- 172.21.15.3:0/2403297564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8c806bd80 msgr2=0x7fc8c80d0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:14.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.380+0000 7fc8cd193700 1 --2- 172.21.15.3:0/2403297564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8c806bd80 0x7fc8c80d0940 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc009870 tx=0x7fc8bc009b80 comp rx=0 tx=0).stop 2024-05-08T21:22:14.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.380+0000 7fc8cd193700 1 -- 172.21.15.3:0/2403297564 shutdown_connections 2024-05-08T21:22:14.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.380+0000 7fc8cd193700 1 --2- 172.21.15.3:0/2403297564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8c806bd80 0x7fc8c80d0940 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:14.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.380+0000 7fc8cd193700 1 -- 172.21.15.3:0/2403297564 >> 172.21.15.3:0/2403297564 conn(0x7fc8c80b3010 msgr2=0x7fc8c80b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:14.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.380+0000 7fc8cd193700 1 -- 172.21.15.3:0/2403297564 shutdown_connections 2024-05-08T21:22:14.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.380+0000 7fc8cd193700 1 -- 172.21.15.3:0/2403297564 wait complete. 2024-05-08T21:22:14.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.382+0000 7fc8cd193700 1 Processor -- start 2024-05-08T21:22:14.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.382+0000 7fc8cd193700 1 -- start start 2024-05-08T21:22:14.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.383+0000 7fc8cd193700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8c806bd80 0x7fc8c8152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:14.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.383+0000 7fc8cd193700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8c806c3d0 0x7fc8c80ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:14.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.383+0000 7fc8cd193700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8c815da10 0x7fc8c80cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:14.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.383+0000 7fc8cd193700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c80c5e10 con 0x7fc8c815da10 2024-05-08T21:22:14.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.383+0000 7fc8cd193700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c80b6320 con 0x7fc8c806bd80 2024-05-08T21:22:14.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.383+0000 7fc8cd193700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c8152ac0 con 0x7fc8c806c3d0 2024-05-08T21:22:14.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.383+0000 7fc8c759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8c815da10 0x7fc8c80cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:14.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.384+0000 7fc8c759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8c815da10 0x7fc8c80cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39896/0 (socket says 172.21.15.3:39896) 2024-05-08T21:22:14.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.384+0000 7fc8c759e700 1 -- 172.21.15.3:0/3480438063 learned_addr learned my addr 172.21.15.3:0/3480438063 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:14.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.384+0000 7fc8c6d9d700 1 --2- 172.21.15.3:0/3480438063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8c806bd80 0x7fc8c8152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:14.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.384+0000 7fc8c659c700 1 --2- 172.21.15.3:0/3480438063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8c806c3d0 0x7fc8c80ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:14.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.384+0000 7fc8c759e700 1 -- 172.21.15.3:0/3480438063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8c806c3d0 msgr2=0x7fc8c80ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:14.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.384+0000 7fc8c759e700 1 --2- 172.21.15.3:0/3480438063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8c806c3d0 0x7fc8c80ccc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:14.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.384+0000 7fc8c759e700 1 -- 172.21.15.3:0/3480438063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8c806bd80 msgr2=0x7fc8c8152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:14.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.384+0000 7fc8c759e700 1 --2- 172.21.15.3:0/3480438063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8c806bd80 0x7fc8c8152580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:14.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.384+0000 7fc8c759e700 1 -- 172.21.15.3:0/3480438063 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc0094a0 con 0x7fc8c815da10 2024-05-08T21:22:14.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.384+0000 7fc8c759e700 1 --2- 172.21.15.3:0/3480438063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8c815da10 0x7fc8c80cd140 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000cba0 tx=0x7fc8c000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:14.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.384+0000 7fc8af7fe700 1 -- 172.21.15.3:0/3480438063 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c000aa40 con 0x7fc8c815da10 2024-05-08T21:22:14.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.384+0000 7fc8af7fe700 1 -- 172.21.15.3:0/3480438063 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c000aba0 con 0x7fc8c815da10 2024-05-08T21:22:14.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.384+0000 7fc8af7fe700 1 -- 172.21.15.3:0/3480438063 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0012400 con 0x7fc8c815da10 2024-05-08T21:22:14.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.384+0000 7fc8cd193700 1 -- 172.21.15.3:0/3480438063 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c80c51d0 con 0x7fc8c815da10 2024-05-08T21:22:14.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.384+0000 7fc8cd193700 1 -- 172.21.15.3:0/3480438063 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c80c5750 con 0x7fc8c815da10 2024-05-08T21:22:14.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.385+0000 7fc8cd193700 1 -- 172.21.15.3:0/3480438063 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c80c5f50 con 0x7fc8c815da10 2024-05-08T21:22:14.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.386+0000 7fc8af7fe700 1 -- 172.21.15.3:0/3480438063 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc8c001b3b0 con 0x7fc8c815da10 2024-05-08T21:22:14.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.386+0000 7fc8af7fe700 1 --2- 172.21.15.3:0/3480438063 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc8b406ac90 0x7fc8b406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:14.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.386+0000 7fc8af7fe700 1 -- 172.21.15.3:0/3480438063 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8c0088f20 con 0x7fc8c815da10 2024-05-08T21:22:14.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.387+0000 7fc8c6d9d700 1 --2- 172.21.15.3:0/3480438063 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc8b406ac90 0x7fc8b406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:14.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.387+0000 7fc8c6d9d700 1 --2- 172.21.15.3:0/3480438063 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc8b406ac90 0x7fc8b406d150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc8b80011e0 tx=0x7fc8b8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:14.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.391+0000 7fc8af7fe700 1 -- 172.21.15.3:0/3480438063 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc8c80c5f50 con 0x7fc8c815da10 2024-05-08T21:22:14.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.663+0000 7fc8cd193700 1 -- 172.21.15.3:0/3480438063 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc8c80c5f50 con 0x7fc8c815da10 2024-05-08T21:22:14.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.664+0000 7fc8af7fe700 1 -- 172.21.15.3:0/3480438063 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc8c0017060 con 0x7fc8c815da10 2024-05-08T21:22:14.666 INFO:teuthology.orchestra.run.smithi003.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-08T21:22:14.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.666+0000 7fc8cd193700 1 -- 172.21.15.3:0/3480438063 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc8b406ac90 msgr2=0x7fc8b406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:14.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.666+0000 7fc8cd193700 1 --2- 172.21.15.3:0/3480438063 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc8b406ac90 0x7fc8b406d150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc8b80011e0 tx=0x7fc8b8009040 comp rx=0 tx=0).stop 2024-05-08T21:22:14.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.666+0000 7fc8cd193700 1 -- 172.21.15.3:0/3480438063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8c815da10 msgr2=0x7fc8c80cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:14.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.666+0000 7fc8cd193700 1 --2- 172.21.15.3:0/3480438063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8c815da10 0x7fc8c80cd140 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000cba0 tx=0x7fc8c000ceb0 comp rx=0 tx=0).stop 2024-05-08T21:22:14.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.666+0000 7fc8cd193700 1 -- 172.21.15.3:0/3480438063 shutdown_connections 2024-05-08T21:22:14.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.666+0000 7fc8cd193700 1 --2- 172.21.15.3:0/3480438063 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc8b406ac90 0x7fc8b406d150 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:14.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.666+0000 7fc8cd193700 1 --2- 172.21.15.3:0/3480438063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8c815da10 0x7fc8c80cd140 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:14.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.666+0000 7fc8cd193700 1 --2- 172.21.15.3:0/3480438063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8c806bd80 0x7fc8c8152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:14.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.666+0000 7fc8cd193700 1 --2- 172.21.15.3:0/3480438063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8c806c3d0 0x7fc8c80ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:14.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.666+0000 7fc8cd193700 1 -- 172.21.15.3:0/3480438063 >> 172.21.15.3:0/3480438063 conn(0x7fc8c80b3010 msgr2=0x7fc8c806c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:14.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.666+0000 7fc8cd193700 1 -- 172.21.15.3:0/3480438063 shutdown_connections 2024-05-08T21:22:14.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:14.666+0000 7fc8cd193700 1 -- 172.21.15.3:0/3480438063 wait complete. 2024-05-08T21:22:14.681 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-08T21:22:14.682 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-08T21:22:14.682 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi186.front.sepia.ceph.com 2024-05-08T21:22:14.682 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-05-08T21:22:14.682 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-05-08T21:22:14.682 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-08T21:22:14.682 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-08T21:22:14.682 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-08T21:22:14.682 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-05-08T21:22:14.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:14.682 DEBUG:teuthology.orchestra.run.smithi186:> ip addr 2024-05-08T21:22:14.702 INFO:teuthology.orchestra.run.smithi186.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-08T21:22:14.702 INFO:teuthology.orchestra.run.smithi186.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-08T21:22:14.702 INFO:teuthology.orchestra.run.smithi186.stdout: inet 127.0.0.1/8 scope host lo 2024-05-08T21:22:14.702 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:14.702 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 ::1/128 scope host 2024-05-08T21:22:14.702 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:14.702 INFO:teuthology.orchestra.run.smithi186.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:22:14.702 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:d9:38:2c brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:14.703 INFO:teuthology.orchestra.run.smithi186.stdout: altname enp7s0f0 2024-05-08T21:22:14.703 INFO:teuthology.orchestra.run.smithi186.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:22:14.703 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:d9:38:2d brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:14.703 INFO:teuthology.orchestra.run.smithi186.stdout: altname enp7s0f1 2024-05-08T21:22:14.703 INFO:teuthology.orchestra.run.smithi186.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:22:14.703 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:88:6d:f4 brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:14.703 INFO:teuthology.orchestra.run.smithi186.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-08T21:22:14.703 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:88:6d:f5 brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:14.703 INFO:teuthology.orchestra.run.smithi186.stdout: inet 172.21.15.186/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-08T21:22:14.703 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft 42230sec preferred_lft 42230sec 2024-05-08T21:22:14.703 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 fe80::d996:13e2:ca9:325/64 scope link noprefixroute 2024-05-08T21:22:14.703 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:14.703 INFO:teuthology.orchestra.run.smithi186.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-08T21:22:14.703 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether ce:50:ad:bc:b9:c5 brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:14.704 INFO:teuthology.orchestra.run.smithi186.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-08T21:22:14.704 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:14.704 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 fe80::d475:aff:fe90:f4ff/64 scope link 2024-05-08T21:22:14.704 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:14.704 INFO:teuthology.orchestra.run.smithi186.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-08T21:22:14.704 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether ce:50:ad:bc:b9:c5 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-08T21:22:14.704 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 fe80::cc50:adff:febc:b9c5/64 scope link tentative 2024-05-08T21:22:14.704 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:14.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:14.704 DEBUG:teuthology.orchestra.run.smithi186:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-08T21:22:14.769 INFO:teuthology.orchestra.run.smithi186.stdout:1 2024-05-08T21:22:14.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:14.771 DEBUG:teuthology.orchestra.run.smithi186:> route 2024-05-08T21:22:14.831 INFO:teuthology.orchestra.run.smithi186.stdout:Kernel IP routing table 2024-05-08T21:22:14.831 INFO:teuthology.orchestra.run.smithi186.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-08T21:22:14.831 INFO:teuthology.orchestra.run.smithi186.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-08T21:22:14.831 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-08T21:22:14.831 INFO:teuthology.orchestra.run.smithi186.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-08T21:22:14.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:14.831 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2024-05-08T21:22:14.831 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-05-08T21:22:14.832 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-08T21:22:14.832 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-08T21:22:14.832 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-08T21:22:14.832 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-05-08T21:22:14.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:14.937 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2024-05-08T21:22:14.954 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-05-08T21:22:14.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:14.954 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list-id 2024-05-08T21:22:15.011 INFO:teuthology.orchestra.run.smithi186.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-05-08T21:22:15.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:15.011 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2024-05-08T21:22:15.011 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-05-08T21:22:15.011 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-08T21:22:15.011 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-05-08T21:22:15.012 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-05-08T21:22:15.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:15.106 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-05-08T21:22:15.150 INFO:teuthology.orchestra.run.smithi186.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-08T21:22:15.150 INFO:teuthology.orchestra.run.smithi186.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-08T21:22:15.150 INFO:teuthology.orchestra.run.smithi186.stdout: inet 127.0.0.1/8 scope host lo 2024-05-08T21:22:15.151 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:15.151 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 ::1/128 scope host 2024-05-08T21:22:15.151 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:15.151 INFO:teuthology.orchestra.run.smithi186.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-08T21:22:15.151 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 2a:c3:92:e3:c3:91 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-08T21:22:15.151 INFO:teuthology.orchestra.run.smithi186.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-08T21:22:15.151 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:15.151 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 fe80::28c3:92ff:fee3:c391/64 scope link tentative 2024-05-08T21:22:15.151 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:15.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:15.162 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-05-08T21:22:15.198 INFO:teuthology.orchestra.run.smithi186.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-08T21:22:15.198 INFO:teuthology.orchestra.run.smithi186.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-08T21:22:15.198 INFO:teuthology.orchestra.run.smithi186.stdout: inet 127.0.0.1/8 scope host lo 2024-05-08T21:22:15.199 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:15.199 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 ::1/128 scope host 2024-05-08T21:22:15.199 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:15.199 INFO:teuthology.orchestra.run.smithi186.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-08T21:22:15.199 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 2a:c3:92:e3:c3:91 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-08T21:22:15.199 INFO:teuthology.orchestra.run.smithi186.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-08T21:22:15.199 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:15.199 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 fe80::28c3:92ff:fee3:c391/64 scope link 2024-05-08T21:22:15.199 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:15.210 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-05-08T21:22:15.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:15.210 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2024-05-08T21:22:15.210 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-05-08T21:22:15.210 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-05-08T21:22:15.210 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-05-08T21:22:15.210 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-05-08T21:22:15.210 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-05-08T21:22:15.210 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-05-08T21:22:15.210 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-05-08T21:22:15.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:15.459 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2024-05-08T21:22:15.459 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-05-08T21:22:15.459 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link set brx.1 up 2024-05-08T21:22:15.459 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link set dev brx.1 master ceph-brx 2024-05-08T21:22:15.459 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-05-08T21:22:15.557 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-08T21:22:15.557 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-05-08T21:22:15.574 INFO:teuthology.orchestra.run.smithi186.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-05-08T21:22:15.574 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-08T21:22:15.574 DEBUG:teuthology.orchestra.run.smithi186:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-05-08T21:22:15.631 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-05-08T21:22:15.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:15.631 DEBUG:teuthology.orchestra.run.smithi186:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-05-08T21:22:15.717 INFO:teuthology.orchestra.run.smithi186.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-05-08T21:22:15.717 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-05-08T21:22:15.717 INFO:teuthology.orchestra.run.smithi186.stdout:invalid new device string format 2024-05-08T21:22:15.726 INFO:teuthology.orchestra.run.smithi186.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-05-08T21:22:15.726 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-05-08T21:22:15.726 INFO:teuthology.orchestra.run.smithi186.stdout:invalid new device string format 2024-05-08T21:22:15.726 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: resolved to: "172.21.15.3:3300,172.21.15.186:3300,172.21.15.186:3301" 2024-05-08T21:22:15.727 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-08T21:22:15.745 INFO:teuthology.orchestra.run.smithi186.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-05-08T21:22:15.745 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-05-08T21:22:15.745 INFO:teuthology.orchestra.run.smithi186.stdout:invalid new device string format 2024-05-08T21:22:15.745 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: resolved to: "172.21.15.3:3300,172.21.15.186:3300,172.21.15.186:3301" 2024-05-08T21:22:15.745 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-08T21:22:15.745 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: resolved to: "172.21.15.3:6789,172.21.15.186:6789,172.21.15.186:6790" 2024-05-08T21:22:15.745 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.3:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2024-05-08T21:22:15.745 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=d0f58275-758f-4470-b003-84d6abe25489" will pass to kernel 2024-05-08T21:22:15.747 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-08T21:22:15.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:15.748 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-05-08T21:22:15.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:15.774 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-08T21:22:15.846 INFO:teuthology.orchestra.run.smithi186.stdout:client4621 2024-05-08T21:22:15.847 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:22:15.847 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:15.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:15.904 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-08T21:22:15.976 INFO:teuthology.orchestra.run.smithi186.stdout:client4621 2024-05-08T21:22:15.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:15.976 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-08T21:22:16.048 INFO:teuthology.orchestra.run.smithi186.stdout:client4621 2024-05-08T21:22:16.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:16.049 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/d0f58275-758f-4470-b003-84d6abe25489.client4621/status') 2024-05-08T21:22:16.130 INFO:teuthology.orchestra.run.smithi186.stdout:instance: client.4621 (3)192.168.0.2:0/314705659 2024-05-08T21:22:16.130 INFO:teuthology.orchestra.run.smithi186.stdout:blocklisted: false 2024-05-08T21:22:16.130 INFO:teuthology.orchestra.run.smithi186.stderr:0+1 records in 2024-05-08T21:22:16.130 INFO:teuthology.orchestra.run.smithi186.stderr:0+1 records out 2024-05-08T21:22:16.130 INFO:teuthology.orchestra.run.smithi186.stderr:68 bytes copied, 1.9614e-05 s, 3.5 MB/s 2024-05-08T21:22:16.131 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-05-08T21:22:16.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:16.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.294+0000 7fdc3d681700 1 Processor -- start 2024-05-08T21:22:16.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.294+0000 7fdc3d681700 1 -- start start 2024-05-08T21:22:16.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.294+0000 7fdc3d681700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdc3815db10 0x7fdc380d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:16.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.294+0000 7fdc3d681700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc380b2700 0x7fdc3806c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:16.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.294+0000 7fdc3d681700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdc3806cdd0 0x7fdc380cbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:16.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc3d681700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc3806c400 con 0x7fdc3806cdd0 2024-05-08T21:22:16.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc3d681700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc380d0cd0 con 0x7fdc3815db10 2024-05-08T21:22:16.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc3d681700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc380d09f0 con 0x7fdc380b2700 2024-05-08T21:22:16.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc377fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdc3806cdd0 0x7fdc380cbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:16.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc377fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdc3806cdd0 0x7fdc380cbb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39912/0 (socket says 172.21.15.3:39912) 2024-05-08T21:22:16.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc377fe700 1 -- 172.21.15.3:0/1979013534 learned_addr learned my addr 172.21.15.3:0/1979013534 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:16.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc36ffd700 1 --2- 172.21.15.3:0/1979013534 >> v2:172.21.15.186:3300/0 conn(0x7fdc3815db10 0x7fdc380d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:16.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc367fc700 1 --2- 172.21.15.3:0/1979013534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc380b2700 0x7fdc3806c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:16.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc377fe700 1 -- 172.21.15.3:0/1979013534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc380b2700 msgr2=0x7fdc3806c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:16.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc377fe700 1 --2- 172.21.15.3:0/1979013534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc380b2700 0x7fdc3806c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc377fe700 1 -- 172.21.15.3:0/1979013534 >> v2:172.21.15.186:3300/0 conn(0x7fdc3815db10 msgr2=0x7fdc380d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:16.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc377fe700 1 --2- 172.21.15.3:0/1979013534 >> v2:172.21.15.186:3300/0 conn(0x7fdc3815db10 0x7fdc380d0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc377fe700 1 -- 172.21.15.3:0/1979013534 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3806bdb0 con 0x7fdc3806cdd0 2024-05-08T21:22:16.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc377fe700 1 --2- 172.21.15.3:0/1979013534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3806cdd0 0x7fdc380cbb00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdc30009e50 tx=0x7fdc3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edfa38a9d573191 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:16.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc35ffb700 1 -- 172.21.15.3:0/1979013534 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3000e070 con 0x7fdc3806cdd0 2024-05-08T21:22:16.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc35ffb700 1 -- 172.21.15.3:0/1979013534 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3000b8d0 con 0x7fdc3806cdd0 2024-05-08T21:22:16.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.295+0000 7fdc35ffb700 1 -- 172.21.15.3:0/1979013534 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3000bc80 con 0x7fdc3806cdd0 2024-05-08T21:22:16.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.296+0000 7fdc3d681700 1 -- 172.21.15.3:0/1979013534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3806cdd0 msgr2=0x7fdc380cbb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:16.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.296+0000 7fdc3d681700 1 --2- 172.21.15.3:0/1979013534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3806cdd0 0x7fdc380cbb00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdc30009e50 tx=0x7fdc3000b330 comp rx=0 tx=0).stop 2024-05-08T21:22:16.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.296+0000 7fdc3d681700 1 -- 172.21.15.3:0/1979013534 shutdown_connections 2024-05-08T21:22:16.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.296+0000 7fdc3d681700 1 --2- 172.21.15.3:0/1979013534 >> v2:172.21.15.186:3300/0 conn(0x7fdc3815db10 0x7fdc380d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.296+0000 7fdc3d681700 1 --2- 172.21.15.3:0/1979013534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3806cdd0 0x7fdc380cbb00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.296+0000 7fdc3d681700 1 --2- 172.21.15.3:0/1979013534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc380b2700 0x7fdc3806c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.296+0000 7fdc3d681700 1 -- 172.21.15.3:0/1979013534 >> 172.21.15.3:0/1979013534 conn(0x7fdc380b2fa0 msgr2=0x7fdc380b33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:16.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.296+0000 7fdc3d681700 1 -- 172.21.15.3:0/1979013534 shutdown_connections 2024-05-08T21:22:16.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.296+0000 7fdc3d681700 1 -- 172.21.15.3:0/1979013534 wait complete. 2024-05-08T21:22:16.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.298+0000 7fdc3d681700 1 Processor -- start 2024-05-08T21:22:16.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.298+0000 7fdc3d681700 1 -- start start 2024-05-08T21:22:16.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.298+0000 7fdc3d681700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3806cdd0 0x7fdc3815ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:16.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.298+0000 7fdc3d681700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc380b2700 0x7fdc381f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:16.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.298+0000 7fdc3d681700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc3815db10 0x7fdc380b5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:16.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.298+0000 7fdc3d681700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3815f1a0 con 0x7fdc3806cdd0 2024-05-08T21:22:16.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.298+0000 7fdc3d681700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3815f4e0 con 0x7fdc3815db10 2024-05-08T21:22:16.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.298+0000 7fdc3d681700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc380cb410 con 0x7fdc380b2700 2024-05-08T21:22:16.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.298+0000 7fdc36ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3806cdd0 0x7fdc3815ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:16.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.298+0000 7fdc36ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3806cdd0 0x7fdc3815ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39924/0 (socket says 172.21.15.3:39924) 2024-05-08T21:22:16.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.299+0000 7fdc36ffd700 1 -- 172.21.15.3:0/2110550474 learned_addr learned my addr 172.21.15.3:0/2110550474 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:16.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.299+0000 7fdc367fc700 1 --2- 172.21.15.3:0/2110550474 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc380b2700 0x7fdc381f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:16.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.299+0000 7fdc377fe700 1 --2- 172.21.15.3:0/2110550474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc3815db10 0x7fdc380b5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:16.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.299+0000 7fdc36ffd700 1 -- 172.21.15.3:0/2110550474 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc380b2700 msgr2=0x7fdc381f2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:16.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.299+0000 7fdc36ffd700 1 --2- 172.21.15.3:0/2110550474 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc380b2700 0x7fdc381f2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.299+0000 7fdc36ffd700 1 -- 172.21.15.3:0/2110550474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc3815db10 msgr2=0x7fdc380b5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:16.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.299+0000 7fdc36ffd700 1 --2- 172.21.15.3:0/2110550474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc3815db10 0x7fdc380b5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.299+0000 7fdc36ffd700 1 -- 172.21.15.3:0/2110550474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc30009b30 con 0x7fdc3806cdd0 2024-05-08T21:22:16.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.299+0000 7fdc36ffd700 1 --2- 172.21.15.3:0/2110550474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3806cdd0 0x7fdc3815ec60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800e870 tx=0x7fdc2800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:16.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.299+0000 7fdc1f7fe700 1 -- 172.21.15.3:0/2110550474 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc28009590 con 0x7fdc3806cdd0 2024-05-08T21:22:16.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.299+0000 7fdc1f7fe700 1 -- 172.21.15.3:0/2110550474 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc28013680 con 0x7fdc3806cdd0 2024-05-08T21:22:16.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.299+0000 7fdc3d681700 1 -- 172.21.15.3:0/2110550474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc381f6fc0 con 0x7fdc3806cdd0 2024-05-08T21:22:16.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.299+0000 7fdc1f7fe700 1 -- 172.21.15.3:0/2110550474 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc280139f0 con 0x7fdc3806cdd0 2024-05-08T21:22:16.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.299+0000 7fdc3d681700 1 -- 172.21.15.3:0/2110550474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc381f7580 con 0x7fdc3806cdd0 2024-05-08T21:22:16.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.300+0000 7fdc3d681700 1 -- 172.21.15.3:0/2110550474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3815f7c0 con 0x7fdc3806cdd0 2024-05-08T21:22:16.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.301+0000 7fdc1f7fe700 1 -- 172.21.15.3:0/2110550474 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdc28013b50 con 0x7fdc3806cdd0 2024-05-08T21:22:16.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.301+0000 7fdc1f7fe700 1 --2- 172.21.15.3:0/2110550474 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdc2406ac90 0x7fdc2406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:16.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.301+0000 7fdc1f7fe700 1 -- 172.21.15.3:0/2110550474 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc28089110 con 0x7fdc3806cdd0 2024-05-08T21:22:16.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.302+0000 7fdc367fc700 1 --2- 172.21.15.3:0/2110550474 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdc2406ac90 0x7fdc2406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:16.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.302+0000 7fdc367fc700 1 --2- 172.21.15.3:0/2110550474 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdc2406ac90 0x7fdc2406d150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009b60 tx=0x7fdc2c009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:16.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.306+0000 7fdc1f7fe700 1 -- 172.21.15.3:0/2110550474 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fdc28054380 con 0x7fdc3806cdd0 2024-05-08T21:22:16.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.522+0000 7fdc3d681700 1 -- 172.21.15.3:0/2110550474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc380678f0 con 0x7fdc3806cdd0 2024-05-08T21:22:16.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.523+0000 7fdc1f7fe700 1 -- 172.21.15.3:0/2110550474 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15989 (secure 0 0 0) 0x7fdc280579a0 con 0x7fdc3806cdd0 2024-05-08T21:22:16.525 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:16.525 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:53.064342+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:16.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.526+0000 7fdc3d681700 1 -- 172.21.15.3:0/2110550474 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdc2406ac90 msgr2=0x7fdc2406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:16.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.526+0000 7fdc3d681700 1 --2- 172.21.15.3:0/2110550474 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdc2406ac90 0x7fdc2406d150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009b60 tx=0x7fdc2c009210 comp rx=0 tx=0).stop 2024-05-08T21:22:16.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.526+0000 7fdc3d681700 1 -- 172.21.15.3:0/2110550474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3806cdd0 msgr2=0x7fdc3815ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:16.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.526+0000 7fdc3d681700 1 --2- 172.21.15.3:0/2110550474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3806cdd0 0x7fdc3815ec60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800e870 tx=0x7fdc2800eb80 comp rx=0 tx=0).stop 2024-05-08T21:22:16.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.526+0000 7fdc3d681700 1 -- 172.21.15.3:0/2110550474 shutdown_connections 2024-05-08T21:22:16.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.526+0000 7fdc3d681700 1 --2- 172.21.15.3:0/2110550474 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdc2406ac90 0x7fdc2406d150 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.526+0000 7fdc3d681700 1 --2- 172.21.15.3:0/2110550474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc3815db10 0x7fdc380b5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.526+0000 7fdc3d681700 1 --2- 172.21.15.3:0/2110550474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3806cdd0 0x7fdc3815ec60 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.526+0000 7fdc3d681700 1 --2- 172.21.15.3:0/2110550474 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc380b2700 0x7fdc381f2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.526+0000 7fdc3d681700 1 -- 172.21.15.3:0/2110550474 >> 172.21.15.3:0/2110550474 conn(0x7fdc380b2fa0 msgr2=0x7fdc380cdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:16.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.526+0000 7fdc3d681700 1 -- 172.21.15.3:0/2110550474 shutdown_connections 2024-05-08T21:22:16.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.526+0000 7fdc3d681700 1 -- 172.21.15.3:0/2110550474 wait complete. 2024-05-08T21:22:16.541 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-08T21:21:49.638998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:21:51.024632+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:22:16.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:22:16.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.680+0000 7f9d7e618700 1 Processor -- start 2024-05-08T21:22:16.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.681+0000 7f9d7e618700 1 -- start start 2024-05-08T21:22:16.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.681+0000 7f9d7e618700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d78161ff0 0x7f9d7806b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:16.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.681+0000 7f9d7e618700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d7815dce0 con 0x7f9d7806beb0 2024-05-08T21:22:16.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.681+0000 7f9d7e618700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d780ce5b0 con 0x7f9d7815d900 2024-05-08T21:22:16.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.681+0000 7f9d7e618700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7806c320 con 0x7f9d78161ff0 2024-05-08T21:22:16.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.682+0000 7f9d7cbb5700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9d7806beb0 0x7f9d780ada00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55242/0 (socket says 172.21.15.3:55242) 2024-05-08T21:22:16.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.682+0000 7f9d7cbb5700 1 -- 172.21.15.3:0/3578208294 learned_addr learned my addr 172.21.15.3:0/3578208294 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:16.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.682+0000 7f9d777fe700 1 --2- 172.21.15.3:0/3578208294 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d78161ff0 0x7f9d7806b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:16.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.682+0000 7f9d76ffd700 1 -- 172.21.15.3:0/3578208294 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499793770 0 0) 0x7f9d7815dce0 con 0x7f9d7806beb0 2024-05-08T21:22:16.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.682+0000 7f9d76ffd700 1 -- 172.21.15.3:0/3578208294 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d600035b0 con 0x7f9d7806beb0 2024-05-08T21:22:16.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.682+0000 7f9d76ffd700 1 -- 172.21.15.3:0/3578208294 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911392562 0 0) 0x7f9d780ce5b0 con 0x7f9d7815d900 2024-05-08T21:22:16.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.682+0000 7f9d76ffd700 1 -- 172.21.15.3:0/3578208294 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d7815dce0 con 0x7f9d7815d900 2024-05-08T21:22:16.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.682+0000 7f9d777fe700 1 -- 172.21.15.3:0/3578208294 >> v1:172.21.15.186:6789/0 conn(0x7f9d7815d900 legacy=0x7f9d780cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:16.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.682+0000 7f9d777fe700 1 -- 172.21.15.3:0/3578208294 >> v1:172.21.15.3:6789/0 conn(0x7f9d7806beb0 legacy=0x7f9d780ada00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:16.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.682+0000 7f9d777fe700 1 -- 172.21.15.3:0/3578208294 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d78161510 con 0x7f9d78161ff0 2024-05-08T21:22:16.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.682+0000 7f9d777fe700 1 --2- 172.21.15.3:0/3578208294 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d78161ff0 0x7f9d7806b970 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c009870 tx=0x7f9d6c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e112f8b144c22f2b server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:16.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.682+0000 7f9d76ffd700 1 -- 172.21.15.3:0/3578208294 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c011830 con 0x7f9d78161ff0 2024-05-08T21:22:16.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.682+0000 7f9d76ffd700 1 -- 172.21.15.3:0/3578208294 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6c011990 con 0x7f9d78161ff0 2024-05-08T21:22:16.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.682+0000 7f9d76ffd700 1 -- 172.21.15.3:0/3578208294 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c011d20 con 0x7f9d78161ff0 2024-05-08T21:22:16.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.683+0000 7f9d7e618700 1 -- 172.21.15.3:0/3578208294 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d78161ff0 msgr2=0x7f9d7806b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:16.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.683+0000 7f9d7e618700 1 --2- 172.21.15.3:0/3578208294 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d78161ff0 0x7f9d7806b970 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c009870 tx=0x7f9d6c009b80 comp rx=0 tx=0).stop 2024-05-08T21:22:16.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.683+0000 7f9d7e618700 1 -- 172.21.15.3:0/3578208294 shutdown_connections 2024-05-08T21:22:16.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.683+0000 7f9d7e618700 1 --2- 172.21.15.3:0/3578208294 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d78161ff0 0x7f9d7806b970 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.683+0000 7f9d7e618700 1 -- 172.21.15.3:0/3578208294 >> 172.21.15.3:0/3578208294 conn(0x7f9d780713c0 msgr2=0x7f9d780717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:16.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.683+0000 7f9d7e618700 1 -- 172.21.15.3:0/3578208294 shutdown_connections 2024-05-08T21:22:16.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.683+0000 7f9d7e618700 1 -- 172.21.15.3:0/3578208294 wait complete. 2024-05-08T21:22:16.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.684+0000 7f9d7e618700 1 Processor -- start 2024-05-08T21:22:16.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.685+0000 7f9d7e618700 1 -- start start 2024-05-08T21:22:16.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.685+0000 7f9d7e618700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d7806beb0 0x7f9d78142c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.685+0000 7f9d7e618700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d7815d900 0x7f9d780afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.685+0000 7f9d7e618700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d78161ff0 0x7f9d780b0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.685+0000 7f9d7e618700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d781554f0 con 0x7f9d7815d900 2024-05-08T21:22:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.685+0000 7f9d7e618700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d78155060 con 0x7f9d7806beb0 2024-05-08T21:22:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.685+0000 7f9d7e618700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d78154490 con 0x7f9d78161ff0 2024-05-08T21:22:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.686+0000 7f9d777fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d7815d900 0x7f9d780afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.686+0000 7f9d777fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d7815d900 0x7f9d780afbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39932/0 (socket says 172.21.15.3:39932) 2024-05-08T21:22:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.686+0000 7f9d777fe700 1 -- 172.21.15.3:0/1318891108 learned_addr learned my addr 172.21.15.3:0/1318891108 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.686+0000 7f9d7cbb5700 1 --2- 172.21.15.3:0/1318891108 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d78161ff0 0x7f9d780b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.686+0000 7f9d77fff700 1 --2- 172.21.15.3:0/1318891108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d7806beb0 0x7f9d78142c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.686+0000 7f9d777fe700 1 -- 172.21.15.3:0/1318891108 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d78161ff0 msgr2=0x7f9d780b0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.686+0000 7f9d777fe700 1 --2- 172.21.15.3:0/1318891108 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d78161ff0 0x7f9d780b0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.686+0000 7f9d777fe700 1 -- 172.21.15.3:0/1318891108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d7806beb0 msgr2=0x7f9d78142c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:16.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.686+0000 7f9d777fe700 1 --2- 172.21.15.3:0/1318891108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d7806beb0 0x7f9d78142c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.686+0000 7f9d777fe700 1 -- 172.21.15.3:0/1318891108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6c0094a0 con 0x7f9d7815d900 2024-05-08T21:22:16.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.686+0000 7f9d777fe700 1 --2- 172.21.15.3:0/1318891108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d7815d900 0x7f9d780afbd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c009bc0 tx=0x7f9d6c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:16.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.686+0000 7f9d74ff9700 1 -- 172.21.15.3:0/1318891108 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c01a6e0 con 0x7f9d7815d900 2024-05-08T21:22:16.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.686+0000 7f9d74ff9700 1 -- 172.21.15.3:0/1318891108 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6c01a860 con 0x7f9d7815d900 2024-05-08T21:22:16.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.686+0000 7f9d7e618700 1 -- 172.21.15.3:0/1318891108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d78154730 con 0x7f9d7815d900 2024-05-08T21:22:16.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.686+0000 7f9d74ff9700 1 -- 172.21.15.3:0/1318891108 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c01ac10 con 0x7f9d7815d900 2024-05-08T21:22:16.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.686+0000 7f9d7e618700 1 -- 172.21.15.3:0/1318891108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d78154cf0 con 0x7f9d7815d900 2024-05-08T21:22:16.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.687+0000 7f9d7e618700 1 -- 172.21.15.3:0/1318891108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d78074bc0 con 0x7f9d7815d900 2024-05-08T21:22:16.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.688+0000 7f9d74ff9700 1 -- 172.21.15.3:0/1318891108 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9d6c022070 con 0x7f9d7815d900 2024-05-08T21:22:16.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.688+0000 7f9d74ff9700 1 --2- 172.21.15.3:0/1318891108 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9d6406ac90 0x7f9d6406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:16.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.688+0000 7f9d74ff9700 1 -- 172.21.15.3:0/1318891108 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d6c08a0e0 con 0x7f9d7815d900 2024-05-08T21:22:16.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.689+0000 7f9d77fff700 1 --2- 172.21.15.3:0/1318891108 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9d6406ac90 0x7f9d6406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:16.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.689+0000 7f9d77fff700 1 --2- 172.21.15.3:0/1318891108 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9d6406ac90 0x7f9d6406d150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9d680011e0 tx=0x7f9d68009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:16.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.693+0000 7f9d74ff9700 1 -- 172.21.15.3:0/1318891108 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9d6c055350 con 0x7f9d7815d900 2024-05-08T21:22:16.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.911+0000 7f9d7e618700 1 -- 172.21.15.3:0/1318891108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9d7815de70 con 0x7f9d7815d900 2024-05-08T21:22:16.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.911+0000 7f9d74ff9700 1 -- 172.21.15.3:0/1318891108 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9d6c058970 con 0x7f9d7815d900 2024-05-08T21:22:16.913 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:22:16.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.913+0000 7f9d7e618700 1 -- 172.21.15.3:0/1318891108 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9d6406ac90 msgr2=0x7f9d6406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:16.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.913+0000 7f9d7e618700 1 --2- 172.21.15.3:0/1318891108 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9d6406ac90 0x7f9d6406d150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9d680011e0 tx=0x7f9d68009040 comp rx=0 tx=0).stop 2024-05-08T21:22:16.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.913+0000 7f9d7e618700 1 -- 172.21.15.3:0/1318891108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d7815d900 msgr2=0x7f9d780afbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:16.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.913+0000 7f9d7e618700 1 --2- 172.21.15.3:0/1318891108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d7815d900 0x7f9d780afbd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c009bc0 tx=0x7f9d6c004750 comp rx=0 tx=0).stop 2024-05-08T21:22:16.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.913+0000 7f9d7e618700 1 -- 172.21.15.3:0/1318891108 shutdown_connections 2024-05-08T21:22:16.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.913+0000 7f9d7e618700 1 --2- 172.21.15.3:0/1318891108 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d78161ff0 0x7f9d780b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.913+0000 7f9d7e618700 1 --2- 172.21.15.3:0/1318891108 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9d6406ac90 0x7f9d6406d150 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.913+0000 7f9d7e618700 1 --2- 172.21.15.3:0/1318891108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d7806beb0 0x7f9d78142c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.914+0000 7f9d7e618700 1 --2- 172.21.15.3:0/1318891108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d7815d900 0x7f9d780afbd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:16.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.914+0000 7f9d7e618700 1 -- 172.21.15.3:0/1318891108 >> 172.21.15.3:0/1318891108 conn(0x7f9d780713c0 msgr2=0x7f9d7806dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:16.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.914+0000 7f9d7e618700 1 -- 172.21.15.3:0/1318891108 shutdown_connections 2024-05-08T21:22:16.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:16.914+0000 7f9d7e618700 1 -- 172.21.15.3:0/1318891108 wait complete. 2024-05-08T21:22:16.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-08T21:22:17.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.067+0000 7f2f72a19700 1 Processor -- start 2024-05-08T21:22:17.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.067+0000 7f2f72a19700 1 -- start start 2024-05-08T21:22:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.068+0000 7f2f72a19700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2f6c06a960 0x7f2f6c099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.068+0000 7f2f72a19700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f6c0aea70 0x7f2f6c0aef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.068+0000 7f2f72a19700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6c15c920 con 0x7f2f6c0b5500 2024-05-08T21:22:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.068+0000 7f2f72a19700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6c0ad6b0 con 0x7f2f6c06a960 2024-05-08T21:22:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.068+0000 7f2f72a19700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0b2700 con 0x7f2f6c0aea70 2024-05-08T21:22:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.068+0000 7f2f70fb6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f6c0aea70 0x7f2f6c0aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.068+0000 7f2f6bfff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2f6c06a960 0x7f2f6c099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.068+0000 7f2f70fb6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f6c0aea70 0x7f2f6c0aef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.3:37876/0 (socket says 172.21.15.3:37876) 2024-05-08T21:22:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.068+0000 7f2f70fb6700 1 -- 172.21.15.3:0/2038884431 learned_addr learned my addr 172.21.15.3:0/2038884431 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.068+0000 7f2f6b7fe700 1 --1- 172.21.15.3:0/2038884431 >> v1:172.21.15.186:6789/0 conn(0x7f2f6c0b5500 0x7f2f6c06ad40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:50504/0 (socket says 172.21.15.3:50504) 2024-05-08T21:22:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.068+0000 7f2f70fb6700 1 -- 172.21.15.3:0/2038884431 >> v2:172.21.15.186:3300/0 conn(0x7f2f6c06a960 msgr2=0x7f2f6c099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.068+0000 7f2f70fb6700 1 --2- 172.21.15.3:0/2038884431 >> v2:172.21.15.186:3300/0 conn(0x7f2f6c06a960 0x7f2f6c099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.068+0000 7f2f70fb6700 1 -- 172.21.15.3:0/2038884431 >> v1:172.21.15.186:6789/0 conn(0x7f2f6c0b5500 legacy=0x7f2f6c06ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.068+0000 7f2f70fb6700 1 -- 172.21.15.3:0/2038884431 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c0b2910 con 0x7f2f6c0aea70 2024-05-08T21:22:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.068+0000 7f2f70fb6700 1 --2- 172.21.15.3:0/2038884431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f6c0aea70 0x7f2f6c0aef00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2f64009e80 tx=0x7f2f6400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3debdb411760a840 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.069+0000 7f2f6affd700 1 -- 172.21.15.3:0/2038884431 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6400e070 con 0x7f2f6c0aea70 2024-05-08T21:22:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.069+0000 7f2f6affd700 1 -- 172.21.15.3:0/2038884431 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6400b8d0 con 0x7f2f6c0aea70 2024-05-08T21:22:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.069+0000 7f2f6affd700 1 -- 172.21.15.3:0/2038884431 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6400bc80 con 0x7f2f6c0aea70 2024-05-08T21:22:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.069+0000 7f2f72a19700 1 -- 172.21.15.3:0/2038884431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f6c0aea70 msgr2=0x7f2f6c0aef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.069+0000 7f2f72a19700 1 --2- 172.21.15.3:0/2038884431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f6c0aea70 0x7f2f6c0aef00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2f64009e80 tx=0x7f2f6400b330 comp rx=0 tx=0).stop 2024-05-08T21:22:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.069+0000 7f2f72a19700 1 -- 172.21.15.3:0/2038884431 shutdown_connections 2024-05-08T21:22:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.069+0000 7f2f72a19700 1 --2- 172.21.15.3:0/2038884431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f6c0aea70 0x7f2f6c0aef00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.069+0000 7f2f72a19700 1 --2- 172.21.15.3:0/2038884431 >> v2:172.21.15.186:3300/0 conn(0x7f2f6c06a960 0x7f2f6c099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.069+0000 7f2f72a19700 1 -- 172.21.15.3:0/2038884431 >> 172.21.15.3:0/2038884431 conn(0x7f2f6c0b5a30 msgr2=0x7f2f6c0b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.069+0000 7f2f72a19700 1 -- 172.21.15.3:0/2038884431 shutdown_connections 2024-05-08T21:22:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.069+0000 7f2f72a19700 1 -- 172.21.15.3:0/2038884431 wait complete. 2024-05-08T21:22:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.071+0000 7f2f72a19700 1 Processor -- start 2024-05-08T21:22:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.071+0000 7f2f72a19700 1 -- start start 2024-05-08T21:22:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f72a19700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f6c06a960 0x7f2f6c144f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f72a19700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f6c0aea70 0x7f2f6c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f72a19700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f6c0b5500 0x7f2f6c159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f72a19700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c1576d0 con 0x7f2f6c0aea70 2024-05-08T21:22:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f72a19700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c156ac0 con 0x7f2f6c0b5500 2024-05-08T21:22:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f72a19700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c156da0 con 0x7f2f6c06a960 2024-05-08T21:22:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f6b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f6c0aea70 0x7f2f6c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f6b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f6c0aea70 0x7f2f6c150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39940/0 (socket says 172.21.15.3:39940) 2024-05-08T21:22:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f6b7fe700 1 -- 172.21.15.3:0/2394347071 learned_addr learned my addr 172.21.15.3:0/2394347071 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f6bfff700 1 --2- 172.21.15.3:0/2394347071 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f6c06a960 0x7f2f6c144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f70fb6700 1 --2- 172.21.15.3:0/2394347071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f6c0b5500 0x7f2f6c159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f6b7fe700 1 -- 172.21.15.3:0/2394347071 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f6c06a960 msgr2=0x7f2f6c144f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f6b7fe700 1 --2- 172.21.15.3:0/2394347071 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f6c06a960 0x7f2f6c144f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f6b7fe700 1 -- 172.21.15.3:0/2394347071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f6c0b5500 msgr2=0x7f2f6c159750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f6b7fe700 1 --2- 172.21.15.3:0/2394347071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f6c0b5500 0x7f2f6c159750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f6b7fe700 1 -- 172.21.15.3:0/2394347071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f64009b30 con 0x7f2f6c0aea70 2024-05-08T21:22:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.072+0000 7f2f6b7fe700 1 --2- 172.21.15.3:0/2394347071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f6c0aea70 0x7f2f6c150ad0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000c520 tx=0x7f2f6000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.073+0000 7f2f68ff9700 1 -- 172.21.15.3:0/2394347071 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f60005cb0 con 0x7f2f6c0aea70 2024-05-08T21:22:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.073+0000 7f2f68ff9700 1 -- 172.21.15.3:0/2394347071 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f60005e10 con 0x7f2f6c0aea70 2024-05-08T21:22:17.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.073+0000 7f2f68ff9700 1 -- 172.21.15.3:0/2394347071 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f600072d0 con 0x7f2f6c0aea70 2024-05-08T21:22:17.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.073+0000 7f2f72a19700 1 -- 172.21.15.3:0/2394347071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c157080 con 0x7f2f6c0aea70 2024-05-08T21:22:17.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.073+0000 7f2f72a19700 1 -- 172.21.15.3:0/2394347071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6c157b60 con 0x7f2f6c0aea70 2024-05-08T21:22:17.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.074+0000 7f2f72a19700 1 -- 172.21.15.3:0/2394347071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c157d80 con 0x7f2f6c0aea70 2024-05-08T21:22:17.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.074+0000 7f2f68ff9700 1 -- 172.21.15.3:0/2394347071 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2f60007430 con 0x7f2f6c0aea70 2024-05-08T21:22:17.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.075+0000 7f2f68ff9700 1 --2- 172.21.15.3:0/2394347071 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2f5806ac90 0x7f2f5806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.075+0000 7f2f68ff9700 1 -- 172.21.15.3:0/2394347071 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f60088510 con 0x7f2f6c0aea70 2024-05-08T21:22:17.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.075+0000 7f2f6bfff700 1 --2- 172.21.15.3:0/2394347071 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2f5806ac90 0x7f2f5806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.075+0000 7f2f6bfff700 1 --2- 172.21.15.3:0/2394347071 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2f5806ac90 0x7f2f5806d150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0ae190 tx=0x7f2f5c009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:17.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.079+0000 7f2f68ff9700 1 -- 172.21.15.3:0/2394347071 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2f60053830 con 0x7f2f6c0aea70 2024-05-08T21:22:17.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.295+0000 7f2f72a19700 1 -- 172.21.15.3:0/2394347071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2f6c074bc0 con 0x7f2f6c0aea70 2024-05-08T21:22:17.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.296+0000 7f2f68ff9700 1 -- 172.21.15.3:0/2394347071 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2f60086e60 con 0x7f2f6c0aea70 2024-05-08T21:22:17.298 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:22:17.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.298+0000 7f2f72a19700 1 -- 172.21.15.3:0/2394347071 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2f5806ac90 msgr2=0x7f2f5806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:17.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.298+0000 7f2f72a19700 1 --2- 172.21.15.3:0/2394347071 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2f5806ac90 0x7f2f5806d150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0ae190 tx=0x7f2f5c009210 comp rx=0 tx=0).stop 2024-05-08T21:22:17.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.298+0000 7f2f72a19700 1 -- 172.21.15.3:0/2394347071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f6c0aea70 msgr2=0x7f2f6c150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:17.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.298+0000 7f2f72a19700 1 --2- 172.21.15.3:0/2394347071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f6c0aea70 0x7f2f6c150ad0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000c520 tx=0x7f2f6000c830 comp rx=0 tx=0).stop 2024-05-08T21:22:17.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.298+0000 7f2f72a19700 1 -- 172.21.15.3:0/2394347071 shutdown_connections 2024-05-08T21:22:17.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.298+0000 7f2f72a19700 1 --2- 172.21.15.3:0/2394347071 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f6c06a960 0x7f2f6c144f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.298+0000 7f2f72a19700 1 --2- 172.21.15.3:0/2394347071 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2f5806ac90 0x7f2f5806d150 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.298+0000 7f2f72a19700 1 --2- 172.21.15.3:0/2394347071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f6c0b5500 0x7f2f6c159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.298+0000 7f2f72a19700 1 --2- 172.21.15.3:0/2394347071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f6c0aea70 0x7f2f6c150ad0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.298+0000 7f2f72a19700 1 -- 172.21.15.3:0/2394347071 >> 172.21.15.3:0/2394347071 conn(0x7f2f6c0b5a30 msgr2=0x7f2f6c1557f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:17.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.298+0000 7f2f72a19700 1 -- 172.21.15.3:0/2394347071 shutdown_connections 2024-05-08T21:22:17.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.298+0000 7f2f72a19700 1 -- 172.21.15.3:0/2394347071 wait complete. 2024-05-08T21:22:17.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:22:17.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.452+0000 7f226b3f9700 1 Processor -- start 2024-05-08T21:22:17.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.452+0000 7f226b3f9700 1 -- start start 2024-05-08T21:22:17.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.453+0000 7f226b3f9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226415bb20 0x7f226406e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.453+0000 7f226b3f9700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f226406ec20 0x7f2264159f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.453+0000 7f226b3f9700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22640cc980 con 0x7f22640cfd10 2024-05-08T21:22:17.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.453+0000 7f226b3f9700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f226406f0d0 con 0x7f226406ec20 2024-05-08T21:22:17.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.453+0000 7f226b3f9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22640d00f0 con 0x7f226415bb20 2024-05-08T21:22:17.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.453+0000 7f2269996700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f226406ec20 0x7f2264159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.453+0000 7f2269195700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f22640cfd10 0x7f226406dee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55258/0 (socket says 172.21.15.3:55258) 2024-05-08T21:22:17.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.453+0000 7f2269195700 1 -- 172.21.15.3:0/3785882555 learned_addr learned my addr 172.21.15.3:0/3785882555 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:17.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.453+0000 7f2268994700 1 --2- 172.21.15.3:0/3785882555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226415bb20 0x7f226406e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.453+0000 7f2269996700 1 -- 172.21.15.3:0/3785882555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226415bb20 msgr2=0x7f226406e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:17.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.453+0000 7f2269996700 1 --2- 172.21.15.3:0/3785882555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226415bb20 0x7f226406e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.453+0000 7f2269996700 1 -- 172.21.15.3:0/3785882555 >> v1:172.21.15.3:6789/0 conn(0x7f22640cfd10 legacy=0x7f226406dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:17.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.453+0000 7f2269996700 1 -- 172.21.15.3:0/3785882555 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22640ccbd0 con 0x7f226406ec20 2024-05-08T21:22:17.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.453+0000 7f225bfff700 1 -- 172.21.15.3:0/3785882555 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581234803 0 0) 0x7f22640cc980 con 0x7f22640cfd10 2024-05-08T21:22:17.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.454+0000 7f2269996700 1 --2- 172.21.15.3:0/3785882555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f226406ec20 0x7f2264159f80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f225c009d90 tx=0x7f225c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a42777c125b3165c server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:17.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.454+0000 7f225bfff700 1 -- 172.21.15.3:0/3785882555 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c00e070 con 0x7f226406ec20 2024-05-08T21:22:17.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.454+0000 7f225bfff700 1 -- 172.21.15.3:0/3785882555 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225c00bbd0 con 0x7f226406ec20 2024-05-08T21:22:17.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.454+0000 7f225bfff700 1 -- 172.21.15.3:0/3785882555 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c012430 con 0x7f226406ec20 2024-05-08T21:22:17.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.454+0000 7f226b3f9700 1 -- 172.21.15.3:0/3785882555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f226406ec20 msgr2=0x7f2264159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:17.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.454+0000 7f226b3f9700 1 --2- 172.21.15.3:0/3785882555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f226406ec20 0x7f2264159f80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f225c009d90 tx=0x7f225c00b330 comp rx=0 tx=0).stop 2024-05-08T21:22:17.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.454+0000 7f226b3f9700 1 -- 172.21.15.3:0/3785882555 shutdown_connections 2024-05-08T21:22:17.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.454+0000 7f226b3f9700 1 --2- 172.21.15.3:0/3785882555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f226406ec20 0x7f2264159f80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.454+0000 7f226b3f9700 1 --2- 172.21.15.3:0/3785882555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226415bb20 0x7f226406e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.454+0000 7f226b3f9700 1 -- 172.21.15.3:0/3785882555 >> 172.21.15.3:0/3785882555 conn(0x7f2264099a50 msgr2=0x7f22640d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:17.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.454+0000 7f226b3f9700 1 -- 172.21.15.3:0/3785882555 shutdown_connections 2024-05-08T21:22:17.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.454+0000 7f226b3f9700 1 -- 172.21.15.3:0/3785882555 wait complete. 2024-05-08T21:22:17.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.456+0000 7f226b3f9700 1 Processor -- start 2024-05-08T21:22:17.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.456+0000 7f226b3f9700 1 -- start start 2024-05-08T21:22:17.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.457+0000 7f226b3f9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f226406ec20 0x7f22640c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.457+0000 7f226b3f9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22640cfd10 0x7f2264150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.457+0000 7f226b3f9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226415bb20 0x7f22640ae230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.457+0000 7f226b3f9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2264150ea0 con 0x7f226406ec20 2024-05-08T21:22:17.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.457+0000 7f226b3f9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226415c360 con 0x7f22640cfd10 2024-05-08T21:22:17.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.457+0000 7f226b3f9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22641f1db0 con 0x7f226415bb20 2024-05-08T21:22:17.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.457+0000 7f2269195700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f226406ec20 0x7f22640c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.457+0000 7f2269195700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f226406ec20 0x7f22640c6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39952/0 (socket says 172.21.15.3:39952) 2024-05-08T21:22:17.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.457+0000 7f2269195700 1 -- 172.21.15.3:0/1059012227 learned_addr learned my addr 172.21.15.3:0/1059012227 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:17.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.458+0000 7f2269996700 1 --2- 172.21.15.3:0/1059012227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226415bb20 0x7f22640ae230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.458+0000 7f2268994700 1 --2- 172.21.15.3:0/1059012227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22640cfd10 0x7f2264150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.458+0000 7f2269195700 1 -- 172.21.15.3:0/1059012227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226415bb20 msgr2=0x7f22640ae230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:17.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.458+0000 7f2269195700 1 --2- 172.21.15.3:0/1059012227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226415bb20 0x7f22640ae230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.458+0000 7f2269195700 1 -- 172.21.15.3:0/1059012227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22640cfd10 msgr2=0x7f2264150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:17.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.458+0000 7f2269195700 1 --2- 172.21.15.3:0/1059012227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22640cfd10 0x7f2264150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.458+0000 7f2269195700 1 -- 172.21.15.3:0/1059012227 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c009a70 con 0x7f226406ec20 2024-05-08T21:22:17.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.458+0000 7f2268994700 1 --2- 172.21.15.3:0/1059012227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22640cfd10 0x7f2264150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:17.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.458+0000 7f2269195700 1 --2- 172.21.15.3:0/1059012227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f226406ec20 0x7f22640c6830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f22600011e0 tx=0x7f226000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:17.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.458+0000 7f2259ffb700 1 -- 172.21.15.3:0/1059012227 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2260004c80 con 0x7f226406ec20 2024-05-08T21:22:17.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.458+0000 7f2259ffb700 1 -- 172.21.15.3:0/1059012227 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2260004e00 con 0x7f226406ec20 2024-05-08T21:22:17.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.458+0000 7f226b3f9700 1 -- 172.21.15.3:0/1059012227 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22641f2090 con 0x7f226406ec20 2024-05-08T21:22:17.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.458+0000 7f2259ffb700 1 -- 172.21.15.3:0/1059012227 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2260007310 con 0x7f226406ec20 2024-05-08T21:22:17.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.458+0000 7f226b3f9700 1 -- 172.21.15.3:0/1059012227 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226415bff0 con 0x7f226406ec20 2024-05-08T21:22:17.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.459+0000 7f226b3f9700 1 -- 172.21.15.3:0/1059012227 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2264074bc0 con 0x7f226406ec20 2024-05-08T21:22:17.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.460+0000 7f2259ffb700 1 -- 172.21.15.3:0/1059012227 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2260007470 con 0x7f226406ec20 2024-05-08T21:22:17.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.460+0000 7f2259ffb700 1 --2- 172.21.15.3:0/1059012227 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f224806ac90 0x7f224806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.460+0000 7f2259ffb700 1 -- 172.21.15.3:0/1059012227 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f22600890a0 con 0x7f226406ec20 2024-05-08T21:22:17.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.460+0000 7f2268994700 1 --2- 172.21.15.3:0/1059012227 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f224806ac90 0x7f224806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.461+0000 7f2268994700 1 --2- 172.21.15.3:0/1059012227 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f224806ac90 0x7f224806d150 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f226415bf20 tx=0x7f2254008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:17.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.465+0000 7f2259ffb700 1 -- 172.21.15.3:0/1059012227 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2260054280 con 0x7f226406ec20 2024-05-08T21:22:17.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.683+0000 7f226b3f9700 1 -- 172.21.15.3:0/1059012227 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f22640d0a40 con 0x7f226406ec20 2024-05-08T21:22:17.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.683+0000 7f2259ffb700 1 -- 172.21.15.3:0/1059012227 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f22600578a0 con 0x7f226406ec20 2024-05-08T21:22:17.685 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:22:17.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.686+0000 7f226b3f9700 1 -- 172.21.15.3:0/1059012227 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f224806ac90 msgr2=0x7f224806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:17.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.686+0000 7f226b3f9700 1 --2- 172.21.15.3:0/1059012227 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f224806ac90 0x7f224806d150 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f226415bf20 tx=0x7f2254008040 comp rx=0 tx=0).stop 2024-05-08T21:22:17.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.686+0000 7f226b3f9700 1 -- 172.21.15.3:0/1059012227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f226406ec20 msgr2=0x7f22640c6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:17.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.686+0000 7f226b3f9700 1 --2- 172.21.15.3:0/1059012227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f226406ec20 0x7f22640c6830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f22600011e0 tx=0x7f226000cae0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.686+0000 7f226b3f9700 1 -- 172.21.15.3:0/1059012227 shutdown_connections 2024-05-08T21:22:17.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.686+0000 7f226b3f9700 1 --2- 172.21.15.3:0/1059012227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226415bb20 0x7f22640ae230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.686+0000 7f226b3f9700 1 --2- 172.21.15.3:0/1059012227 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f224806ac90 0x7f224806d150 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.686+0000 7f226b3f9700 1 --2- 172.21.15.3:0/1059012227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f226406ec20 0x7f22640c6830 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.686+0000 7f226b3f9700 1 --2- 172.21.15.3:0/1059012227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22640cfd10 0x7f2264150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.686+0000 7f226b3f9700 1 -- 172.21.15.3:0/1059012227 >> 172.21.15.3:0/1059012227 conn(0x7f2264099a50 msgr2=0x7f22640737c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:17.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.686+0000 7f226b3f9700 1 -- 172.21.15.3:0/1059012227 shutdown_connections 2024-05-08T21:22:17.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.686+0000 7f226b3f9700 1 -- 172.21.15.3:0/1059012227 wait complete. 2024-05-08T21:22:17.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:17.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.835+0000 7fb91b905700 1 Processor -- start 2024-05-08T21:22:17.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.835+0000 7fb91b905700 1 -- start start 2024-05-08T21:22:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.836+0000 7fb91b905700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb91406bd80 0x7fb9140d0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.836+0000 7fb91b905700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb91406c3d0 0x7fb91406c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.836+0000 7fb91b905700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9140cf990 con 0x7fb91415da10 2024-05-08T21:22:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.836+0000 7fb91b905700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb91409ab40 con 0x7fb91406bd80 2024-05-08T21:22:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.836+0000 7fb91b905700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91406cda0 con 0x7fb91406c3d0 2024-05-08T21:22:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.836+0000 7fb918ea0700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb91406bd80 0x7fb9140d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.836+0000 7fb919ea2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb91406c3d0 0x7fb91406c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.836+0000 7fb918ea0700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb91406bd80 0x7fb9140d0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:58618/0 (socket says 172.21.15.3:58618) 2024-05-08T21:22:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.836+0000 7fb918ea0700 1 -- 172.21.15.3:0/1346610191 learned_addr learned my addr 172.21.15.3:0/1346610191 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.836+0000 7fb9196a1700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fb91415da10 0x7fb91409a430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:50514/0 (socket says 172.21.15.3:50514) 2024-05-08T21:22:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.837+0000 7fb919ea2700 1 -- 172.21.15.3:0/1346610191 >> v2:172.21.15.186:3300/0 conn(0x7fb91406bd80 msgr2=0x7fb9140d0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.837+0000 7fb919ea2700 1 --2- 172.21.15.3:0/1346610191 >> v2:172.21.15.186:3300/0 conn(0x7fb91406bd80 0x7fb9140d0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.837+0000 7fb919ea2700 1 -- 172.21.15.3:0/1346610191 >> v1:172.21.15.186:6789/0 conn(0x7fb91415da10 legacy=0x7fb91409a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.837+0000 7fb919ea2700 1 -- 172.21.15.3:0/1346610191 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9140d04b0 con 0x7fb91406c3d0 2024-05-08T21:22:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.837+0000 7fb918ea0700 1 --2- 172.21.15.3:0/1346610191 >> v2:172.21.15.186:3300/0 conn(0x7fb91406bd80 0x7fb9140d0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.837+0000 7fb919ea2700 1 --2- 172.21.15.3:0/1346610191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb91406c3d0 0x7fb91406c860 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb90c009d90 tx=0x7fb90c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92af9c0ac39f1e30 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.837+0000 7fb90bfff700 1 -- 172.21.15.3:0/1346610191 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c00e070 con 0x7fb91406c3d0 2024-05-08T21:22:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.837+0000 7fb90bfff700 1 -- 172.21.15.3:0/1346610191 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90c00bb20 con 0x7fb91406c3d0 2024-05-08T21:22:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.837+0000 7fb90bfff700 1 -- 172.21.15.3:0/1346610191 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c012430 con 0x7fb91406c3d0 2024-05-08T21:22:17.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.838+0000 7fb91b905700 1 -- 172.21.15.3:0/1346610191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb91406c3d0 msgr2=0x7fb91406c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:17.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.838+0000 7fb91b905700 1 --2- 172.21.15.3:0/1346610191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb91406c3d0 0x7fb91406c860 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb90c009d90 tx=0x7fb90c00b330 comp rx=0 tx=0).stop 2024-05-08T21:22:17.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.838+0000 7fb91b905700 1 -- 172.21.15.3:0/1346610191 shutdown_connections 2024-05-08T21:22:17.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.838+0000 7fb91b905700 1 --2- 172.21.15.3:0/1346610191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb91406c3d0 0x7fb91406c860 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.838+0000 7fb91b905700 1 --2- 172.21.15.3:0/1346610191 >> v2:172.21.15.186:3300/0 conn(0x7fb91406bd80 0x7fb9140d0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.838+0000 7fb91b905700 1 -- 172.21.15.3:0/1346610191 >> 172.21.15.3:0/1346610191 conn(0x7fb9140b3010 msgr2=0x7fb9140b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:17.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.838+0000 7fb91b905700 1 -- 172.21.15.3:0/1346610191 shutdown_connections 2024-05-08T21:22:17.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.838+0000 7fb91b905700 1 -- 172.21.15.3:0/1346610191 wait complete. 2024-05-08T21:22:17.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.839+0000 7fb91b905700 1 Processor -- start 2024-05-08T21:22:17.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.839+0000 7fb91b905700 1 -- start start 2024-05-08T21:22:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.840+0000 7fb91b905700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb91406bd80 0x7fb9140b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.840+0000 7fb91b905700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb91406c3d0 0x7fb9140b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.840+0000 7fb91b905700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb91415da10 0x7fb91415bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.840+0000 7fb91b905700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9140b4990 con 0x7fb91406bd80 2024-05-08T21:22:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.840+0000 7fb91b905700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9140b4c70 con 0x7fb91406c3d0 2024-05-08T21:22:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.840+0000 7fb91b905700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9140b4f50 con 0x7fb91415da10 2024-05-08T21:22:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.840+0000 7fb9196a1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb91406bd80 0x7fb9140b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.840+0000 7fb9196a1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb91406bd80 0x7fb9140b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39956/0 (socket says 172.21.15.3:39956) 2024-05-08T21:22:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.840+0000 7fb9196a1700 1 -- 172.21.15.3:0/1568650023 learned_addr learned my addr 172.21.15.3:0/1568650023 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.840+0000 7fb918ea0700 1 --2- 172.21.15.3:0/1568650023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb91406c3d0 0x7fb9140b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.840+0000 7fb919ea2700 1 --2- 172.21.15.3:0/1568650023 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb91415da10 0x7fb91415bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.841+0000 7fb9196a1700 1 -- 172.21.15.3:0/1568650023 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb91415da10 msgr2=0x7fb91415bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.841+0000 7fb9196a1700 1 --2- 172.21.15.3:0/1568650023 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb91415da10 0x7fb91415bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.841+0000 7fb9196a1700 1 -- 172.21.15.3:0/1568650023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb91406c3d0 msgr2=0x7fb9140b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.841+0000 7fb9196a1700 1 --2- 172.21.15.3:0/1568650023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb91406c3d0 0x7fb9140b4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.841+0000 7fb9196a1700 1 -- 172.21.15.3:0/1568650023 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb904009550 con 0x7fb91406bd80 2024-05-08T21:22:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.841+0000 7fb9196a1700 1 --2- 172.21.15.3:0/1568650023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb91406bd80 0x7fb9140b5a30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb91000c520 tx=0x7fb91000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.841+0000 7fb909ffb700 1 -- 172.21.15.3:0/1568650023 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb910005cb0 con 0x7fb91406bd80 2024-05-08T21:22:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.841+0000 7fb909ffb700 1 -- 172.21.15.3:0/1568650023 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb910005e10 con 0x7fb91406bd80 2024-05-08T21:22:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.841+0000 7fb91b905700 1 -- 172.21.15.3:0/1568650023 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90c009a70 con 0x7fb91406bd80 2024-05-08T21:22:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.841+0000 7fb909ffb700 1 -- 172.21.15.3:0/1568650023 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9100072d0 con 0x7fb91406bd80 2024-05-08T21:22:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.841+0000 7fb91b905700 1 -- 172.21.15.3:0/1568650023 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9140b5510 con 0x7fb91406bd80 2024-05-08T21:22:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.842+0000 7fb91b905700 1 -- 172.21.15.3:0/1568650023 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9140cfba0 con 0x7fb91406bd80 2024-05-08T21:22:17.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.843+0000 7fb909ffb700 1 -- 172.21.15.3:0/1568650023 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb910007430 con 0x7fb91406bd80 2024-05-08T21:22:17.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.843+0000 7fb909ffb700 1 --2- 172.21.15.3:0/1568650023 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb8f806ac90 0x7fb8f806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:17.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.843+0000 7fb909ffb700 1 -- 172.21.15.3:0/1568650023 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb910088660 con 0x7fb91406bd80 2024-05-08T21:22:17.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.843+0000 7fb918ea0700 1 --2- 172.21.15.3:0/1568650023 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb8f806ac90 0x7fb8f806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:17.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.844+0000 7fb918ea0700 1 --2- 172.21.15.3:0/1568650023 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb8f806ac90 0x7fb8f806d150 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb9141626c0 tx=0x7fb9040092c0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:17.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:17.848+0000 7fb909ffb700 1 -- 172.21.15.3:0/1568650023 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb9140cfba0 con 0x7fb91406bd80 2024-05-08T21:22:18.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.067+0000 7fb91b905700 1 -- 172.21.15.3:0/1568650023 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9140cfba0 con 0x7fb91406bd80 2024-05-08T21:22:18.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.068+0000 7fb909ffb700 1 -- 172.21.15.3:0/1568650023 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15989 (secure 0 0 0) 0x7fb9140cfba0 con 0x7fb91406bd80 2024-05-08T21:22:18.070 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:18.070 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:53.064342+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:18.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.070+0000 7fb91b905700 1 -- 172.21.15.3:0/1568650023 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb8f806ac90 msgr2=0x7fb8f806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:18.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.070+0000 7fb91b905700 1 --2- 172.21.15.3:0/1568650023 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb8f806ac90 0x7fb8f806d150 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb9141626c0 tx=0x7fb9040092c0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.070+0000 7fb91b905700 1 -- 172.21.15.3:0/1568650023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb91406bd80 msgr2=0x7fb9140b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:18.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.070+0000 7fb91b905700 1 --2- 172.21.15.3:0/1568650023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb91406bd80 0x7fb9140b5a30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb91000c520 tx=0x7fb91000c830 comp rx=0 tx=0).stop 2024-05-08T21:22:18.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.070+0000 7fb91b905700 1 -- 172.21.15.3:0/1568650023 shutdown_connections 2024-05-08T21:22:18.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.070+0000 7fb91b905700 1 --2- 172.21.15.3:0/1568650023 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb91415da10 0x7fb91415bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.070+0000 7fb91b905700 1 --2- 172.21.15.3:0/1568650023 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb8f806ac90 0x7fb8f806d150 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.070+0000 7fb91b905700 1 --2- 172.21.15.3:0/1568650023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb91406bd80 0x7fb9140b5a30 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.070+0000 7fb91b905700 1 --2- 172.21.15.3:0/1568650023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb91406c3d0 0x7fb9140b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.070+0000 7fb91b905700 1 -- 172.21.15.3:0/1568650023 >> 172.21.15.3:0/1568650023 conn(0x7fb9140b3010 msgr2=0x7fb9140b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:18.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.070+0000 7fb91b905700 1 -- 172.21.15.3:0/1568650023 shutdown_connections 2024-05-08T21:22:18.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.071+0000 7fb91b905700 1 -- 172.21.15.3:0/1568650023 wait complete. 2024-05-08T21:22:18.086 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-08T21:21:49.638998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:21:51.024632+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:22:18.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:22:18.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.224+0000 7fc212b50700 1 Processor -- start 2024-05-08T21:22:18.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.224+0000 7fc212b50700 1 -- start start 2024-05-08T21:22:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.225+0000 7fc212b50700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc20c06edf0 0x7fc20c159f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.225+0000 7fc212b50700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20c0cc980 con 0x7fc20c15bb20 2024-05-08T21:22:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.225+0000 7fc212b50700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20c0d00f0 con 0x7fc20c0cfd10 2024-05-08T21:22:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.225+0000 7fc212b50700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc20c0ccbd0 con 0x7fc20c06edf0 2024-05-08T21:22:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.225+0000 7fc20bfff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc20c15bb20 0x7fc20c06e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55264/0 (socket says 172.21.15.3:55264) 2024-05-08T21:22:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.225+0000 7fc20bfff700 1 -- 172.21.15.3:0/415778276 learned_addr learned my addr 172.21.15.3:0/415778276 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.225+0000 7fc2110ed700 1 --2- 172.21.15.3:0/415778276 >> v2:172.21.15.186:3300/0 conn(0x7fc20c06edf0 0x7fc20c159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.225+0000 7fc20b7fe700 1 -- 172.21.15.3:0/415778276 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781313352 0 0) 0x7fc20c0cc980 con 0x7fc20c15bb20 2024-05-08T21:22:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.226+0000 7fc20b7fe700 1 -- 172.21.15.3:0/415778276 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f80035b0 con 0x7fc20c15bb20 2024-05-08T21:22:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.226+0000 7fc2110ed700 1 -- 172.21.15.3:0/415778276 >> v1:172.21.15.186:6789/0 conn(0x7fc20c0cfd10 legacy=0x7fc20c06dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.226+0000 7fc2110ed700 1 -- 172.21.15.3:0/415778276 >> v1:172.21.15.3:6789/0 conn(0x7fc20c15bb20 legacy=0x7fc20c06e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.226+0000 7fc2110ed700 1 -- 172.21.15.3:0/415778276 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20c0cceb0 con 0x7fc20c06edf0 2024-05-08T21:22:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.226+0000 7fc2110ed700 1 --2- 172.21.15.3:0/415778276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc20c06edf0 0x7fc20c159f80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc204009d90 tx=0x7fc20400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d38bd43db40d0df server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:18.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.226+0000 7fc20b7fe700 1 -- 172.21.15.3:0/415778276 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20400e070 con 0x7fc20c06edf0 2024-05-08T21:22:18.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.226+0000 7fc20b7fe700 1 -- 172.21.15.3:0/415778276 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20400bb20 con 0x7fc20c06edf0 2024-05-08T21:22:18.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.226+0000 7fc20b7fe700 1 -- 172.21.15.3:0/415778276 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc204012430 con 0x7fc20c06edf0 2024-05-08T21:22:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.227+0000 7fc212b50700 1 -- 172.21.15.3:0/415778276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc20c06edf0 msgr2=0x7fc20c159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.227+0000 7fc212b50700 1 --2- 172.21.15.3:0/415778276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc20c06edf0 0x7fc20c159f80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc204009d90 tx=0x7fc20400b330 comp rx=0 tx=0).stop 2024-05-08T21:22:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.227+0000 7fc212b50700 1 -- 172.21.15.3:0/415778276 shutdown_connections 2024-05-08T21:22:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.227+0000 7fc212b50700 1 --2- 172.21.15.3:0/415778276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc20c06edf0 0x7fc20c159f80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.227+0000 7fc212b50700 1 -- 172.21.15.3:0/415778276 >> 172.21.15.3:0/415778276 conn(0x7fc20c099a50 msgr2=0x7fc20c0d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.227+0000 7fc212b50700 1 -- 172.21.15.3:0/415778276 shutdown_connections 2024-05-08T21:22:18.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.227+0000 7fc212b50700 1 -- 172.21.15.3:0/415778276 wait complete. 2024-05-08T21:22:18.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.228+0000 7fc212b50700 1 Processor -- start 2024-05-08T21:22:18.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.228+0000 7fc212b50700 1 -- start start 2024-05-08T21:22:18.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.229+0000 7fc212b50700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc20c06edf0 0x7fc20c06a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:18.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.229+0000 7fc212b50700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc20c0cfd10 0x7fc20c06aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:18.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.229+0000 7fc212b50700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc20c15bb20 0x7fc20c0c4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.229+0000 7fc212b50700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c15e560 con 0x7fc20c0cfd10 2024-05-08T21:22:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.229+0000 7fc212b50700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c15dbc0 con 0x7fc20c06edf0 2024-05-08T21:22:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.229+0000 7fc212b50700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c15dea0 con 0x7fc20c15bb20 2024-05-08T21:22:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.229+0000 7fc20bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc20c0cfd10 0x7fc20c06aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.229+0000 7fc20bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc20c0cfd10 0x7fc20c06aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39964/0 (socket says 172.21.15.3:39964) 2024-05-08T21:22:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.229+0000 7fc20bfff700 1 -- 172.21.15.3:0/3479355031 learned_addr learned my addr 172.21.15.3:0/3479355031 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.229+0000 7fc2110ed700 1 --2- 172.21.15.3:0/3479355031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc20c15bb20 0x7fc20c0c4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.229+0000 7fc2108ec700 1 --2- 172.21.15.3:0/3479355031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc20c06edf0 0x7fc20c06a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.230+0000 7fc20bfff700 1 -- 172.21.15.3:0/3479355031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc20c15bb20 msgr2=0x7fc20c0c4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.230+0000 7fc20bfff700 1 --2- 172.21.15.3:0/3479355031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc20c15bb20 0x7fc20c0c4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.230+0000 7fc20bfff700 1 -- 172.21.15.3:0/3479355031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc20c06edf0 msgr2=0x7fc20c06a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.230+0000 7fc20bfff700 1 --2- 172.21.15.3:0/3479355031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc20c06edf0 0x7fc20c06a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.230+0000 7fc20bfff700 1 -- 172.21.15.3:0/3479355031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc204009a70 con 0x7fc20c0cfd10 2024-05-08T21:22:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.230+0000 7fc20bfff700 1 --2- 172.21.15.3:0/3479355031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc20c0cfd10 0x7fc20c06aea0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4005910 tx=0x7fc1f400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.230+0000 7fc2097fa700 1 -- 172.21.15.3:0/3479355031 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4005d70 con 0x7fc20c0cfd10 2024-05-08T21:22:18.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.230+0000 7fc212b50700 1 -- 172.21.15.3:0/3479355031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc20c15e180 con 0x7fc20c0cfd10 2024-05-08T21:22:18.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.230+0000 7fc2097fa700 1 -- 172.21.15.3:0/3479355031 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f400ce80 con 0x7fc20c0cfd10 2024-05-08T21:22:18.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.230+0000 7fc2097fa700 1 -- 172.21.15.3:0/3479355031 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f40072f0 con 0x7fc20c0cfd10 2024-05-08T21:22:18.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.230+0000 7fc212b50700 1 -- 172.21.15.3:0/3479355031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20c15e980 con 0x7fc20c0cfd10 2024-05-08T21:22:18.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.231+0000 7fc212b50700 1 -- 172.21.15.3:0/3479355031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc20c15e6a0 con 0x7fc20c0cfd10 2024-05-08T21:22:18.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.232+0000 7fc2097fa700 1 -- 172.21.15.3:0/3479355031 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc1f4007450 con 0x7fc20c0cfd10 2024-05-08T21:22:18.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.232+0000 7fc2097fa700 1 --2- 172.21.15.3:0/3479355031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc1ec06ac90 0x7fc1ec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:18.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.232+0000 7fc2097fa700 1 -- 172.21.15.3:0/3479355031 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1f4088700 con 0x7fc20c0cfd10 2024-05-08T21:22:18.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.232+0000 7fc2108ec700 1 --2- 172.21.15.3:0/3479355031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc1ec06ac90 0x7fc1ec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:18.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.233+0000 7fc2108ec700 1 --2- 172.21.15.3:0/3479355031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc1ec06ac90 0x7fc1ec06d150 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc007fd0 tx=0x7fc1fc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:18.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.237+0000 7fc2097fa700 1 -- 172.21.15.3:0/3479355031 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc20c15e6a0 con 0x7fc20c0cfd10 2024-05-08T21:22:18.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.454+0000 7fc212b50700 1 -- 172.21.15.3:0/3479355031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc20c15e6a0 con 0x7fc20c0cfd10 2024-05-08T21:22:18.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.454+0000 7fc2097fa700 1 -- 172.21.15.3:0/3479355031 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc20c15e6a0 con 0x7fc20c0cfd10 2024-05-08T21:22:18.456 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:22:18.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.457+0000 7fc212b50700 1 -- 172.21.15.3:0/3479355031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc1ec06ac90 msgr2=0x7fc1ec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:18.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.457+0000 7fc212b50700 1 --2- 172.21.15.3:0/3479355031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc1ec06ac90 0x7fc1ec06d150 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc007fd0 tx=0x7fc1fc009040 comp rx=0 tx=0).stop 2024-05-08T21:22:18.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.457+0000 7fc212b50700 1 -- 172.21.15.3:0/3479355031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc20c0cfd10 msgr2=0x7fc20c06aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:18.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.457+0000 7fc212b50700 1 --2- 172.21.15.3:0/3479355031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc20c0cfd10 0x7fc20c06aea0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4005910 tx=0x7fc1f400c7b0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.457+0000 7fc212b50700 1 -- 172.21.15.3:0/3479355031 shutdown_connections 2024-05-08T21:22:18.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.457+0000 7fc212b50700 1 --2- 172.21.15.3:0/3479355031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc20c15bb20 0x7fc20c0c4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.457+0000 7fc212b50700 1 --2- 172.21.15.3:0/3479355031 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc1ec06ac90 0x7fc1ec06d150 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.457+0000 7fc212b50700 1 --2- 172.21.15.3:0/3479355031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc20c06edf0 0x7fc20c06a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.457+0000 7fc212b50700 1 --2- 172.21.15.3:0/3479355031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc20c0cfd10 0x7fc20c06aea0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.457+0000 7fc212b50700 1 -- 172.21.15.3:0/3479355031 >> 172.21.15.3:0/3479355031 conn(0x7fc20c099a50 msgr2=0x7fc20c0cebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:18.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.457+0000 7fc212b50700 1 -- 172.21.15.3:0/3479355031 shutdown_connections 2024-05-08T21:22:18.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.457+0000 7fc212b50700 1 -- 172.21.15.3:0/3479355031 wait complete. 2024-05-08T21:22:18.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-08T21:22:18.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.612+0000 7f6f33740700 1 Processor -- start 2024-05-08T21:22:18.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.612+0000 7f6f33740700 1 -- start start 2024-05-08T21:22:18.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.613+0000 7f6f33740700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6f2c15d8b0 0x7f6f2c06a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:18.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.613+0000 7f6f33740700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f2c06bfe0 0x7f6f2c15dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:18.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.613+0000 7f6f33740700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2c06bb50 con 0x7f6f2c0b3730 2024-05-08T21:22:18.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.613+0000 7f6f33740700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f2c06af40 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.613+0000 7f6f33740700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c06b220 con 0x7f6f2c06bfe0 2024-05-08T21:22:18.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.613+0000 7f6f314dc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6f2c15d8b0 0x7f6f2c06a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:18.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.613+0000 7f6f31cdd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6f2c0b3730 0x7f6f2c161500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55268/0 (socket says 172.21.15.3:55268) 2024-05-08T21:22:18.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.613+0000 7f6f31cdd700 1 -- 172.21.15.3:0/3863557837 learned_addr learned my addr 172.21.15.3:0/3863557837 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:18.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.613+0000 7f6f30cdb700 1 --2- 172.21.15.3:0/3863557837 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f2c06bfe0 0x7f6f2c15dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:18.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.613+0000 7f6f314dc700 1 -- 172.21.15.3:0/3863557837 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f2c06bfe0 msgr2=0x7f6f2c15dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:18.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.613+0000 7f6f314dc700 1 --2- 172.21.15.3:0/3863557837 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f2c06bfe0 0x7f6f2c15dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.613+0000 7f6f314dc700 1 -- 172.21.15.3:0/3863557837 >> v1:172.21.15.3:6789/0 conn(0x7f6f2c0b3730 legacy=0x7f6f2c161500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:18.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.613+0000 7f6f314dc700 1 -- 172.21.15.3:0/3863557837 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2c06b500 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.613+0000 7f6f23fff700 1 -- 172.21.15.3:0/3863557837 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261003110 0 0) 0x7f6f2c06bb50 con 0x7f6f2c0b3730 2024-05-08T21:22:18.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.614+0000 7f6f314dc700 1 --2- 172.21.15.3:0/3863557837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c15d8b0 0x7f6f2c06a960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6f28009840 tx=0x7f6f28009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87151f55db262551 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:18.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.614+0000 7f6f23fff700 1 -- 172.21.15.3:0/3863557837 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f28011830 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.614+0000 7f6f23fff700 1 -- 172.21.15.3:0/3863557837 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f28011990 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.614+0000 7f6f23fff700 1 -- 172.21.15.3:0/3863557837 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f28011d20 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.614+0000 7f6f33740700 1 -- 172.21.15.3:0/3863557837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c15d8b0 msgr2=0x7f6f2c06a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:18.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.614+0000 7f6f33740700 1 --2- 172.21.15.3:0/3863557837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c15d8b0 0x7f6f2c06a960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6f28009840 tx=0x7f6f28009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:18.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.614+0000 7f6f33740700 1 -- 172.21.15.3:0/3863557837 shutdown_connections 2024-05-08T21:22:18.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.614+0000 7f6f33740700 1 --2- 172.21.15.3:0/3863557837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c15d8b0 0x7f6f2c06a960 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.614+0000 7f6f33740700 1 --2- 172.21.15.3:0/3863557837 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f2c06bfe0 0x7f6f2c15dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.614+0000 7f6f33740700 1 -- 172.21.15.3:0/3863557837 >> 172.21.15.3:0/3863557837 conn(0x7f6f2c0cfbb0 msgr2=0x7f6f2c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:18.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.615+0000 7f6f33740700 1 -- 172.21.15.3:0/3863557837 shutdown_connections 2024-05-08T21:22:18.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.615+0000 7f6f33740700 1 -- 172.21.15.3:0/3863557837 wait complete. 2024-05-08T21:22:18.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.616+0000 7f6f33740700 1 Processor -- start 2024-05-08T21:22:18.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.616+0000 7f6f33740700 1 -- start start 2024-05-08T21:22:18.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f33740700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f2c06bfe0 0x7f6f2c0ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:18.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f33740700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f2c0b3730 0x7f6f2c0c5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f33740700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c15d8b0 0x7f6f2c0ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f33740700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c0c5a90 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f33740700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c14f600 con 0x7f6f2c0b3730 2024-05-08T21:22:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f33740700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c0cd510 con 0x7f6f2c06bfe0 2024-05-08T21:22:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f31cdd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c15d8b0 0x7f6f2c0ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f31cdd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c15d8b0 0x7f6f2c0ce5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39980/0 (socket says 172.21.15.3:39980) 2024-05-08T21:22:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f31cdd700 1 -- 172.21.15.3:0/1957325764 learned_addr learned my addr 172.21.15.3:0/1957325764 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f314dc700 1 --2- 172.21.15.3:0/1957325764 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f2c06bfe0 0x7f6f2c0ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f30cdb700 1 --2- 172.21.15.3:0/1957325764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f2c0b3730 0x7f6f2c0c5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f31cdd700 1 -- 172.21.15.3:0/1957325764 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f2c06bfe0 msgr2=0x7f6f2c0ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f31cdd700 1 --2- 172.21.15.3:0/1957325764 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f2c06bfe0 0x7f6f2c0ccef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f31cdd700 1 -- 172.21.15.3:0/1957325764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f2c0b3730 msgr2=0x7f6f2c0c5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f31cdd700 1 --2- 172.21.15.3:0/1957325764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f2c0b3730 0x7f6f2c0c5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f31cdd700 1 -- 172.21.15.3:0/1957325764 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f280094a0 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f31cdd700 1 --2- 172.21.15.3:0/1957325764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c15d8b0 0x7f6f2c0ce5b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6f24002230 tx=0x7f6f2400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:18.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f21ffb700 1 -- 172.21.15.3:0/1957325764 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2400aac0 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.617+0000 7f6f21ffb700 1 -- 172.21.15.3:0/1957325764 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2400ac40 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.618+0000 7f6f33740700 1 -- 172.21.15.3:0/1957325764 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f2c0cd7f0 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.618+0000 7f6f21ffb700 1 -- 172.21.15.3:0/1957325764 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f240070d0 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.618+0000 7f6f33740700 1 -- 172.21.15.3:0/1957325764 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f2c0cddb0 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.619+0000 7f6f33740700 1 -- 172.21.15.3:0/1957325764 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f2c074bc0 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.619+0000 7f6f21ffb700 1 -- 172.21.15.3:0/1957325764 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6f24007250 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.620+0000 7f6f21ffb700 1 --2- 172.21.15.3:0/1957325764 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6f1006ac90 0x7f6f1006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:18.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.620+0000 7f6f21ffb700 1 -- 172.21.15.3:0/1957325764 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f24089090 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.620+0000 7f6f314dc700 1 --2- 172.21.15.3:0/1957325764 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6f1006ac90 0x7f6f1006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:18.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.620+0000 7f6f314dc700 1 --2- 172.21.15.3:0/1957325764 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6f1006ac90 0x7f6f1006d150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6f280090b0 tx=0x7f6f2801c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:18.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.624+0000 7f6f21ffb700 1 -- 172.21.15.3:0/1957325764 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6f24054270 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.843+0000 7f6f33740700 1 -- 172.21.15.3:0/1957325764 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6f2c06a960 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.844+0000 7f6f21ffb700 1 -- 172.21.15.3:0/1957325764 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6f24057890 con 0x7f6f2c15d8b0 2024-05-08T21:22:18.845 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:22:18.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.846+0000 7f6f33740700 1 -- 172.21.15.3:0/1957325764 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6f1006ac90 msgr2=0x7f6f1006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:18.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.846+0000 7f6f33740700 1 --2- 172.21.15.3:0/1957325764 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6f1006ac90 0x7f6f1006d150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6f280090b0 tx=0x7f6f2801c040 comp rx=0 tx=0).stop 2024-05-08T21:22:18.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.846+0000 7f6f33740700 1 -- 172.21.15.3:0/1957325764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c15d8b0 msgr2=0x7f6f2c0ce5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.846+0000 7f6f33740700 1 --2- 172.21.15.3:0/1957325764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c15d8b0 0x7f6f2c0ce5b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6f24002230 tx=0x7f6f2400cf50 comp rx=0 tx=0).stop 2024-05-08T21:22:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.846+0000 7f6f33740700 1 -- 172.21.15.3:0/1957325764 shutdown_connections 2024-05-08T21:22:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.846+0000 7f6f33740700 1 --2- 172.21.15.3:0/1957325764 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f2c06bfe0 0x7f6f2c0ccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.846+0000 7f6f33740700 1 --2- 172.21.15.3:0/1957325764 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6f1006ac90 0x7f6f1006d150 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.846+0000 7f6f33740700 1 --2- 172.21.15.3:0/1957325764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c15d8b0 0x7f6f2c0ce5b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.846+0000 7f6f33740700 1 --2- 172.21.15.3:0/1957325764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f2c0b3730 0x7f6f2c0c5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.846+0000 7f6f33740700 1 -- 172.21.15.3:0/1957325764 >> 172.21.15.3:0/1957325764 conn(0x7f6f2c0cfbb0 msgr2=0x7f6f2c0b2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.846+0000 7f6f33740700 1 -- 172.21.15.3:0/1957325764 shutdown_connections 2024-05-08T21:22:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.846+0000 7f6f33740700 1 -- 172.21.15.3:0/1957325764 wait complete. 2024-05-08T21:22:18.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:22:19.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.998+0000 7f6c9a3ba700 1 Processor -- start 2024-05-08T21:22:19.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.998+0000 7f6c9a3ba700 1 -- start start 2024-05-08T21:22:19.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.999+0000 7f6c9a3ba700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6c940b44e0 0x7f6c9406b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.999+0000 7f6c9a3ba700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6c940b48c0 0x7f6c9406beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.999+0000 7f6c9a3ba700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9415f3c0 con 0x7f6c9415bf60 2024-05-08T21:22:19.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.999+0000 7f6c9a3ba700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9415cae0 con 0x7f6c940b48c0 2024-05-08T21:22:19.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.999+0000 7f6c9a3ba700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c940a84b0 con 0x7f6c940b44e0 2024-05-08T21:22:19.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.999+0000 7f6c937fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6c940b48c0 0x7f6c9406beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.999+0000 7f6c937fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6c940b48c0 0x7f6c9406beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39982/0 (socket says 172.21.15.3:39982) 2024-05-08T21:22:19.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:18.999+0000 7f6c937fe700 1 -- 172.21.15.3:0/2666786220 learned_addr learned my addr 172.21.15.3:0/2666786220 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:19.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.000+0000 7f6c93fff700 1 --2- 172.21.15.3:0/2666786220 >> v2:172.21.15.186:3300/0 conn(0x7f6c940b44e0 0x7f6c9406b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.000+0000 7f6c937fe700 1 -- 172.21.15.3:0/2666786220 >> v2:172.21.15.186:3300/0 conn(0x7f6c940b44e0 msgr2=0x7f6c9406b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:19.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.000+0000 7f6c937fe700 1 --2- 172.21.15.3:0/2666786220 >> v2:172.21.15.186:3300/0 conn(0x7f6c940b44e0 0x7f6c9406b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.000+0000 7f6c937fe700 1 -- 172.21.15.3:0/2666786220 >> v1:172.21.15.186:6789/0 conn(0x7f6c9415bf60 legacy=0x7f6c9415c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:19.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.000+0000 7f6c937fe700 1 -- 172.21.15.3:0/2666786220 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c94160160 con 0x7f6c940b48c0 2024-05-08T21:22:19.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.000+0000 7f6c937fe700 1 --2- 172.21.15.3:0/2666786220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c940b48c0 0x7f6c9406beb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6c9406ff10 tx=0x7f6c88009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ef5d0ccbd0fe68 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:19.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.000+0000 7f6c92ffd700 1 -- 172.21.15.3:0/2666786220 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88004230 con 0x7f6c940b48c0 2024-05-08T21:22:19.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.000+0000 7f6c92ffd700 1 -- 172.21.15.3:0/2666786220 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c88004390 con 0x7f6c940b48c0 2024-05-08T21:22:19.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.000+0000 7f6c92ffd700 1 -- 172.21.15.3:0/2666786220 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c880036a0 con 0x7f6c940b48c0 2024-05-08T21:22:19.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.001+0000 7f6c9a3ba700 1 -- 172.21.15.3:0/2666786220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c940b48c0 msgr2=0x7f6c9406beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.001+0000 7f6c9a3ba700 1 --2- 172.21.15.3:0/2666786220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c940b48c0 0x7f6c9406beb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6c9406ff10 tx=0x7f6c88009bd0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.001+0000 7f6c9a3ba700 1 -- 172.21.15.3:0/2666786220 shutdown_connections 2024-05-08T21:22:19.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.001+0000 7f6c9a3ba700 1 --2- 172.21.15.3:0/2666786220 >> v2:172.21.15.186:3300/0 conn(0x7f6c940b44e0 0x7f6c9406b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.001+0000 7f6c9a3ba700 1 --2- 172.21.15.3:0/2666786220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c940b48c0 0x7f6c9406beb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.001+0000 7f6c9a3ba700 1 -- 172.21.15.3:0/2666786220 >> 172.21.15.3:0/2666786220 conn(0x7f6c940713c0 msgr2=0x7f6c940717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:19.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.001+0000 7f6c9a3ba700 1 -- 172.21.15.3:0/2666786220 shutdown_connections 2024-05-08T21:22:19.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.001+0000 7f6c9a3ba700 1 -- 172.21.15.3:0/2666786220 wait complete. 2024-05-08T21:22:19.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.002+0000 7f6c9a3ba700 1 Processor -- start 2024-05-08T21:22:19.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.002+0000 7f6c9a3ba700 1 -- start start 2024-05-08T21:22:19.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.003+0000 7f6c9a3ba700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c940b44e0 0x7f6c941f0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.003+0000 7f6c9a3ba700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c940b48c0 0x7f6c941f50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.003+0000 7f6c9a3ba700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9415bf60 0x7f6c9415a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.003+0000 7f6c9a3ba700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9415a440 con 0x7f6c9415bf60 2024-05-08T21:22:19.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.003+0000 7f6c9a3ba700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c941f5620 con 0x7f6c940b48c0 2024-05-08T21:22:19.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.003+0000 7f6c9a3ba700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c94159830 con 0x7f6c940b44e0 2024-05-08T21:22:19.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.003+0000 7f6c98957700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9415bf60 0x7f6c9415a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.003+0000 7f6c98957700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9415bf60 0x7f6c9415a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39988/0 (socket says 172.21.15.3:39988) 2024-05-08T21:22:19.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.003+0000 7f6c98957700 1 -- 172.21.15.3:0/3075887089 learned_addr learned my addr 172.21.15.3:0/3075887089 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:19.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.004+0000 7f6c937fe700 1 --2- 172.21.15.3:0/3075887089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c940b48c0 0x7f6c941f50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.004+0000 7f6c93fff700 1 --2- 172.21.15.3:0/3075887089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c940b44e0 0x7f6c941f0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.004+0000 7f6c98957700 1 -- 172.21.15.3:0/3075887089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c940b44e0 msgr2=0x7f6c941f0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.004+0000 7f6c98957700 1 --2- 172.21.15.3:0/3075887089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c940b44e0 0x7f6c941f0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.004+0000 7f6c98957700 1 -- 172.21.15.3:0/3075887089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c940b48c0 msgr2=0x7f6c941f50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.004+0000 7f6c98957700 1 --2- 172.21.15.3:0/3075887089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c940b48c0 0x7f6c941f50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.004+0000 7f6c98957700 1 -- 172.21.15.3:0/3075887089 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c88009550 con 0x7f6c9415bf60 2024-05-08T21:22:19.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.004+0000 7f6c98957700 1 --2- 172.21.15.3:0/3075887089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9415bf60 0x7f6c9415a8d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00cb60 tx=0x7f6c8c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:19.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.004+0000 7f6c90ff9700 1 -- 172.21.15.3:0/3075887089 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c00ab10 con 0x7f6c9415bf60 2024-05-08T21:22:19.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.004+0000 7f6c90ff9700 1 -- 172.21.15.3:0/3075887089 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8c00ac70 con 0x7f6c9415bf60 2024-05-08T21:22:19.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.004+0000 7f6c9a3ba700 1 -- 172.21.15.3:0/3075887089 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c94159b10 con 0x7f6c9415bf60 2024-05-08T21:22:19.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.004+0000 7f6c90ff9700 1 -- 172.21.15.3:0/3075887089 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c012420 con 0x7f6c9415bf60 2024-05-08T21:22:19.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.004+0000 7f6c9a3ba700 1 -- 172.21.15.3:0/3075887089 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9415a0d0 con 0x7f6c9415bf60 2024-05-08T21:22:19.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.005+0000 7f6c9a3ba700 1 -- 172.21.15.3:0/3075887089 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c94074bc0 con 0x7f6c9415bf60 2024-05-08T21:22:19.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.006+0000 7f6c90ff9700 1 -- 172.21.15.3:0/3075887089 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6c8c012580 con 0x7f6c9415bf60 2024-05-08T21:22:19.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.006+0000 7f6c90ff9700 1 --2- 172.21.15.3:0/3075887089 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6c8006ac90 0x7f6c8006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.006+0000 7f6c90ff9700 1 -- 172.21.15.3:0/3075887089 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c8c0896d0 con 0x7f6c9415bf60 2024-05-08T21:22:19.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.006+0000 7f6c93fff700 1 --2- 172.21.15.3:0/3075887089 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6c8006ac90 0x7f6c8006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.007+0000 7f6c93fff700 1 --2- 172.21.15.3:0/3075887089 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6c8006ac90 0x7f6c8006d150 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400b9b0 tx=0x7f6c84008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.011+0000 7f6c90ff9700 1 -- 172.21.15.3:0/3075887089 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6c8c0548b0 con 0x7f6c9415bf60 2024-05-08T21:22:19.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.231+0000 7f6c9a3ba700 1 -- 172.21.15.3:0/3075887089 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6c9415a580 con 0x7f6c9415bf60 2024-05-08T21:22:19.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.231+0000 7f6c90ff9700 1 -- 172.21.15.3:0/3075887089 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6c8c057ed0 con 0x7f6c9415bf60 2024-05-08T21:22:19.233 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:22:19.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.234+0000 7f6c9a3ba700 1 -- 172.21.15.3:0/3075887089 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6c8006ac90 msgr2=0x7f6c8006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.234+0000 7f6c9a3ba700 1 --2- 172.21.15.3:0/3075887089 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6c8006ac90 0x7f6c8006d150 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400b9b0 tx=0x7f6c84008040 comp rx=0 tx=0).stop 2024-05-08T21:22:19.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.234+0000 7f6c9a3ba700 1 -- 172.21.15.3:0/3075887089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9415bf60 msgr2=0x7f6c9415a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.234+0000 7f6c9a3ba700 1 --2- 172.21.15.3:0/3075887089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9415bf60 0x7f6c9415a8d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00cb60 tx=0x7f6c8c00cf20 comp rx=0 tx=0).stop 2024-05-08T21:22:19.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.234+0000 7f6c9a3ba700 1 -- 172.21.15.3:0/3075887089 shutdown_connections 2024-05-08T21:22:19.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.234+0000 7f6c9a3ba700 1 --2- 172.21.15.3:0/3075887089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c940b44e0 0x7f6c941f0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.234+0000 7f6c9a3ba700 1 --2- 172.21.15.3:0/3075887089 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6c8006ac90 0x7f6c8006d150 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.234+0000 7f6c9a3ba700 1 --2- 172.21.15.3:0/3075887089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9415bf60 0x7f6c9415a8d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.234+0000 7f6c9a3ba700 1 --2- 172.21.15.3:0/3075887089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c940b48c0 0x7f6c941f50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.234+0000 7f6c9a3ba700 1 -- 172.21.15.3:0/3075887089 >> 172.21.15.3:0/3075887089 conn(0x7f6c940713c0 msgr2=0x7f6c9415e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:19.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.234+0000 7f6c9a3ba700 1 -- 172.21.15.3:0/3075887089 shutdown_connections 2024-05-08T21:22:19.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.234+0000 7f6c9a3ba700 1 -- 172.21.15.3:0/3075887089 wait complete. 2024-05-08T21:22:19.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:19.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.385+0000 7ff082e2b700 1 Processor -- start 2024-05-08T21:22:19.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.385+0000 7ff082e2b700 1 -- start start 2024-05-08T21:22:19.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.386+0000 7ff082e2b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff07c06dd40 0x7ff07c0d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.386+0000 7ff082e2b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07c0ce670 con 0x7ff07c0d0830 2024-05-08T21:22:19.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.386+0000 7ff082e2b700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07c06fd90 con 0x7ff07c15a680 2024-05-08T21:22:19.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.386+0000 7ff082e2b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c0d0ca0 con 0x7ff07c06dd40 2024-05-08T21:22:19.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.386+0000 7ff0813c8700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff07c0d0830 0x7ff07c0ceb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55272/0 (socket says 172.21.15.3:55272) 2024-05-08T21:22:19.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.386+0000 7ff0813c8700 1 -- 172.21.15.3:0/2235881943 learned_addr learned my addr 172.21.15.3:0/2235881943 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:19.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.386+0000 7ff07bfff700 1 --2- 172.21.15.3:0/2235881943 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff07c06dd40 0x7ff07c0d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.386+0000 7ff07b7fe700 1 -- 172.21.15.3:0/2235881943 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321798182 0 0) 0x7ff07c0ce670 con 0x7ff07c0d0830 2024-05-08T21:22:19.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.386+0000 7ff07b7fe700 1 -- 172.21.15.3:0/2235881943 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff06c0035b0 con 0x7ff07c0d0830 2024-05-08T21:22:19.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.386+0000 7ff07bfff700 1 -- 172.21.15.3:0/2235881943 >> v1:172.21.15.186:6789/0 conn(0x7ff07c15a680 legacy=0x7ff07c15b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.386+0000 7ff07b7fe700 1 -- 172.21.15.3:0/2235881943 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009024934 0 0) 0x7ff07c06fd90 con 0x7ff07c15a680 2024-05-08T21:22:19.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.386+0000 7ff07bfff700 1 -- 172.21.15.3:0/2235881943 >> v1:172.21.15.3:6789/0 conn(0x7ff07c0d0830 legacy=0x7ff07c0ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.386+0000 7ff07bfff700 1 -- 172.21.15.3:0/2235881943 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07c06f7d0 con 0x7ff07c06dd40 2024-05-08T21:22:19.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.387+0000 7ff07bfff700 1 --2- 172.21.15.3:0/2235881943 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff07c06dd40 0x7ff07c0d02f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff068009870 tx=0x7ff068009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2a6c9d05d7f80bf server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:19.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.387+0000 7ff07b7fe700 1 -- 172.21.15.3:0/2235881943 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff068011830 con 0x7ff07c06dd40 2024-05-08T21:22:19.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.387+0000 7ff07b7fe700 1 -- 172.21.15.3:0/2235881943 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff068011990 con 0x7ff07c06dd40 2024-05-08T21:22:19.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.387+0000 7ff07b7fe700 1 -- 172.21.15.3:0/2235881943 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff068011d20 con 0x7ff07c06dd40 2024-05-08T21:22:19.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.388+0000 7ff082e2b700 1 -- 172.21.15.3:0/2235881943 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff07c06dd40 msgr2=0x7ff07c0d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.388+0000 7ff082e2b700 1 --2- 172.21.15.3:0/2235881943 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff07c06dd40 0x7ff07c0d02f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff068009870 tx=0x7ff068009b80 comp rx=0 tx=0).stop 2024-05-08T21:22:19.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.388+0000 7ff082e2b700 1 -- 172.21.15.3:0/2235881943 shutdown_connections 2024-05-08T21:22:19.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.388+0000 7ff082e2b700 1 --2- 172.21.15.3:0/2235881943 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff07c06dd40 0x7ff07c0d02f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.388+0000 7ff082e2b700 1 -- 172.21.15.3:0/2235881943 >> 172.21.15.3:0/2235881943 conn(0x7ff07c0b5810 msgr2=0x7ff07c0b5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:19.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.388+0000 7ff082e2b700 1 -- 172.21.15.3:0/2235881943 shutdown_connections 2024-05-08T21:22:19.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.388+0000 7ff082e2b700 1 -- 172.21.15.3:0/2235881943 wait complete. 2024-05-08T21:22:19.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.389+0000 7ff082e2b700 1 Processor -- start 2024-05-08T21:22:19.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.389+0000 7ff082e2b700 1 -- start start 2024-05-08T21:22:19.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff082e2b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff07c06dd40 0x7ff07c1f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff082e2b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff07c0d0830 0x7ff07c09dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff082e2b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff07c15a680 0x7ff07c14ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff082e2b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c09e0f0 con 0x7ff07c0d0830 2024-05-08T21:22:19.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff082e2b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c09a410 con 0x7ff07c15a680 2024-05-08T21:22:19.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff082e2b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c1f4980 con 0x7ff07c06dd40 2024-05-08T21:22:19.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff07bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff07c0d0830 0x7ff07c09dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff07bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff07c0d0830 0x7ff07c09dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39992/0 (socket says 172.21.15.3:39992) 2024-05-08T21:22:19.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff07bfff700 1 -- 172.21.15.3:0/3769391686 learned_addr learned my addr 172.21.15.3:0/3769391686 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:19.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff080bc7700 1 --2- 172.21.15.3:0/3769391686 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff07c06dd40 0x7ff07c1f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff0813c8700 1 --2- 172.21.15.3:0/3769391686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff07c15a680 0x7ff07c14ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff07bfff700 1 -- 172.21.15.3:0/3769391686 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff07c06dd40 msgr2=0x7ff07c1f3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff07bfff700 1 --2- 172.21.15.3:0/3769391686 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff07c06dd40 0x7ff07c1f3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff07bfff700 1 -- 172.21.15.3:0/3769391686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff07c15a680 msgr2=0x7ff07c14ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff07bfff700 1 --2- 172.21.15.3:0/3769391686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff07c15a680 0x7ff07c14ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff07bfff700 1 -- 172.21.15.3:0/3769391686 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0680094a0 con 0x7ff07c0d0830 2024-05-08T21:22:19.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff080bc7700 1 --2- 172.21.15.3:0/3769391686 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff07c06dd40 0x7ff07c1f3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:19.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff07bfff700 1 --2- 172.21.15.3:0/3769391686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff07c0d0830 0x7ff07c09dbb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff0680090b0 tx=0x7ff0680047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:19.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.391+0000 7ff0797fa700 1 -- 172.21.15.3:0/3769391686 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06801bac0 con 0x7ff07c0d0830 2024-05-08T21:22:19.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.392+0000 7ff0797fa700 1 -- 172.21.15.3:0/3769391686 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06801bc20 con 0x7ff07c0d0830 2024-05-08T21:22:19.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.392+0000 7ff082e2b700 1 -- 172.21.15.3:0/3769391686 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07c1f40e0 con 0x7ff07c0d0830 2024-05-08T21:22:19.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.392+0000 7ff0797fa700 1 -- 172.21.15.3:0/3769391686 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0680253f0 con 0x7ff07c0d0830 2024-05-08T21:22:19.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.392+0000 7ff082e2b700 1 -- 172.21.15.3:0/3769391686 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07c1f46a0 con 0x7ff07c0d0830 2024-05-08T21:22:19.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.393+0000 7ff082e2b700 1 -- 172.21.15.3:0/3769391686 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07c158ba0 con 0x7ff07c0d0830 2024-05-08T21:22:19.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.393+0000 7ff0797fa700 1 -- 172.21.15.3:0/3769391686 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff068024390 con 0x7ff07c0d0830 2024-05-08T21:22:19.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.393+0000 7ff0797fa700 1 --2- 172.21.15.3:0/3769391686 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff06006ac90 0x7ff06006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.394+0000 7ff0797fa700 1 -- 172.21.15.3:0/3769391686 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff06808c3c0 con 0x7ff07c0d0830 2024-05-08T21:22:19.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.394+0000 7ff080bc7700 1 --2- 172.21.15.3:0/3769391686 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff06006ac90 0x7ff06006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.394+0000 7ff080bc7700 1 --2- 172.21.15.3:0/3769391686 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff06006ac90 0x7ff06006d150 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff070000f60 tx=0x7ff070009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:19.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.398+0000 7ff0797fa700 1 -- 172.21.15.3:0/3769391686 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff068057800 con 0x7ff07c0d0830 2024-05-08T21:22:19.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.621+0000 7ff082e2b700 1 -- 172.21.15.3:0/3769391686 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff07c068000 con 0x7ff07c0d0830 2024-05-08T21:22:19.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.623+0000 7ff0797fa700 1 -- 172.21.15.3:0/3769391686 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15989 (secure 0 0 0) 0x7ff068015090 con 0x7ff07c0d0830 2024-05-08T21:22:19.624 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:19.625 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:53.064342+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:19.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.625+0000 7ff082e2b700 1 -- 172.21.15.3:0/3769391686 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff06006ac90 msgr2=0x7ff06006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.625+0000 7ff082e2b700 1 --2- 172.21.15.3:0/3769391686 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff06006ac90 0x7ff06006d150 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff070000f60 tx=0x7ff070009040 comp rx=0 tx=0).stop 2024-05-08T21:22:19.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.625+0000 7ff082e2b700 1 -- 172.21.15.3:0/3769391686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff07c0d0830 msgr2=0x7ff07c09dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.625+0000 7ff082e2b700 1 --2- 172.21.15.3:0/3769391686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff07c0d0830 0x7ff07c09dbb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff0680090b0 tx=0x7ff0680047c0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.625+0000 7ff082e2b700 1 -- 172.21.15.3:0/3769391686 shutdown_connections 2024-05-08T21:22:19.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.625+0000 7ff082e2b700 1 --2- 172.21.15.3:0/3769391686 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff07c06dd40 0x7ff07c1f3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.625+0000 7ff082e2b700 1 --2- 172.21.15.3:0/3769391686 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff06006ac90 0x7ff06006d150 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.625+0000 7ff082e2b700 1 --2- 172.21.15.3:0/3769391686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff07c15a680 0x7ff07c14ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.625+0000 7ff082e2b700 1 --2- 172.21.15.3:0/3769391686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff07c0d0830 0x7ff07c09dbb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.625+0000 7ff082e2b700 1 -- 172.21.15.3:0/3769391686 >> 172.21.15.3:0/3769391686 conn(0x7ff07c0b5810 msgr2=0x7ff07c0b50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:19.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.625+0000 7ff082e2b700 1 -- 172.21.15.3:0/3769391686 shutdown_connections 2024-05-08T21:22:19.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.625+0000 7ff082e2b700 1 -- 172.21.15.3:0/3769391686 wait complete. 2024-05-08T21:22:19.641 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-08T21:21:49.638998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:21:51.024632+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:22:19.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:22:19.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.779+0000 7f3ac611a700 1 Processor -- start 2024-05-08T21:22:19.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.779+0000 7f3ac611a700 1 -- start start 2024-05-08T21:22:19.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.780+0000 7f3ac611a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3ac00cff30 0x7f3ac00d0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.780+0000 7f3ac611a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac00d0850 con 0x7f3ac00b4a90 2024-05-08T21:22:19.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.780+0000 7f3ac611a700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac00cf610 con 0x7f3ac00d3d20 2024-05-08T21:22:19.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.780+0000 7f3ac611a700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ac00d1fc0 con 0x7f3ac00cff30 2024-05-08T21:22:19.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.780+0000 7f3abffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3ac00b4a90 0x7f3ac00ce970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55288/0 (socket says 172.21.15.3:55288) 2024-05-08T21:22:19.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.780+0000 7f3abffff700 1 -- 172.21.15.3:0/115571938 learned_addr learned my addr 172.21.15.3:0/115571938 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:19.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.780+0000 7f3abf7fe700 1 --2- 172.21.15.3:0/115571938 >> v2:172.21.15.186:3300/0 conn(0x7f3ac00cff30 0x7f3ac00d0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.780+0000 7f3abe7fc700 1 -- 172.21.15.3:0/115571938 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371891775 0 0) 0x7f3ac00d0850 con 0x7f3ac00b4a90 2024-05-08T21:22:19.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.780+0000 7f3abe7fc700 1 -- 172.21.15.3:0/115571938 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aac0035b0 con 0x7f3ac00b4a90 2024-05-08T21:22:19.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.780+0000 7f3abe7fc700 1 -- 172.21.15.3:0/115571938 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 63122657 0 0) 0x7f3aac0035b0 con 0x7f3ac00b4a90 2024-05-08T21:22:19.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.780+0000 7f3abe7fc700 1 -- 172.21.15.3:0/115571938 >> v2:172.21.15.186:3300/0 conn(0x7f3ac00cff30 msgr2=0x7f3ac00d0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:19.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.780+0000 7f3abe7fc700 1 --2- 172.21.15.3:0/115571938 >> v2:172.21.15.186:3300/0 conn(0x7f3ac00cff30 0x7f3ac00d0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.780+0000 7f3abe7fc700 1 -- 172.21.15.3:0/115571938 >> v1:172.21.15.186:6789/0 conn(0x7f3ac00d3d20 legacy=0x7f3ac00ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.780+0000 7f3abe7fc700 1 -- 172.21.15.3:0/115571938 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac00d22a0 con 0x7f3ac00b4a90 2024-05-08T21:22:19.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.780+0000 7f3abe7fc700 1 -- 172.21.15.3:0/115571938 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f3ab80019f0 con 0x7f3ac00b4a90 2024-05-08T21:22:19.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abe7fc700 1 -- 172.21.15.3:0/115571938 >> v1:172.21.15.3:6789/0 conn(0x7f3ac00b4a90 legacy=0x7f3ac00ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abe7fc700 1 --2- 172.21.15.3:0/115571938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac003a00 0x7f3aac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abe7fc700 1 --2- 172.21.15.3:0/115571938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3aac004850 0x7f3aac004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abe7fc700 1 --2- 172.21.15.3:0/115571938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac005210 0x7f3aac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abe7fc700 1 -- 172.21.15.3:0/115571938 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac00d1fc0 con 0x7f3aac004850 2024-05-08T21:22:19.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abe7fc700 1 -- 172.21.15.3:0/115571938 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac00c0b0 con 0x7f3aac003a00 2024-05-08T21:22:19.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abe7fc700 1 -- 172.21.15.3:0/115571938 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac00c270 con 0x7f3aac005210 2024-05-08T21:22:19.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abeffd700 1 --2- 172.21.15.3:0/115571938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3aac004850 0x7f3aac004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abf7fe700 1 --2- 172.21.15.3:0/115571938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac003a00 0x7f3aac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abffff700 1 --2- 172.21.15.3:0/115571938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac005210 0x7f3aac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abeffd700 1 -- 172.21.15.3:0/115571938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac005210 msgr2=0x7f3aac00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:19.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abeffd700 1 --2- 172.21.15.3:0/115571938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac005210 0x7f3aac00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abeffd700 1 -- 172.21.15.3:0/115571938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac003a00 msgr2=0x7f3aac003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:19.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abeffd700 1 --2- 172.21.15.3:0/115571938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac003a00 0x7f3aac003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abeffd700 1 -- 172.21.15.3:0/115571938 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac0035b0 con 0x7f3aac004850 2024-05-08T21:22:19.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abffff700 1 -- 172.21.15.3:0/115571938 reap_dead start 2024-05-08T21:22:19.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abeffd700 1 --2- 172.21.15.3:0/115571938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3aac004850 0x7f3aac004cd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80017c0 tx=0x7f3aac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af4a9fa35b4590c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:19.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abe7fc700 1 -- 172.21.15.3:0/115571938 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa80052d0 con 0x7f3aac004850 2024-05-08T21:22:19.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abe7fc700 1 -- 172.21.15.3:0/115571938 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa8005430 con 0x7f3aac004850 2024-05-08T21:22:19.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.781+0000 7f3abe7fc700 1 -- 172.21.15.3:0/115571938 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa80038c0 con 0x7f3aac004850 2024-05-08T21:22:19.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.782+0000 7f3ac611a700 1 -- 172.21.15.3:0/115571938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3aac004850 msgr2=0x7f3aac004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.782+0000 7f3ac611a700 1 --2- 172.21.15.3:0/115571938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3aac004850 0x7f3aac004cd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80017c0 tx=0x7f3aac005880 comp rx=0 tx=0).stop 2024-05-08T21:22:19.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.782+0000 7f3ac611a700 1 -- 172.21.15.3:0/115571938 shutdown_connections 2024-05-08T21:22:19.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.782+0000 7f3ac611a700 1 --2- 172.21.15.3:0/115571938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3aac004850 0x7f3aac004cd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.782+0000 7f3ac611a700 1 -- 172.21.15.3:0/115571938 >> 172.21.15.3:0/115571938 conn(0x7f3ac008afe0 msgr2=0x7f3ac00d3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:19.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.782+0000 7f3ac611a700 1 -- 172.21.15.3:0/115571938 shutdown_connections 2024-05-08T21:22:19.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.782+0000 7f3ac611a700 1 -- 172.21.15.3:0/115571938 wait complete. 2024-05-08T21:22:19.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.783+0000 7f3ac611a700 1 Processor -- start 2024-05-08T21:22:19.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.784+0000 7f3ac611a700 1 -- start start 2024-05-08T21:22:19.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.784+0000 7f3ac611a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac004850 0x7f3ac00b6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.784+0000 7f3ac611a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ac009efd0 0x7f3ac009d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.784+0000 7f3ac611a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ac009d910 0x7f3ac00a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.784+0000 7f3ac611a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac01510d0 con 0x7f3ac009efd0 2024-05-08T21:22:19.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.784+0000 7f3ac611a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac009f560 con 0x7f3ac009d910 2024-05-08T21:22:19.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.784+0000 7f3ac611a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac009f840 con 0x7f3aac004850 2024-05-08T21:22:19.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.784+0000 7f3abeffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ac009efd0 0x7f3ac009d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.784+0000 7f3abeffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ac009efd0 0x7f3ac009d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40018/0 (socket says 172.21.15.3:40018) 2024-05-08T21:22:19.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.784+0000 7f3abeffd700 1 -- 172.21.15.3:0/1869856735 learned_addr learned my addr 172.21.15.3:0/1869856735 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:19.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.785+0000 7f3abf7fe700 1 --2- 172.21.15.3:0/1869856735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac004850 0x7f3ac00b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.785+0000 7f3abffff700 1 --2- 172.21.15.3:0/1869856735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ac009d910 0x7f3ac00a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.785+0000 7f3abeffd700 1 -- 172.21.15.3:0/1869856735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac004850 msgr2=0x7f3ac00b6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.785+0000 7f3abeffd700 1 --2- 172.21.15.3:0/1869856735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac004850 0x7f3ac00b6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.785+0000 7f3abeffd700 1 -- 172.21.15.3:0/1869856735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ac009d910 msgr2=0x7f3ac00a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:19.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.785+0000 7f3abeffd700 1 --2- 172.21.15.3:0/1869856735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ac009d910 0x7f3ac00a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:19.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.785+0000 7f3abeffd700 1 -- 172.21.15.3:0/1869856735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac009fb20 con 0x7f3ac009efd0 2024-05-08T21:22:19.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.785+0000 7f3abeffd700 1 --2- 172.21.15.3:0/1869856735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ac009efd0 0x7f3ac009d3d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8005040 tx=0x7f3aa8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:19.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.785+0000 7f3ab7fff700 1 -- 172.21.15.3:0/1869856735 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8001290 con 0x7f3ac009efd0 2024-05-08T21:22:19.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.785+0000 7f3ab7fff700 1 -- 172.21.15.3:0/1869856735 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa8007040 con 0x7f3ac009efd0 2024-05-08T21:22:19.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.785+0000 7f3ac611a700 1 -- 172.21.15.3:0/1869856735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac009fe00 con 0x7f3ac009efd0 2024-05-08T21:22:19.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.785+0000 7f3ab7fff700 1 -- 172.21.15.3:0/1869856735 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa80073b0 con 0x7f3ac009efd0 2024-05-08T21:22:19.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.785+0000 7f3ac611a700 1 -- 172.21.15.3:0/1869856735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac00b6da0 con 0x7f3ac009efd0 2024-05-08T21:22:19.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.786+0000 7f3ac611a700 1 -- 172.21.15.3:0/1869856735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac0074bc0 con 0x7f3ac009efd0 2024-05-08T21:22:19.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.787+0000 7f3ab7fff700 1 -- 172.21.15.3:0/1869856735 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3aa8007510 con 0x7f3ac009efd0 2024-05-08T21:22:19.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.787+0000 7f3ab7fff700 1 --2- 172.21.15.3:0/1869856735 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3aa006ac90 0x7f3aa006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:19.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.787+0000 7f3ab7fff700 1 -- 172.21.15.3:0/1869856735 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3aa8089d10 con 0x7f3ac009efd0 2024-05-08T21:22:19.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.787+0000 7f3abf7fe700 1 --2- 172.21.15.3:0/1869856735 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3aa006ac90 0x7f3aa006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:19.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.788+0000 7f3abf7fe700 1 --2- 172.21.15.3:0/1869856735 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3aa006ac90 0x7f3aa006d150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0069190 tx=0x7f3ab0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:19.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:19.792+0000 7f3ab7fff700 1 -- 172.21.15.3:0/1869856735 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f3aa80550a0 con 0x7f3ac009efd0 2024-05-08T21:22:20.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.008+0000 7f3ac611a700 1 -- 172.21.15.3:0/1869856735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3ac00d4520 con 0x7f3ac009efd0 2024-05-08T21:22:20.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.008+0000 7f3ab7fff700 1 -- 172.21.15.3:0/1869856735 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3aa80170a0 con 0x7f3ac009efd0 2024-05-08T21:22:20.010 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:22:20.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.010+0000 7f3ac611a700 1 -- 172.21.15.3:0/1869856735 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3aa006ac90 msgr2=0x7f3aa006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.010+0000 7f3ac611a700 1 --2- 172.21.15.3:0/1869856735 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3aa006ac90 0x7f3aa006d150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0069190 tx=0x7f3ab0008040 comp rx=0 tx=0).stop 2024-05-08T21:22:20.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.010+0000 7f3ac611a700 1 -- 172.21.15.3:0/1869856735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ac009efd0 msgr2=0x7f3ac009d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.010+0000 7f3ac611a700 1 --2- 172.21.15.3:0/1869856735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ac009efd0 0x7f3ac009d3d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8005040 tx=0x7f3aa8001120 comp rx=0 tx=0).stop 2024-05-08T21:22:20.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.010+0000 7f3ac611a700 1 -- 172.21.15.3:0/1869856735 shutdown_connections 2024-05-08T21:22:20.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.010+0000 7f3ac611a700 1 --2- 172.21.15.3:0/1869856735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac004850 0x7f3ac00b6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.010+0000 7f3ac611a700 1 --2- 172.21.15.3:0/1869856735 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3aa006ac90 0x7f3aa006d150 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.010+0000 7f3ac611a700 1 --2- 172.21.15.3:0/1869856735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ac009d910 0x7f3ac00a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.010+0000 7f3ac611a700 1 --2- 172.21.15.3:0/1869856735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ac009efd0 0x7f3ac009d3d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.010+0000 7f3ac611a700 1 -- 172.21.15.3:0/1869856735 >> 172.21.15.3:0/1869856735 conn(0x7f3ac008afe0 msgr2=0x7f3ac0160410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:20.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.010+0000 7f3ac611a700 1 -- 172.21.15.3:0/1869856735 shutdown_connections 2024-05-08T21:22:20.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.010+0000 7f3ac611a700 1 -- 172.21.15.3:0/1869856735 wait complete. 2024-05-08T21:22:20.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-08T21:22:20.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.163+0000 7f23d8564700 1 Processor -- start 2024-05-08T21:22:20.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.163+0000 7f23d8564700 1 -- start start 2024-05-08T21:22:20.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d8564700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23d00d13a0 0x7f23d00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d8564700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f23d006bd60 0x7f23d00cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d8564700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f23d00cdc90 0x7f23d00b19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d8564700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d00b1200 con 0x7f23d00cdc90 2024-05-08T21:22:20.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d8564700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d00d1cc0 con 0x7f23d006bd60 2024-05-08T21:22:20.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d8564700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d00cfcc0 con 0x7f23d00d13a0 2024-05-08T21:22:20.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d6b01700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f23d00cdc90 0x7f23d00b19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d6b01700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f23d00cdc90 0x7f23d00b19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40032/0 (socket says 172.21.15.3:40032) 2024-05-08T21:22:20.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d6b01700 1 -- 172.21.15.3:0/3637431781 learned_addr learned my addr 172.21.15.3:0/3637431781 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:20.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d5aff700 1 --2- 172.21.15.3:0/3637431781 >> v2:172.21.15.186:3300/0 conn(0x7f23d006bd60 0x7f23d00cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d6300700 1 --2- 172.21.15.3:0/3637431781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23d00d13a0 0x7f23d00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d6b01700 1 -- 172.21.15.3:0/3637431781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23d00d13a0 msgr2=0x7f23d00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:20.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d6b01700 1 --2- 172.21.15.3:0/3637431781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23d00d13a0 0x7f23d00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d6b01700 1 -- 172.21.15.3:0/3637431781 >> v2:172.21.15.186:3300/0 conn(0x7f23d006bd60 msgr2=0x7f23d00cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:20.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d6b01700 1 --2- 172.21.15.3:0/3637431781 >> v2:172.21.15.186:3300/0 conn(0x7f23d006bd60 0x7f23d00cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d6b01700 1 -- 172.21.15.3:0/3637431781 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d00ce5c0 con 0x7f23d00cdc90 2024-05-08T21:22:20.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.164+0000 7f23d6b01700 1 --2- 172.21.15.3:0/3637431781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23d00cdc90 0x7f23d00b19c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f23d00b3f30 tx=0x7f23c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc9da7e5b2b49b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:20.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.165+0000 7f23d52fe700 1 -- 172.21.15.3:0/3637431781 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c800e070 con 0x7f23d00cdc90 2024-05-08T21:22:20.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.165+0000 7f23d52fe700 1 -- 172.21.15.3:0/3637431781 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c800b9c0 con 0x7f23d00cdc90 2024-05-08T21:22:20.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.165+0000 7f23d52fe700 1 -- 172.21.15.3:0/3637431781 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c800bd70 con 0x7f23d00cdc90 2024-05-08T21:22:20.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.165+0000 7f23d8564700 1 -- 172.21.15.3:0/3637431781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23d00cdc90 msgr2=0x7f23d00b19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.165+0000 7f23d8564700 1 --2- 172.21.15.3:0/3637431781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23d00cdc90 0x7f23d00b19c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f23d00b3f30 tx=0x7f23c800b330 comp rx=0 tx=0).stop 2024-05-08T21:22:20.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.165+0000 7f23d8564700 1 -- 172.21.15.3:0/3637431781 shutdown_connections 2024-05-08T21:22:20.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.165+0000 7f23d8564700 1 --2- 172.21.15.3:0/3637431781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23d00cdc90 0x7f23d00b19c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.165+0000 7f23d8564700 1 --2- 172.21.15.3:0/3637431781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23d00d13a0 0x7f23d00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.165+0000 7f23d8564700 1 --2- 172.21.15.3:0/3637431781 >> v2:172.21.15.186:3300/0 conn(0x7f23d006bd60 0x7f23d00cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.165+0000 7f23d8564700 1 -- 172.21.15.3:0/3637431781 >> 172.21.15.3:0/3637431781 conn(0x7f23d008afe0 msgr2=0x7f23d008b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:20.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.165+0000 7f23d8564700 1 -- 172.21.15.3:0/3637431781 shutdown_connections 2024-05-08T21:22:20.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.165+0000 7f23d8564700 1 -- 172.21.15.3:0/3637431781 wait complete. 2024-05-08T21:22:20.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.167+0000 7f23d8564700 1 Processor -- start 2024-05-08T21:22:20.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.167+0000 7f23d8564700 1 -- start start 2024-05-08T21:22:20.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.167+0000 7f23d8564700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23d006bd60 0x7f23d014f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.167+0000 7f23d8564700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23d00cdc90 0x7f23d015f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.167+0000 7f23d8564700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23d00d13a0 0x7f23d00cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.167+0000 7f23d8564700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d00c84e0 con 0x7f23d00cdc90 2024-05-08T21:22:20.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.167+0000 7f23d8564700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d014ff30 con 0x7f23d006bd60 2024-05-08T21:22:20.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.167+0000 7f23d8564700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d00c8730 con 0x7f23d00d13a0 2024-05-08T21:22:20.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.168+0000 7f23d5aff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23d00cdc90 0x7f23d015f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.168+0000 7f23d5aff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23d00cdc90 0x7f23d015f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40046/0 (socket says 172.21.15.3:40046) 2024-05-08T21:22:20.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.168+0000 7f23d6300700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23d006bd60 0x7f23d014f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.168+0000 7f23d5aff700 1 -- 172.21.15.3:0/1241551739 learned_addr learned my addr 172.21.15.3:0/1241551739 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:20.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.168+0000 7f23d6b01700 1 --2- 172.21.15.3:0/1241551739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23d00d13a0 0x7f23d00cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.168+0000 7f23d5aff700 1 -- 172.21.15.3:0/1241551739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23d00d13a0 msgr2=0x7f23d00cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.168+0000 7f23d5aff700 1 --2- 172.21.15.3:0/1241551739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23d00d13a0 0x7f23d00cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.168+0000 7f23d5aff700 1 -- 172.21.15.3:0/1241551739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23d006bd60 msgr2=0x7f23d014f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.168+0000 7f23d5aff700 1 --2- 172.21.15.3:0/1241551739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23d006bd60 0x7f23d014f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.168+0000 7f23d5aff700 1 -- 172.21.15.3:0/1241551739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c8009b30 con 0x7f23d00cdc90 2024-05-08T21:22:20.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.168+0000 7f23d5aff700 1 --2- 172.21.15.3:0/1241551739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23d00cdc90 0x7f23d015f940 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f23c000d800 tx=0x7f23c000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:20.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.168+0000 7f23c6ffd700 1 -- 172.21.15.3:0/1241551739 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c00125f0 con 0x7f23d00cdc90 2024-05-08T21:22:20.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.168+0000 7f23c6ffd700 1 -- 172.21.15.3:0/1241551739 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c0012750 con 0x7f23d00cdc90 2024-05-08T21:22:20.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.168+0000 7f23d8564700 1 -- 172.21.15.3:0/1241551739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d00c8a10 con 0x7f23d00cdc90 2024-05-08T21:22:20.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.168+0000 7f23c6ffd700 1 -- 172.21.15.3:0/1241551739 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c0012b00 con 0x7f23d00cdc90 2024-05-08T21:22:20.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.168+0000 7f23d8564700 1 -- 172.21.15.3:0/1241551739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d00ca700 con 0x7f23d00cdc90 2024-05-08T21:22:20.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.169+0000 7f23d8564700 1 -- 172.21.15.3:0/1241551739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d015e220 con 0x7f23d00cdc90 2024-05-08T21:22:20.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.170+0000 7f23c6ffd700 1 -- 172.21.15.3:0/1241551739 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f23c0012c60 con 0x7f23d00cdc90 2024-05-08T21:22:20.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.170+0000 7f23c6ffd700 1 --2- 172.21.15.3:0/1241551739 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f23b406ac90 0x7f23b406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.170+0000 7f23c6ffd700 1 -- 172.21.15.3:0/1241551739 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23c0089420 con 0x7f23d00cdc90 2024-05-08T21:22:20.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.170+0000 7f23d6300700 1 --2- 172.21.15.3:0/1241551739 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f23b406ac90 0x7f23b406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.171+0000 7f23d6300700 1 --2- 172.21.15.3:0/1241551739 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f23b406ac90 0x7f23b406d150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f23cc006b20 tx=0x7f23cc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:20.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.175+0000 7f23c6ffd700 1 -- 172.21.15.3:0/1241551739 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f23c0054570 con 0x7f23d00cdc90 2024-05-08T21:22:20.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.393+0000 7f23d8564700 1 -- 172.21.15.3:0/1241551739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f23d0074bc0 con 0x7f23d00cdc90 2024-05-08T21:22:20.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.393+0000 7f23c6ffd700 1 -- 172.21.15.3:0/1241551739 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f23c0057b90 con 0x7f23d00cdc90 2024-05-08T21:22:20.395 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:22:20.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.395+0000 7f23d8564700 1 -- 172.21.15.3:0/1241551739 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f23b406ac90 msgr2=0x7f23b406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.395+0000 7f23d8564700 1 --2- 172.21.15.3:0/1241551739 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f23b406ac90 0x7f23b406d150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f23cc006b20 tx=0x7f23cc008040 comp rx=0 tx=0).stop 2024-05-08T21:22:20.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.395+0000 7f23d8564700 1 -- 172.21.15.3:0/1241551739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23d00cdc90 msgr2=0x7f23d015f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.395+0000 7f23d8564700 1 --2- 172.21.15.3:0/1241551739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23d00cdc90 0x7f23d015f940 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f23c000d800 tx=0x7f23c000db10 comp rx=0 tx=0).stop 2024-05-08T21:22:20.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.396+0000 7f23d8564700 1 -- 172.21.15.3:0/1241551739 shutdown_connections 2024-05-08T21:22:20.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.396+0000 7f23d8564700 1 --2- 172.21.15.3:0/1241551739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23d00d13a0 0x7f23d00cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.396+0000 7f23d8564700 1 --2- 172.21.15.3:0/1241551739 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f23b406ac90 0x7f23b406d150 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.396+0000 7f23d8564700 1 --2- 172.21.15.3:0/1241551739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23d006bd60 0x7f23d014f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.396+0000 7f23d8564700 1 --2- 172.21.15.3:0/1241551739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23d00cdc90 0x7f23d015f940 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.396+0000 7f23d8564700 1 -- 172.21.15.3:0/1241551739 >> 172.21.15.3:0/1241551739 conn(0x7f23d008afe0 msgr2=0x7f23d00d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:20.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.396+0000 7f23d8564700 1 -- 172.21.15.3:0/1241551739 shutdown_connections 2024-05-08T21:22:20.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.396+0000 7f23d8564700 1 -- 172.21.15.3:0/1241551739 wait complete. 2024-05-08T21:22:20.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:22:20.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.549+0000 7ffb544f1700 1 Processor -- start 2024-05-08T21:22:20.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.549+0000 7ffb544f1700 1 -- start start 2024-05-08T21:22:20.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.550+0000 7ffb544f1700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ffb4c072170 0x7ffb4c0ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.550+0000 7ffb544f1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb4c0ae620 0x7ffb4c06e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.550+0000 7ffb544f1700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c0aed90 con 0x7ffb4c15dc60 2024-05-08T21:22:20.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.550+0000 7ffb544f1700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb4c06dcd0 con 0x7ffb4c072170 2024-05-08T21:22:20.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.550+0000 7ffb544f1700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c06e290 con 0x7ffb4c0ae620 2024-05-08T21:22:20.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.550+0000 7ffb51a8c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ffb4c072170 0x7ffb4c0ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.550+0000 7ffb51a8c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ffb4c072170 0x7ffb4c0ae0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40052/0 (socket says 172.21.15.3:40052) 2024-05-08T21:22:20.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.550+0000 7ffb51a8c700 1 -- 172.21.15.3:0/1525023449 learned_addr learned my addr 172.21.15.3:0/1525023449 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:20.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.550+0000 7ffb52a8e700 1 --2- 172.21.15.3:0/1525023449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb4c0ae620 0x7ffb4c06e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.550+0000 7ffb51a8c700 1 -- 172.21.15.3:0/1525023449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb4c0ae620 msgr2=0x7ffb4c06e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.550+0000 7ffb51a8c700 1 --2- 172.21.15.3:0/1525023449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb4c0ae620 0x7ffb4c06e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.550+0000 7ffb51a8c700 1 -- 172.21.15.3:0/1525023449 >> v1:172.21.15.186:6789/0 conn(0x7ffb4c15dc60 legacy=0x7ffb4c0ad9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.550+0000 7ffb51a8c700 1 -- 172.21.15.3:0/1525023449 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c0b2aa0 con 0x7ffb4c072170 2024-05-08T21:22:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.550+0000 7ffb51a8c700 1 --2- 172.21.15.3:0/1525023449 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb4c072170 0x7ffb4c0ae0e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c00bb90 tx=0x7ffb3c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1b3847d9ab337d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.551+0000 7ffb5128b700 1 -- 172.21.15.3:0/1525023449 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c00dd30 con 0x7ffb4c072170 2024-05-08T21:22:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.551+0000 7ffb5128b700 1 -- 172.21.15.3:0/1525023449 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3c00de90 con 0x7ffb4c072170 2024-05-08T21:22:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.551+0000 7ffb5128b700 1 -- 172.21.15.3:0/1525023449 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c010690 con 0x7ffb4c072170 2024-05-08T21:22:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.551+0000 7ffb544f1700 1 -- 172.21.15.3:0/1525023449 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb4c072170 msgr2=0x7ffb4c0ae0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.551+0000 7ffb544f1700 1 --2- 172.21.15.3:0/1525023449 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb4c072170 0x7ffb4c0ae0e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c00bb90 tx=0x7ffb3c00bea0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.551+0000 7ffb544f1700 1 -- 172.21.15.3:0/1525023449 shutdown_connections 2024-05-08T21:22:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.551+0000 7ffb544f1700 1 --2- 172.21.15.3:0/1525023449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb4c0ae620 0x7ffb4c06e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.551+0000 7ffb544f1700 1 --2- 172.21.15.3:0/1525023449 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb4c072170 0x7ffb4c0ae0e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.551+0000 7ffb544f1700 1 -- 172.21.15.3:0/1525023449 >> 172.21.15.3:0/1525023449 conn(0x7ffb4c0b5a30 msgr2=0x7ffb4c0b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.551+0000 7ffb544f1700 1 -- 172.21.15.3:0/1525023449 shutdown_connections 2024-05-08T21:22:20.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.551+0000 7ffb544f1700 1 -- 172.21.15.3:0/1525023449 wait complete. 2024-05-08T21:22:20.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.553+0000 7ffb544f1700 1 Processor -- start 2024-05-08T21:22:20.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.553+0000 7ffb544f1700 1 -- start start 2024-05-08T21:22:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.554+0000 7ffb544f1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb4c072170 0x7ffb4c1f2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.554+0000 7ffb544f1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb4c0ae620 0x7ffb4c141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.554+0000 7ffb544f1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb4c15dc60 0x7ffb4c156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.554+0000 7ffb544f1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c1423e0 con 0x7ffb4c15dc60 2024-05-08T21:22:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.554+0000 7ffb544f1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c1ef110 con 0x7ffb4c0ae620 2024-05-08T21:22:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.554+0000 7ffb544f1700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c1f33e0 con 0x7ffb4c072170 2024-05-08T21:22:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.554+0000 7ffb52a8e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb4c15dc60 0x7ffb4c156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.554+0000 7ffb52a8e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb4c15dc60 0x7ffb4c156650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40056/0 (socket says 172.21.15.3:40056) 2024-05-08T21:22:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.554+0000 7ffb52a8e700 1 -- 172.21.15.3:0/2105763632 learned_addr learned my addr 172.21.15.3:0/2105763632 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.555+0000 7ffb51a8c700 1 --2- 172.21.15.3:0/2105763632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb4c0ae620 0x7ffb4c141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.555+0000 7ffb5228d700 1 --2- 172.21.15.3:0/2105763632 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb4c072170 0x7ffb4c1f2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.555+0000 7ffb52a8e700 1 -- 172.21.15.3:0/2105763632 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb4c072170 msgr2=0x7ffb4c1f2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.555+0000 7ffb52a8e700 1 --2- 172.21.15.3:0/2105763632 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb4c072170 0x7ffb4c1f2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.555+0000 7ffb52a8e700 1 -- 172.21.15.3:0/2105763632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb4c0ae620 msgr2=0x7ffb4c141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.555+0000 7ffb52a8e700 1 --2- 172.21.15.3:0/2105763632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb4c0ae620 0x7ffb4c141ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.555+0000 7ffb52a8e700 1 -- 172.21.15.3:0/2105763632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3c00b7f0 con 0x7ffb4c15dc60 2024-05-08T21:22:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.555+0000 7ffb52a8e700 1 --2- 172.21.15.3:0/2105763632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb4c15dc60 0x7ffb4c156650 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffb44009dd0 tx=0x7ffb4400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.555+0000 7ffb42ffd700 1 -- 172.21.15.3:0/2105763632 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4400cd40 con 0x7ffb4c15dc60 2024-05-08T21:22:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.555+0000 7ffb42ffd700 1 -- 172.21.15.3:0/2105763632 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4400f040 con 0x7ffb4c15dc60 2024-05-08T21:22:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.555+0000 7ffb544f1700 1 -- 172.21.15.3:0/2105763632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4c1f2b40 con 0x7ffb4c15dc60 2024-05-08T21:22:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.555+0000 7ffb42ffd700 1 -- 172.21.15.3:0/2105763632 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb440146c0 con 0x7ffb4c15dc60 2024-05-08T21:22:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.555+0000 7ffb544f1700 1 -- 172.21.15.3:0/2105763632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4c1f3100 con 0x7ffb4c15dc60 2024-05-08T21:22:20.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.556+0000 7ffb544f1700 1 -- 172.21.15.3:0/2105763632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb4c068410 con 0x7ffb4c15dc60 2024-05-08T21:22:20.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.557+0000 7ffb42ffd700 1 -- 172.21.15.3:0/2105763632 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ffb44014820 con 0x7ffb4c15dc60 2024-05-08T21:22:20.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.557+0000 7ffb42ffd700 1 --2- 172.21.15.3:0/2105763632 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ffb3006ac90 0x7ffb3006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.557+0000 7ffb42ffd700 1 -- 172.21.15.3:0/2105763632 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb44089ce0 con 0x7ffb4c15dc60 2024-05-08T21:22:20.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.557+0000 7ffb5228d700 1 --2- 172.21.15.3:0/2105763632 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ffb3006ac90 0x7ffb3006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.558+0000 7ffb5228d700 1 --2- 172.21.15.3:0/2105763632 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ffb3006ac90 0x7ffb3006d150 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ffb48000f60 tx=0x7ffb48009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.562+0000 7ffb42ffd700 1 -- 172.21.15.3:0/2105763632 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ffb44054ec0 con 0x7ffb4c15dc60 2024-05-08T21:22:20.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.780+0000 7ffb544f1700 1 -- 172.21.15.3:0/2105763632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffb4c1eef20 con 0x7ffb4c15dc60 2024-05-08T21:22:20.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.781+0000 7ffb42ffd700 1 -- 172.21.15.3:0/2105763632 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffb44018020 con 0x7ffb4c15dc60 2024-05-08T21:22:20.783 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:22:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.783+0000 7ffb544f1700 1 -- 172.21.15.3:0/2105763632 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ffb3006ac90 msgr2=0x7ffb3006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.783+0000 7ffb544f1700 1 --2- 172.21.15.3:0/2105763632 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ffb3006ac90 0x7ffb3006d150 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ffb48000f60 tx=0x7ffb48009040 comp rx=0 tx=0).stop 2024-05-08T21:22:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.783+0000 7ffb544f1700 1 -- 172.21.15.3:0/2105763632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb4c15dc60 msgr2=0x7ffb4c156650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.783+0000 7ffb544f1700 1 --2- 172.21.15.3:0/2105763632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb4c15dc60 0x7ffb4c156650 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffb44009dd0 tx=0x7ffb4400c3a0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.783+0000 7ffb544f1700 1 -- 172.21.15.3:0/2105763632 shutdown_connections 2024-05-08T21:22:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.783+0000 7ffb544f1700 1 --2- 172.21.15.3:0/2105763632 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb4c072170 0x7ffb4c1f2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.783+0000 7ffb544f1700 1 --2- 172.21.15.3:0/2105763632 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ffb3006ac90 0x7ffb3006d150 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.783+0000 7ffb544f1700 1 --2- 172.21.15.3:0/2105763632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb4c15dc60 0x7ffb4c156650 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.783+0000 7ffb544f1700 1 --2- 172.21.15.3:0/2105763632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb4c0ae620 0x7ffb4c141ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.783+0000 7ffb544f1700 1 -- 172.21.15.3:0/2105763632 >> 172.21.15.3:0/2105763632 conn(0x7ffb4c0b5a30 msgr2=0x7ffb4c15bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.783+0000 7ffb544f1700 1 -- 172.21.15.3:0/2105763632 shutdown_connections 2024-05-08T21:22:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.783+0000 7ffb544f1700 1 -- 172.21.15.3:0/2105763632 wait complete. 2024-05-08T21:22:20.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:20.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.933+0000 7fa55b10d700 1 Processor -- start 2024-05-08T21:22:20.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.933+0000 7fa55b10d700 1 -- start start 2024-05-08T21:22:20.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.934+0000 7fa55b10d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa55406bd80 0x7fa5540d0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.934+0000 7fa55b10d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa55406c3d0 0x7fa55406c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.934+0000 7fa55b10d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5540cf990 con 0x7fa55415da10 2024-05-08T21:22:20.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.934+0000 7fa55b10d700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa55409ab40 con 0x7fa55406c3d0 2024-05-08T21:22:20.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.934+0000 7fa55b10d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa55406cda0 con 0x7fa55406bd80 2024-05-08T21:22:20.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.934+0000 7fa558ea9700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa55415da10 0x7fa55409a430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55292/0 (socket says 172.21.15.3:55292) 2024-05-08T21:22:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.934+0000 7fa558ea9700 1 -- 172.21.15.3:0/2304861008 learned_addr learned my addr 172.21.15.3:0/2304861008 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.934+0000 7fa5596aa700 1 --2- 172.21.15.3:0/2304861008 >> v2:172.21.15.186:3300/0 conn(0x7fa55406c3d0 0x7fa55406c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.934+0000 7fa553fff700 1 --2- 172.21.15.3:0/2304861008 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa55406bd80 0x7fa5540d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.934+0000 7fa5537fe700 1 -- 172.21.15.3:0/2304861008 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483703452 0 0) 0x7fa5540cf990 con 0x7fa55415da10 2024-05-08T21:22:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.934+0000 7fa5537fe700 1 -- 172.21.15.3:0/2304861008 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5440035b0 con 0x7fa55415da10 2024-05-08T21:22:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa5537fe700 1 -- 172.21.15.3:0/2304861008 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1492989919 0 0) 0x7fa5440035b0 con 0x7fa55415da10 2024-05-08T21:22:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa5537fe700 1 -- 172.21.15.3:0/2304861008 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa55406bd80 msgr2=0x7fa5540d0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa5537fe700 1 --2- 172.21.15.3:0/2304861008 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa55406bd80 0x7fa5540d0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa5537fe700 1 -- 172.21.15.3:0/2304861008 >> v2:172.21.15.186:3300/0 conn(0x7fa55406c3d0 msgr2=0x7fa55406c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa5537fe700 1 --2- 172.21.15.3:0/2304861008 >> v2:172.21.15.186:3300/0 conn(0x7fa55406c3d0 0x7fa55406c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa5537fe700 1 -- 172.21.15.3:0/2304861008 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5540d04b0 con 0x7fa55415da10 2024-05-08T21:22:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa5537fe700 1 -- 172.21.15.3:0/2304861008 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7fa548001670 con 0x7fa55415da10 2024-05-08T21:22:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa5537fe700 1 -- 172.21.15.3:0/2304861008 >> v1:172.21.15.3:6789/0 conn(0x7fa55415da10 legacy=0x7fa55409a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa5537fe700 1 --2- 172.21.15.3:0/2304861008 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa55406bd80 0x7fa544003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa5537fe700 1 --2- 172.21.15.3:0/2304861008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544003fb0 0x7fa544004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa5537fe700 1 --2- 172.21.15.3:0/2304861008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa544004d90 0x7fa544005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa5537fe700 1 -- 172.21.15.3:0/2304861008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55409ab40 con 0x7fa544003fb0 2024-05-08T21:22:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa5537fe700 1 -- 172.21.15.3:0/2304861008 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55406cda0 con 0x7fa544004d90 2024-05-08T21:22:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa5537fe700 1 -- 172.21.15.3:0/2304861008 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54400ba10 con 0x7fa55406bd80 2024-05-08T21:22:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa553fff700 1 --2- 172.21.15.3:0/2304861008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544003fb0 0x7fa544004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa558ea9700 1 --2- 172.21.15.3:0/2304861008 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa55406bd80 0x7fa544003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa5596aa700 1 --2- 172.21.15.3:0/2304861008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa544004d90 0x7fa544005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa553fff700 1 -- 172.21.15.3:0/2304861008 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa55406bd80 msgr2=0x7fa544003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa553fff700 1 --2- 172.21.15.3:0/2304861008 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa55406bd80 0x7fa544003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa553fff700 1 -- 172.21.15.3:0/2304861008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa544004d90 msgr2=0x7fa544005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa553fff700 1 --2- 172.21.15.3:0/2304861008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa544004d90 0x7fa544005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa553fff700 1 -- 172.21.15.3:0/2304861008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5440035b0 con 0x7fa544003fb0 2024-05-08T21:22:20.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.935+0000 7fa553fff700 1 --2- 172.21.15.3:0/2304861008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544003fb0 0x7fa544004850 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa540009b30 tx=0x7fa544005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0303e3b5dcdd5bd server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:20.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.936+0000 7fa5537fe700 1 -- 172.21.15.3:0/2304861008 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54000e6c0 con 0x7fa544003fb0 2024-05-08T21:22:20.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.936+0000 7fa5537fe700 1 -- 172.21.15.3:0/2304861008 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54000e820 con 0x7fa544003fb0 2024-05-08T21:22:20.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.936+0000 7fa5537fe700 1 -- 172.21.15.3:0/2304861008 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54000eb90 con 0x7fa544003fb0 2024-05-08T21:22:20.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.936+0000 7fa55b10d700 1 -- 172.21.15.3:0/2304861008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544003fb0 msgr2=0x7fa544004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.936+0000 7fa55b10d700 1 --2- 172.21.15.3:0/2304861008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544003fb0 0x7fa544004850 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa540009b30 tx=0x7fa544005880 comp rx=0 tx=0).stop 2024-05-08T21:22:20.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.936+0000 7fa5596aa700 1 -- 172.21.15.3:0/2304861008 reap_dead start 2024-05-08T21:22:20.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.936+0000 7fa55b10d700 1 -- 172.21.15.3:0/2304861008 shutdown_connections 2024-05-08T21:22:20.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.936+0000 7fa55b10d700 1 -- 172.21.15.3:0/2304861008 >> 172.21.15.3:0/2304861008 conn(0x7fa5540b3010 msgr2=0x7fa5540b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:20.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.936+0000 7fa55b10d700 1 -- 172.21.15.3:0/2304861008 shutdown_connections 2024-05-08T21:22:20.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.936+0000 7fa55b10d700 1 -- 172.21.15.3:0/2304861008 wait complete. 2024-05-08T21:22:20.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.937+0000 7fa55b10d700 1 Processor -- start 2024-05-08T21:22:20.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.938+0000 7fa55b10d700 1 -- start start 2024-05-08T21:22:20.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.938+0000 7fa55b10d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544003fb0 0x7fa5540b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.938+0000 7fa55b10d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5540cbea0 0x7fa5540cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.938+0000 7fa55b10d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5541f6e40 0x7fa5541f72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa55b10d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5541f3640 con 0x7fa544003fb0 2024-05-08T21:22:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa55b10d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5540cc860 con 0x7fa5541f6e40 2024-05-08T21:22:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa55b10d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5541f7bb0 con 0x7fa5540cbea0 2024-05-08T21:22:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa558ea9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544003fb0 0x7fa5540b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa558ea9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544003fb0 0x7fa5540b02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40082/0 (socket says 172.21.15.3:40082) 2024-05-08T21:22:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa558ea9700 1 -- 172.21.15.3:0/591356441 learned_addr learned my addr 172.21.15.3:0/591356441 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa5596aa700 1 --2- 172.21.15.3:0/591356441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5541f6e40 0x7fa5541f72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa553fff700 1 --2- 172.21.15.3:0/591356441 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5540cbea0 0x7fa5540cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa558ea9700 1 -- 172.21.15.3:0/591356441 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5540cbea0 msgr2=0x7fa5540cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa558ea9700 1 --2- 172.21.15.3:0/591356441 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5540cbea0 0x7fa5540cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa558ea9700 1 -- 172.21.15.3:0/591356441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5541f6e40 msgr2=0x7fa5541f72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa558ea9700 1 --2- 172.21.15.3:0/591356441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5541f6e40 0x7fa5541f72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa558ea9700 1 -- 172.21.15.3:0/591356441 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5540b07e0 con 0x7fa544003fb0 2024-05-08T21:22:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa558ea9700 1 --2- 172.21.15.3:0/591356441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544003fb0 0x7fa5540b02a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa5480072c0 tx=0x7fa5480075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa5517fa700 1 -- 172.21.15.3:0/591356441 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa548017070 con 0x7fa544003fb0 2024-05-08T21:22:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa5517fa700 1 -- 172.21.15.3:0/591356441 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa548011580 con 0x7fa544003fb0 2024-05-08T21:22:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa5517fa700 1 -- 172.21.15.3:0/591356441 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa548011710 con 0x7fa544003fb0 2024-05-08T21:22:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa55b10d700 1 -- 172.21.15.3:0/591356441 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5541f45f0 con 0x7fa544003fb0 2024-05-08T21:22:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.939+0000 7fa55b10d700 1 -- 172.21.15.3:0/591356441 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5541f3b10 con 0x7fa544003fb0 2024-05-08T21:22:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.940+0000 7fa55b10d700 1 -- 172.21.15.3:0/591356441 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa554068000 con 0x7fa544003fb0 2024-05-08T21:22:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.941+0000 7fa5517fa700 1 -- 172.21.15.3:0/591356441 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa548011870 con 0x7fa544003fb0 2024-05-08T21:22:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.941+0000 7fa5517fa700 1 --2- 172.21.15.3:0/591356441 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa53806ac90 0x7fa53806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.942+0000 7fa5517fa700 1 -- 172.21.15.3:0/591356441 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa548088ce0 con 0x7fa544003fb0 2024-05-08T21:22:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.942+0000 7fa553fff700 1 --2- 172.21.15.3:0/591356441 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa53806ac90 0x7fa53806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:20.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.942+0000 7fa553fff700 1 --2- 172.21.15.3:0/591356441 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa53806ac90 0x7fa53806d150 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa5400090b0 tx=0x7fa540008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:20.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:20.946+0000 7fa5517fa700 1 -- 172.21.15.3:0/591356441 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fa548053e20 con 0x7fa544003fb0 2024-05-08T21:22:21.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.166+0000 7fa55b10d700 1 -- 172.21.15.3:0/591356441 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5540b2df0 con 0x7fa544003fb0 2024-05-08T21:22:21.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.167+0000 7fa5517fa700 1 -- 172.21.15.3:0/591356441 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15989 (secure 0 0 0) 0x7fa548057440 con 0x7fa544003fb0 2024-05-08T21:22:21.169 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:21.169 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:53.064342+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:21.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.169+0000 7fa55b10d700 1 -- 172.21.15.3:0/591356441 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa53806ac90 msgr2=0x7fa53806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:21.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.169+0000 7fa55b10d700 1 --2- 172.21.15.3:0/591356441 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa53806ac90 0x7fa53806d150 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa5400090b0 tx=0x7fa540008030 comp rx=0 tx=0).stop 2024-05-08T21:22:21.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.169+0000 7fa55b10d700 1 -- 172.21.15.3:0/591356441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544003fb0 msgr2=0x7fa5540b02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:21.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.169+0000 7fa55b10d700 1 --2- 172.21.15.3:0/591356441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544003fb0 0x7fa5540b02a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa5480072c0 tx=0x7fa5480075d0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.170+0000 7fa55b10d700 1 -- 172.21.15.3:0/591356441 shutdown_connections 2024-05-08T21:22:21.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.170+0000 7fa55b10d700 1 --2- 172.21.15.3:0/591356441 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa53806ac90 0x7fa53806d150 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.170+0000 7fa55b10d700 1 --2- 172.21.15.3:0/591356441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5541f6e40 0x7fa5541f72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.170+0000 7fa55b10d700 1 --2- 172.21.15.3:0/591356441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544003fb0 0x7fa5540b02a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.170+0000 7fa55b10d700 1 --2- 172.21.15.3:0/591356441 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5540cbea0 0x7fa5540cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.170+0000 7fa55b10d700 1 -- 172.21.15.3:0/591356441 >> 172.21.15.3:0/591356441 conn(0x7fa5540b3010 msgr2=0x7fa5540739a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:21.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.170+0000 7fa55b10d700 1 -- 172.21.15.3:0/591356441 shutdown_connections 2024-05-08T21:22:21.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.170+0000 7fa55b10d700 1 -- 172.21.15.3:0/591356441 wait complete. 2024-05-08T21:22:21.186 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-08T21:21:49.638998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:21:51.024632+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:22:21.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:22:21.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.323+0000 7fe19d167700 1 Processor -- start 2024-05-08T21:22:21.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.324+0000 7fe19d167700 1 -- start start 2024-05-08T21:22:21.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.324+0000 7fe19d167700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe19815b710 0x7fe198099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:21.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.324+0000 7fe19d167700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe1980b26e0 0x7fe198099f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:21.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.324+0000 7fe19d167700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1980d04b0 con 0x7fe1980cff30 2024-05-08T21:22:21.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.324+0000 7fe19d167700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1980b0910 con 0x7fe1980b26e0 2024-05-08T21:22:21.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.324+0000 7fe19d167700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1980b0bf0 con 0x7fe19815b710 2024-05-08T21:22:21.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.325+0000 7fe19759e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe1980b26e0 0x7fe198099f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:21.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.325+0000 7fe19759e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe1980b26e0 0x7fe198099f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43896/0 (socket says 172.21.15.3:43896) 2024-05-08T21:22:21.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.325+0000 7fe19659c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe1980cff30 0x7fe1980cc130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59410/0 (socket says 172.21.15.3:59410) 2024-05-08T21:22:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.325+0000 7fe19759e700 1 -- 172.21.15.3:0/111122576 learned_addr learned my addr 172.21.15.3:0/111122576 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.325+0000 7fe196d9d700 1 --2- 172.21.15.3:0/111122576 >> v2:172.21.15.186:3300/0 conn(0x7fe19815b710 0x7fe198099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.325+0000 7fe19759e700 1 -- 172.21.15.3:0/111122576 >> v2:172.21.15.186:3300/0 conn(0x7fe19815b710 msgr2=0x7fe198099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.325+0000 7fe19759e700 1 --2- 172.21.15.3:0/111122576 >> v2:172.21.15.186:3300/0 conn(0x7fe19815b710 0x7fe198099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.325+0000 7fe19759e700 1 -- 172.21.15.3:0/111122576 >> v1:172.21.15.3:6789/0 conn(0x7fe1980cff30 legacy=0x7fe1980cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.325+0000 7fe19759e700 1 -- 172.21.15.3:0/111122576 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1980b0ed0 con 0x7fe1980b26e0 2024-05-08T21:22:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.325+0000 7fe195d9b700 1 -- 172.21.15.3:0/111122576 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790640915 0 0) 0x7fe1980d04b0 con 0x7fe1980cff30 2024-05-08T21:22:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.325+0000 7fe19759e700 1 --2- 172.21.15.3:0/111122576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1980b26e0 0x7fe198099f90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe1900098f0 tx=0x7fe190009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=704542b56529e45f server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.325+0000 7fe195d9b700 1 -- 172.21.15.3:0/111122576 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe190011830 con 0x7fe1980b26e0 2024-05-08T21:22:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.325+0000 7fe195d9b700 1 -- 172.21.15.3:0/111122576 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe190011990 con 0x7fe1980b26e0 2024-05-08T21:22:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.325+0000 7fe195d9b700 1 -- 172.21.15.3:0/111122576 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe190011d20 con 0x7fe1980b26e0 2024-05-08T21:22:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.326+0000 7fe19d167700 1 -- 172.21.15.3:0/111122576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1980b26e0 msgr2=0x7fe198099f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.326+0000 7fe19d167700 1 --2- 172.21.15.3:0/111122576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1980b26e0 0x7fe198099f90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe1900098f0 tx=0x7fe190009c00 comp rx=0 tx=0).stop 2024-05-08T21:22:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.326+0000 7fe19d167700 1 -- 172.21.15.3:0/111122576 shutdown_connections 2024-05-08T21:22:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.326+0000 7fe19d167700 1 --2- 172.21.15.3:0/111122576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1980b26e0 0x7fe198099f90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.326+0000 7fe19d167700 1 --2- 172.21.15.3:0/111122576 >> v2:172.21.15.186:3300/0 conn(0x7fe19815b710 0x7fe198099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.326+0000 7fe19d167700 1 -- 172.21.15.3:0/111122576 >> 172.21.15.3:0/111122576 conn(0x7fe1980b2f60 msgr2=0x7fe1980b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:21.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.326+0000 7fe19d167700 1 -- 172.21.15.3:0/111122576 shutdown_connections 2024-05-08T21:22:21.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.326+0000 7fe19d167700 1 -- 172.21.15.3:0/111122576 wait complete. 2024-05-08T21:22:21.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.328+0000 7fe19d167700 1 Processor -- start 2024-05-08T21:22:21.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.328+0000 7fe19d167700 1 -- start start 2024-05-08T21:22:21.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.328+0000 7fe19d167700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1980b26e0 0x7fe198159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:21.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.328+0000 7fe19d167700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1980cff30 0x7fe1981f2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:21.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe19d167700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe19815b710 0x7fe1980c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:21.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe19d167700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19806bf80 con 0x7fe1980b26e0 2024-05-08T21:22:21.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe19d167700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1981f3d90 con 0x7fe19815b710 2024-05-08T21:22:21.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe19d167700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1981f37d0 con 0x7fe1980cff30 2024-05-08T21:22:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe196d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1980b26e0 0x7fe198159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe196d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1980b26e0 0x7fe198159420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43898/0 (socket says 172.21.15.3:43898) 2024-05-08T21:22:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe196d9d700 1 -- 172.21.15.3:0/2712588735 learned_addr learned my addr 172.21.15.3:0/2712588735 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe19759e700 1 --2- 172.21.15.3:0/2712588735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe19815b710 0x7fe1980c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe19659c700 1 --2- 172.21.15.3:0/2712588735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1980cff30 0x7fe1981f2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe196d9d700 1 -- 172.21.15.3:0/2712588735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1980cff30 msgr2=0x7fe1981f2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe196d9d700 1 --2- 172.21.15.3:0/2712588735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1980cff30 0x7fe1981f2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe196d9d700 1 -- 172.21.15.3:0/2712588735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe19815b710 msgr2=0x7fe1980c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe196d9d700 1 --2- 172.21.15.3:0/2712588735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe19815b710 0x7fe1980c8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe196d9d700 1 -- 172.21.15.3:0/2712588735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe190009550 con 0x7fe1980b26e0 2024-05-08T21:22:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe196d9d700 1 --2- 172.21.15.3:0/2712588735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1980b26e0 0x7fe198159420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe18800efc0 tx=0x7fe18800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe17f7fe700 1 -- 172.21.15.3:0/2712588735 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18800cdc0 con 0x7fe1980b26e0 2024-05-08T21:22:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe17f7fe700 1 -- 172.21.15.3:0/2712588735 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18800f040 con 0x7fe1980b26e0 2024-05-08T21:22:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe19d167700 1 -- 172.21.15.3:0/2712588735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1981f3ab0 con 0x7fe1980b26e0 2024-05-08T21:22:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe17f7fe700 1 -- 172.21.15.3:0/2712588735 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1880146a0 con 0x7fe1980b26e0 2024-05-08T21:22:21.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.329+0000 7fe19d167700 1 -- 172.21.15.3:0/2712588735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe198159960 con 0x7fe1980b26e0 2024-05-08T21:22:21.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.330+0000 7fe19d167700 1 -- 172.21.15.3:0/2712588735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1980cd310 con 0x7fe1980b26e0 2024-05-08T21:22:21.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.331+0000 7fe17f7fe700 1 -- 172.21.15.3:0/2712588735 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe188014800 con 0x7fe1980b26e0 2024-05-08T21:22:21.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.331+0000 7fe17f7fe700 1 --2- 172.21.15.3:0/2712588735 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe18406ac90 0x7fe18406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:21.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.332+0000 7fe17f7fe700 1 -- 172.21.15.3:0/2712588735 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe188089f20 con 0x7fe1980b26e0 2024-05-08T21:22:21.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.332+0000 7fe19659c700 1 --2- 172.21.15.3:0/2712588735 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe18406ac90 0x7fe18406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:21.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.332+0000 7fe19659c700 1 --2- 172.21.15.3:0/2712588735 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe18406ac90 0x7fe18406d150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe1980b6ef0 tx=0x7fe18c007be0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:21.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.336+0000 7fe17f7fe700 1 -- 172.21.15.3:0/2712588735 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe188055190 con 0x7fe1980b26e0 2024-05-08T21:22:21.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.554+0000 7fe19d167700 1 -- 172.21.15.3:0/2712588735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe198074bc0 con 0x7fe1980b26e0 2024-05-08T21:22:21.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.555+0000 7fe17f7fe700 1 -- 172.21.15.3:0/2712588735 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe1880587b0 con 0x7fe1980b26e0 2024-05-08T21:22:21.557 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:22:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.557+0000 7fe19d167700 1 -- 172.21.15.3:0/2712588735 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe18406ac90 msgr2=0x7fe18406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.557+0000 7fe19d167700 1 --2- 172.21.15.3:0/2712588735 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe18406ac90 0x7fe18406d150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe1980b6ef0 tx=0x7fe18c007be0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.557+0000 7fe19d167700 1 -- 172.21.15.3:0/2712588735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1980b26e0 msgr2=0x7fe198159420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.557+0000 7fe19d167700 1 --2- 172.21.15.3:0/2712588735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1980b26e0 0x7fe198159420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe18800efc0 tx=0x7fe18800c3a0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.557+0000 7fe19d167700 1 -- 172.21.15.3:0/2712588735 shutdown_connections 2024-05-08T21:22:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.557+0000 7fe19d167700 1 --2- 172.21.15.3:0/2712588735 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe18406ac90 0x7fe18406d150 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.557+0000 7fe19d167700 1 --2- 172.21.15.3:0/2712588735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe19815b710 0x7fe1980c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.557+0000 7fe19d167700 1 --2- 172.21.15.3:0/2712588735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1980b26e0 0x7fe198159420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.557+0000 7fe19d167700 1 --2- 172.21.15.3:0/2712588735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1980cff30 0x7fe1981f2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.557+0000 7fe19d167700 1 -- 172.21.15.3:0/2712588735 >> 172.21.15.3:0/2712588735 conn(0x7fe1980b2f60 msgr2=0x7fe1980b3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.557+0000 7fe19d167700 1 -- 172.21.15.3:0/2712588735 shutdown_connections 2024-05-08T21:22:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.557+0000 7fe19d167700 1 -- 172.21.15.3:0/2712588735 wait complete. 2024-05-08T21:22:21.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-08T21:22:21.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.710+0000 7f6269a84700 1 Processor -- start 2024-05-08T21:22:21.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.710+0000 7f6269a84700 1 -- start start 2024-05-08T21:22:21.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.710+0000 7f6269a84700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f626415a7c0 0x7f626415aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:21.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.710+0000 7f6269a84700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62640af1d0 0x7f62640af630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:21.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.710+0000 7f6269a84700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62640afb70 con 0x7f62640b3d00 2024-05-08T21:22:21.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.710+0000 7f6269a84700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62640aecf0 con 0x7f626415a7c0 2024-05-08T21:22:21.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.710+0000 7f6269a84700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62640b4950 con 0x7f62640af1d0 2024-05-08T21:22:21.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.711+0000 7f6263fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f62640b3d00 0x7f62640b4160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59412/0 (socket says 172.21.15.3:59412) 2024-05-08T21:22:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.711+0000 7f6263fff700 1 -- 172.21.15.3:0/2820551503 learned_addr learned my addr 172.21.15.3:0/2820551503 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.711+0000 7f6262ffd700 1 --2- 172.21.15.3:0/2820551503 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62640af1d0 0x7f62640af630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.711+0000 7f62637fe700 1 --2- 172.21.15.3:0/2820551503 >> v2:172.21.15.186:3300/0 conn(0x7f626415a7c0 0x7f626415aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.711+0000 7f62627fc700 1 -- 172.21.15.3:0/2820551503 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682823186 0 0) 0x7f62640afb70 con 0x7f62640b3d00 2024-05-08T21:22:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.711+0000 7f62627fc700 1 -- 172.21.15.3:0/2820551503 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f624c0035b0 con 0x7f62640b3d00 2024-05-08T21:22:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.711+0000 7f62637fe700 1 -- 172.21.15.3:0/2820551503 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62640af1d0 msgr2=0x7f62640af630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.711+0000 7f62637fe700 1 --2- 172.21.15.3:0/2820551503 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62640af1d0 0x7f62640af630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.711+0000 7f62637fe700 1 -- 172.21.15.3:0/2820551503 >> v1:172.21.15.3:6789/0 conn(0x7f62640b3d00 legacy=0x7f62640b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.711+0000 7f62637fe700 1 -- 172.21.15.3:0/2820551503 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626406de70 con 0x7f626415a7c0 2024-05-08T21:22:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.711+0000 7f6262ffd700 1 --2- 172.21.15.3:0/2820551503 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62640af1d0 0x7f62640af630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.711+0000 7f62637fe700 1 --2- 172.21.15.3:0/2820551503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f626415a7c0 0x7f626415aba0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6254009840 tx=0x7f6254009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ff3c5c19aaae983 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.712+0000 7f62627fc700 1 -- 172.21.15.3:0/2820551503 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6254011830 con 0x7f626415a7c0 2024-05-08T21:22:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.712+0000 7f62627fc700 1 -- 172.21.15.3:0/2820551503 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6254011990 con 0x7f626415a7c0 2024-05-08T21:22:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.712+0000 7f62627fc700 1 -- 172.21.15.3:0/2820551503 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6254011d20 con 0x7f626415a7c0 2024-05-08T21:22:21.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.712+0000 7f6269a84700 1 -- 172.21.15.3:0/2820551503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f626415a7c0 msgr2=0x7f626415aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:21.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.712+0000 7f6269a84700 1 --2- 172.21.15.3:0/2820551503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f626415a7c0 0x7f626415aba0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6254009840 tx=0x7f6254009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:21.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.712+0000 7f6269a84700 1 -- 172.21.15.3:0/2820551503 shutdown_connections 2024-05-08T21:22:21.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.712+0000 7f6269a84700 1 --2- 172.21.15.3:0/2820551503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f626415a7c0 0x7f626415aba0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.712+0000 7f6269a84700 1 --2- 172.21.15.3:0/2820551503 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62640af1d0 0x7f62640af630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.712+0000 7f6269a84700 1 -- 172.21.15.3:0/2820551503 >> 172.21.15.3:0/2820551503 conn(0x7f62640b5810 msgr2=0x7f62640b5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:21.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.712+0000 7f6269a84700 1 -- 172.21.15.3:0/2820551503 shutdown_connections 2024-05-08T21:22:21.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.712+0000 7f6269a84700 1 -- 172.21.15.3:0/2820551503 wait complete. 2024-05-08T21:22:21.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.714+0000 7f6269a84700 1 Processor -- start 2024-05-08T21:22:21.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.714+0000 7f6269a84700 1 -- start start 2024-05-08T21:22:21.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f6269a84700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62640af1d0 0x7f6264141d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:21.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f6269a84700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62640b3d00 0x7f626415c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:21.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f6269a84700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f626415a7c0 0x7f62641566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:21.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f6269a84700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626415d0b0 con 0x7f62640af1d0 2024-05-08T21:22:21.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f6269a84700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626414d940 con 0x7f626415a7c0 2024-05-08T21:22:21.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f6269a84700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626414cd30 con 0x7f62640b3d00 2024-05-08T21:22:21.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f62637fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62640af1d0 0x7f6264141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:21.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f62637fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62640af1d0 0x7f6264141d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43910/0 (socket says 172.21.15.3:43910) 2024-05-08T21:22:21.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f62637fe700 1 -- 172.21.15.3:0/1890519381 learned_addr learned my addr 172.21.15.3:0/1890519381 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f6263fff700 1 --2- 172.21.15.3:0/1890519381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f626415a7c0 0x7f62641566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f6262ffd700 1 --2- 172.21.15.3:0/1890519381 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62640b3d00 0x7f626415c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f62637fe700 1 -- 172.21.15.3:0/1890519381 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62640b3d00 msgr2=0x7f626415c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f62637fe700 1 --2- 172.21.15.3:0/1890519381 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62640b3d00 0x7f626415c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f62637fe700 1 -- 172.21.15.3:0/1890519381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f626415a7c0 msgr2=0x7f62641566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f62637fe700 1 --2- 172.21.15.3:0/1890519381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f626415a7c0 0x7f62641566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f62637fe700 1 -- 172.21.15.3:0/1890519381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62580094a0 con 0x7f62640af1d0 2024-05-08T21:22:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f6262ffd700 1 --2- 172.21.15.3:0/1890519381 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62640b3d00 0x7f626415c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f6263fff700 1 --2- 172.21.15.3:0/1890519381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f626415a7c0 0x7f62641566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f62637fe700 1 --2- 172.21.15.3:0/1890519381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62640af1d0 0x7f6264141d60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6254009790 tx=0x7f6254004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f624bfff700 1 -- 172.21.15.3:0/1890519381 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62540036a0 con 0x7f62640af1d0 2024-05-08T21:22:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f624bfff700 1 -- 172.21.15.3:0/1890519381 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625401e070 con 0x7f62640af1d0 2024-05-08T21:22:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f6269a84700 1 -- 172.21.15.3:0/1890519381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62540094a0 con 0x7f62640af1d0 2024-05-08T21:22:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f624bfff700 1 -- 172.21.15.3:0/1890519381 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6254003a20 con 0x7f62640af1d0 2024-05-08T21:22:21.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.715+0000 7f6269a84700 1 -- 172.21.15.3:0/1890519381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626414d2f0 con 0x7f62640af1d0 2024-05-08T21:22:21.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.716+0000 7f6269a84700 1 -- 172.21.15.3:0/1890519381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62640b2e00 con 0x7f62640af1d0 2024-05-08T21:22:21.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.717+0000 7f624bfff700 1 -- 172.21.15.3:0/1890519381 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6254003b80 con 0x7f62640af1d0 2024-05-08T21:22:21.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.717+0000 7f624bfff700 1 --2- 172.21.15.3:0/1890519381 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f625006ac90 0x7f625006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:21.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.717+0000 7f624bfff700 1 -- 172.21.15.3:0/1890519381 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f625408a4b0 con 0x7f62640af1d0 2024-05-08T21:22:21.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.718+0000 7f6262ffd700 1 --2- 172.21.15.3:0/1890519381 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f625006ac90 0x7f625006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:21.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.718+0000 7f6262ffd700 1 --2- 172.21.15.3:0/1890519381 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f625006ac90 0x7f625006d150 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f62641489e0 tx=0x7f6258008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:21.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.722+0000 7f624bfff700 1 -- 172.21.15.3:0/1890519381 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6254055600 con 0x7f62640af1d0 2024-05-08T21:22:21.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.937+0000 7f6269a84700 1 -- 172.21.15.3:0/1890519381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f626414dc20 con 0x7f62640af1d0 2024-05-08T21:22:21.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.937+0000 7f624bfff700 1 -- 172.21.15.3:0/1890519381 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6254058c20 con 0x7f62640af1d0 2024-05-08T21:22:21.939 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:22:21.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.939+0000 7f6269a84700 1 -- 172.21.15.3:0/1890519381 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f625006ac90 msgr2=0x7f625006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:21.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.939+0000 7f6269a84700 1 --2- 172.21.15.3:0/1890519381 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f625006ac90 0x7f625006d150 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f62641489e0 tx=0x7f6258008040 comp rx=0 tx=0).stop 2024-05-08T21:22:21.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.939+0000 7f6269a84700 1 -- 172.21.15.3:0/1890519381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62640af1d0 msgr2=0x7f6264141d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:21.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.939+0000 7f6269a84700 1 --2- 172.21.15.3:0/1890519381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62640af1d0 0x7f6264141d60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6254009790 tx=0x7f6254004750 comp rx=0 tx=0).stop 2024-05-08T21:22:21.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.939+0000 7f6269a84700 1 -- 172.21.15.3:0/1890519381 shutdown_connections 2024-05-08T21:22:21.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.939+0000 7f6269a84700 1 --2- 172.21.15.3:0/1890519381 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f625006ac90 0x7f625006d150 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.939+0000 7f6269a84700 1 --2- 172.21.15.3:0/1890519381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f626415a7c0 0x7f62641566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.939+0000 7f6269a84700 1 --2- 172.21.15.3:0/1890519381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62640af1d0 0x7f6264141d60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.939+0000 7f6269a84700 1 --2- 172.21.15.3:0/1890519381 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62640b3d00 0x7f626415c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:21.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.939+0000 7f6269a84700 1 -- 172.21.15.3:0/1890519381 >> 172.21.15.3:0/1890519381 conn(0x7f62640b5810 msgr2=0x7f6264150fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:21.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.939+0000 7f6269a84700 1 -- 172.21.15.3:0/1890519381 shutdown_connections 2024-05-08T21:22:21.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:21.939+0000 7f6269a84700 1 -- 172.21.15.3:0/1890519381 wait complete. 2024-05-08T21:22:21.955 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:22:22.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.092+0000 7f89bb755700 1 Processor -- start 2024-05-08T21:22:22.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.092+0000 7f89bb755700 1 -- start start 2024-05-08T21:22:22.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.093+0000 7f89bb755700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f89b40aea70 0x7f89b40b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.093+0000 7f89bb755700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b40d0a60 0x7f89b40b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.093+0000 7f89bb755700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b415add0 con 0x7f89b4154df0 2024-05-08T21:22:22.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.093+0000 7f89bb755700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b415abe0 con 0x7f89b40aea70 2024-05-08T21:22:22.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.093+0000 7f89bb755700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b415b070 con 0x7f89b40d0a60 2024-05-08T21:22:22.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.093+0000 7f89b8cf0700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f89b40aea70 0x7f89b40b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.093+0000 7f89b8cf0700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f89b40aea70 0x7f89b40b3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43926/0 (socket says 172.21.15.3:43926) 2024-05-08T21:22:22.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.093+0000 7f89b8cf0700 1 -- 172.21.15.3:0/888324037 learned_addr learned my addr 172.21.15.3:0/888324037 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:22.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.093+0000 7f89b9cf2700 1 --2- 172.21.15.3:0/888324037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b40d0a60 0x7f89b40b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.093+0000 7f89b8cf0700 1 -- 172.21.15.3:0/888324037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b40d0a60 msgr2=0x7f89b40b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:22.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.093+0000 7f89b8cf0700 1 --2- 172.21.15.3:0/888324037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b40d0a60 0x7f89b40b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.094+0000 7f89b8cf0700 1 -- 172.21.15.3:0/888324037 >> v1:172.21.15.186:6789/0 conn(0x7f89b4154df0 legacy=0x7f89b40d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:22.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.094+0000 7f89b8cf0700 1 -- 172.21.15.3:0/888324037 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b415b350 con 0x7f89b40aea70 2024-05-08T21:22:22.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.094+0000 7f89b8cf0700 1 --2- 172.21.15.3:0/888324037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b40aea70 0x7f89b40b3e70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f89a40098f0 tx=0x7f89a4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6eb23a61338f864 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:22.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.094+0000 7f89abfff700 1 -- 172.21.15.3:0/888324037 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a400b8a0 con 0x7f89b40aea70 2024-05-08T21:22:22.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.094+0000 7f89abfff700 1 -- 172.21.15.3:0/888324037 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a400ba00 con 0x7f89b40aea70 2024-05-08T21:22:22.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.094+0000 7f89abfff700 1 -- 172.21.15.3:0/888324037 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a400bd90 con 0x7f89b40aea70 2024-05-08T21:22:22.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.094+0000 7f89bb755700 1 -- 172.21.15.3:0/888324037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b40aea70 msgr2=0x7f89b40b3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:22.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.094+0000 7f89bb755700 1 --2- 172.21.15.3:0/888324037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b40aea70 0x7f89b40b3e70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f89a40098f0 tx=0x7f89a4009c00 comp rx=0 tx=0).stop 2024-05-08T21:22:22.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.095+0000 7f89bb755700 1 -- 172.21.15.3:0/888324037 shutdown_connections 2024-05-08T21:22:22.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.095+0000 7f89bb755700 1 --2- 172.21.15.3:0/888324037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b40d0a60 0x7f89b40b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.095+0000 7f89bb755700 1 --2- 172.21.15.3:0/888324037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b40aea70 0x7f89b40b3e70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.095+0000 7f89bb755700 1 -- 172.21.15.3:0/888324037 >> 172.21.15.3:0/888324037 conn(0x7f89b4099a50 msgr2=0x7f89b40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:22.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.095+0000 7f89bb755700 1 -- 172.21.15.3:0/888324037 shutdown_connections 2024-05-08T21:22:22.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.095+0000 7f89bb755700 1 -- 172.21.15.3:0/888324037 wait complete. 2024-05-08T21:22:22.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.096+0000 7f89bb755700 1 Processor -- start 2024-05-08T21:22:22.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.096+0000 7f89bb755700 1 -- start start 2024-05-08T21:22:22.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.097+0000 7f89bb755700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b40aea70 0x7f89b414b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.097+0000 7f89bb755700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b40d0a60 0x7f89b4150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.097+0000 7f89bb755700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b4154df0 0x7f89b40ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.097+0000 7f89bb755700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b414c440 con 0x7f89b40d0a60 2024-05-08T21:22:22.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.097+0000 7f89bb755700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b414c720 con 0x7f89b40aea70 2024-05-08T21:22:22.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.097+0000 7f89bb755700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b414ca00 con 0x7f89b4154df0 2024-05-08T21:22:22.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.098+0000 7f89b8cf0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b40d0a60 0x7f89b4150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.098+0000 7f89b8cf0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b40d0a60 0x7f89b4150890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43942/0 (socket says 172.21.15.3:43942) 2024-05-08T21:22:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.098+0000 7f89b8cf0700 1 -- 172.21.15.3:0/4128992956 learned_addr learned my addr 172.21.15.3:0/4128992956 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.098+0000 7f89b9cf2700 1 --2- 172.21.15.3:0/4128992956 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b4154df0 0x7f89b40ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.098+0000 7f89b94f1700 1 --2- 172.21.15.3:0/4128992956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b40aea70 0x7f89b414b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.098+0000 7f89b8cf0700 1 -- 172.21.15.3:0/4128992956 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b4154df0 msgr2=0x7f89b40ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.098+0000 7f89b8cf0700 1 --2- 172.21.15.3:0/4128992956 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b4154df0 0x7f89b40ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.098+0000 7f89b8cf0700 1 -- 172.21.15.3:0/4128992956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b40aea70 msgr2=0x7f89b414b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.098+0000 7f89b8cf0700 1 --2- 172.21.15.3:0/4128992956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b40aea70 0x7f89b414b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.098+0000 7f89b8cf0700 1 -- 172.21.15.3:0/4128992956 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a4009550 con 0x7f89b40d0a60 2024-05-08T21:22:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.098+0000 7f89b8cf0700 1 --2- 172.21.15.3:0/4128992956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b40d0a60 0x7f89b4150890 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f89a4000c00 tx=0x7f89a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.098+0000 7f89a9ffb700 1 -- 172.21.15.3:0/4128992956 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a4004300 con 0x7f89b40d0a60 2024-05-08T21:22:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.098+0000 7f89a9ffb700 1 -- 172.21.15.3:0/4128992956 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a401f070 con 0x7f89b40d0a60 2024-05-08T21:22:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.098+0000 7f89a9ffb700 1 -- 172.21.15.3:0/4128992956 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a40036a0 con 0x7f89b40d0a60 2024-05-08T21:22:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.098+0000 7f89bb755700 1 -- 172.21.15.3:0/4128992956 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b414db60 con 0x7f89b40d0a60 2024-05-08T21:22:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.099+0000 7f89bb755700 1 -- 172.21.15.3:0/4128992956 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b414cce0 con 0x7f89b40d0a60 2024-05-08T21:22:22.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.100+0000 7f89bb755700 1 -- 172.21.15.3:0/4128992956 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b4074bc0 con 0x7f89b40d0a60 2024-05-08T21:22:22.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.100+0000 7f89a9ffb700 1 -- 172.21.15.3:0/4128992956 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f89a4021030 con 0x7f89b40d0a60 2024-05-08T21:22:22.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.101+0000 7f89a9ffb700 1 --2- 172.21.15.3:0/4128992956 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f899806ac90 0x7f899806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.101+0000 7f89a9ffb700 1 -- 172.21.15.3:0/4128992956 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89a408a6d0 con 0x7f89b40d0a60 2024-05-08T21:22:22.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.101+0000 7f89b94f1700 1 --2- 172.21.15.3:0/4128992956 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f899806ac90 0x7f899806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.101+0000 7f89b94f1700 1 --2- 172.21.15.3:0/4128992956 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f899806ac90 0x7f899806d150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f89b0007fd0 tx=0x7f89b0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:22.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.105+0000 7f89a9ffb700 1 -- 172.21.15.3:0/4128992956 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f89a4055820 con 0x7f89b40d0a60 2024-05-08T21:22:22.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.321+0000 7f89bb755700 1 -- 172.21.15.3:0/4128992956 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f89b4150f70 con 0x7f89b40d0a60 2024-05-08T21:22:22.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.321+0000 7f89a9ffb700 1 -- 172.21.15.3:0/4128992956 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f89a4058e40 con 0x7f89b40d0a60 2024-05-08T21:22:22.323 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:22:22.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.323+0000 7f89bb755700 1 -- 172.21.15.3:0/4128992956 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f899806ac90 msgr2=0x7f899806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:22.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.323+0000 7f89bb755700 1 --2- 172.21.15.3:0/4128992956 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f899806ac90 0x7f899806d150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f89b0007fd0 tx=0x7f89b0009040 comp rx=0 tx=0).stop 2024-05-08T21:22:22.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.323+0000 7f89bb755700 1 -- 172.21.15.3:0/4128992956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b40d0a60 msgr2=0x7f89b4150890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:22.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.323+0000 7f89bb755700 1 --2- 172.21.15.3:0/4128992956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b40d0a60 0x7f89b4150890 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f89a4000c00 tx=0x7f89a4004750 comp rx=0 tx=0).stop 2024-05-08T21:22:22.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.323+0000 7f89bb755700 1 -- 172.21.15.3:0/4128992956 shutdown_connections 2024-05-08T21:22:22.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.323+0000 7f89bb755700 1 --2- 172.21.15.3:0/4128992956 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b4154df0 0x7f89b40ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.323+0000 7f89bb755700 1 --2- 172.21.15.3:0/4128992956 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f899806ac90 0x7f899806d150 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.323+0000 7f89bb755700 1 --2- 172.21.15.3:0/4128992956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b40aea70 0x7f89b414b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.323+0000 7f89bb755700 1 --2- 172.21.15.3:0/4128992956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b40d0a60 0x7f89b4150890 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.323+0000 7f89bb755700 1 -- 172.21.15.3:0/4128992956 >> 172.21.15.3:0/4128992956 conn(0x7f89b4099a50 msgr2=0x7f89b406ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:22.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.323+0000 7f89bb755700 1 -- 172.21.15.3:0/4128992956 shutdown_connections 2024-05-08T21:22:22.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.323+0000 7f89bb755700 1 -- 172.21.15.3:0/4128992956 wait complete. 2024-05-08T21:22:22.359 INFO:tasks.cephfs_test_runner:Starting test: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) 2024-05-08T21:22:22.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:22.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.495+0000 7f4330ff6700 1 Processor -- start 2024-05-08T21:22:22.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.495+0000 7f4330ff6700 1 -- start start 2024-05-08T21:22:22.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.496+0000 7f4330ff6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f432c158130 0x7f432c15ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.496+0000 7f4330ff6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f432c15bbb0 0x7f432c06b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.497+0000 7f4330ff6700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432c15f0f0 con 0x7f432c06beb0 2024-05-08T21:22:22.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.497+0000 7f4330ff6700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f432c06c340 con 0x7f432c158130 2024-05-08T21:22:22.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.497+0000 7f4330ff6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432c0b3cc0 con 0x7f432c15bbb0 2024-05-08T21:22:22.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.497+0000 7f432a59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f432c158130 0x7f432c15ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.497+0000 7f432a59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f432c158130 0x7f432c15ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43948/0 (socket says 172.21.15.3:43948) 2024-05-08T21:22:22.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.497+0000 7f432a59c700 1 -- 172.21.15.3:0/3312593378 learned_addr learned my addr 172.21.15.3:0/3312593378 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:22.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.497+0000 7f432a59c700 1 -- 172.21.15.3:0/3312593378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f432c15bbb0 msgr2=0x7f432c06b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:22.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.497+0000 7f4329d9b700 1 --2- 172.21.15.3:0/3312593378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f432c15bbb0 0x7f432c06b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.497+0000 7f432a59c700 1 --2- 172.21.15.3:0/3312593378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f432c15bbb0 0x7f432c06b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.497+0000 7f432a59c700 1 -- 172.21.15.3:0/3312593378 >> v1:172.21.15.186:6789/0 conn(0x7f432c06beb0 legacy=0x7f432c0ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:22.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.497+0000 7f432a59c700 1 -- 172.21.15.3:0/3312593378 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c15b030 con 0x7f432c158130 2024-05-08T21:22:22.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.497+0000 7f432a59c700 1 --2- 172.21.15.3:0/3312593378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f432c158130 0x7f432c15ebb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f431c0098f0 tx=0x7f431c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c18278b9b40b62 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:22.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.497+0000 7f432959a700 1 -- 172.21.15.3:0/3312593378 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c004230 con 0x7f432c158130 2024-05-08T21:22:22.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.497+0000 7f432959a700 1 -- 172.21.15.3:0/3312593378 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431c004390 con 0x7f432c158130 2024-05-08T21:22:22.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.497+0000 7f432959a700 1 -- 172.21.15.3:0/3312593378 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c0036a0 con 0x7f432c158130 2024-05-08T21:22:22.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.498+0000 7f4330ff6700 1 -- 172.21.15.3:0/3312593378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f432c158130 msgr2=0x7f432c15ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:22.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.498+0000 7f4330ff6700 1 --2- 172.21.15.3:0/3312593378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f432c158130 0x7f432c15ebb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f431c0098f0 tx=0x7f431c009c00 comp rx=0 tx=0).stop 2024-05-08T21:22:22.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.498+0000 7f4330ff6700 1 -- 172.21.15.3:0/3312593378 shutdown_connections 2024-05-08T21:22:22.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.498+0000 7f4330ff6700 1 --2- 172.21.15.3:0/3312593378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f432c158130 0x7f432c15ebb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.498+0000 7f4330ff6700 1 --2- 172.21.15.3:0/3312593378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f432c15bbb0 0x7f432c06b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.498+0000 7f4330ff6700 1 -- 172.21.15.3:0/3312593378 >> 172.21.15.3:0/3312593378 conn(0x7f432c071350 msgr2=0x7f432c071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:22.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.498+0000 7f4330ff6700 1 -- 172.21.15.3:0/3312593378 shutdown_connections 2024-05-08T21:22:22.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.498+0000 7f4330ff6700 1 -- 172.21.15.3:0/3312593378 wait complete. 2024-05-08T21:22:22.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.499+0000 7f4330ff6700 1 Processor -- start 2024-05-08T21:22:22.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.500+0000 7f4330ff6700 1 -- start start 2024-05-08T21:22:22.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.500+0000 7f4330ff6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f432c06beb0 0x7f432c144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.500+0000 7f4330ff6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432c158130 0x7f432c0ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.500+0000 7f4330ff6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f432c15bbb0 0x7f432c06e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.500+0000 7f4330ff6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c15d3c0 con 0x7f432c06beb0 2024-05-08T21:22:22.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.500+0000 7f4330ff6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c15d6a0 con 0x7f432c158130 2024-05-08T21:22:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.500+0000 7f4330ff6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432c15d980 con 0x7f432c15bbb0 2024-05-08T21:22:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.500+0000 7f432a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f432c06beb0 0x7f432c144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.500+0000 7f432a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f432c06beb0 0x7f432c144e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43960/0 (socket says 172.21.15.3:43960) 2024-05-08T21:22:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.500+0000 7f432a59c700 1 -- 172.21.15.3:0/276532056 learned_addr learned my addr 172.21.15.3:0/276532056 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.501+0000 7f4329d9b700 1 --2- 172.21.15.3:0/276532056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432c158130 0x7f432c0ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.501+0000 7f432ad9d700 1 --2- 172.21.15.3:0/276532056 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f432c15bbb0 0x7f432c06e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.501+0000 7f432a59c700 1 -- 172.21.15.3:0/276532056 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f432c15bbb0 msgr2=0x7f432c06e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.501+0000 7f432a59c700 1 --2- 172.21.15.3:0/276532056 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f432c15bbb0 0x7f432c06e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.501+0000 7f432a59c700 1 -- 172.21.15.3:0/276532056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432c158130 msgr2=0x7f432c0ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.501+0000 7f432a59c700 1 --2- 172.21.15.3:0/276532056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432c158130 0x7f432c0ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.501+0000 7f432a59c700 1 -- 172.21.15.3:0/276532056 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c009550 con 0x7f432c06beb0 2024-05-08T21:22:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.501+0000 7f432a59c700 1 --2- 172.21.15.3:0/276532056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f432c06beb0 0x7f432c144e80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f431c009130 tx=0x7f431c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.501+0000 7f431affd700 1 -- 172.21.15.3:0/276532056 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c003e30 con 0x7f432c06beb0 2024-05-08T21:22:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.501+0000 7f431affd700 1 -- 172.21.15.3:0/276532056 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431c01f070 con 0x7f432c06beb0 2024-05-08T21:22:22.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.501+0000 7f4330ff6700 1 -- 172.21.15.3:0/276532056 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432c15dc60 con 0x7f432c06beb0 2024-05-08T21:22:22.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.501+0000 7f431affd700 1 -- 172.21.15.3:0/276532056 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c00b8a0 con 0x7f432c06beb0 2024-05-08T21:22:22.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.501+0000 7f4330ff6700 1 -- 172.21.15.3:0/276532056 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432c147cf0 con 0x7f432c06beb0 2024-05-08T21:22:22.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.502+0000 7f4330ff6700 1 -- 172.21.15.3:0/276532056 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f432c0b2700 con 0x7f432c06beb0 2024-05-08T21:22:22.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.503+0000 7f431affd700 1 -- 172.21.15.3:0/276532056 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f431c021030 con 0x7f432c06beb0 2024-05-08T21:22:22.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.503+0000 7f431affd700 1 --2- 172.21.15.3:0/276532056 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f430c06ac90 0x7f430c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.503+0000 7f431affd700 1 -- 172.21.15.3:0/276532056 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f431c08a860 con 0x7f432c06beb0 2024-05-08T21:22:22.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.503+0000 7f4329d9b700 1 --2- 172.21.15.3:0/276532056 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f430c06ac90 0x7f430c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.504+0000 7f4329d9b700 1 --2- 172.21.15.3:0/276532056 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f430c06ac90 0x7f430c06d150 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f432c0cb880 tx=0x7f4320008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:22.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.508+0000 7f431affd700 1 -- 172.21.15.3:0/276532056 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f431c055ad0 con 0x7f432c06beb0 2024-05-08T21:22:22.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.729+0000 7f4330ff6700 1 -- 172.21.15.3:0/276532056 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f432c15f0f0 con 0x7f432c06beb0 2024-05-08T21:22:22.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.730+0000 7f431affd700 1 -- 172.21.15.3:0/276532056 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15989 (secure 0 0 0) 0x7f432c15f0f0 con 0x7f432c06beb0 2024-05-08T21:22:22.732 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:22.732 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:53.064342+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:22.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.733+0000 7f4330ff6700 1 -- 172.21.15.3:0/276532056 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f430c06ac90 msgr2=0x7f430c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:22.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.733+0000 7f4330ff6700 1 --2- 172.21.15.3:0/276532056 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f430c06ac90 0x7f430c06d150 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f432c0cb880 tx=0x7f4320008040 comp rx=0 tx=0).stop 2024-05-08T21:22:22.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.733+0000 7f4330ff6700 1 -- 172.21.15.3:0/276532056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f432c06beb0 msgr2=0x7f432c144e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:22.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.733+0000 7f4330ff6700 1 --2- 172.21.15.3:0/276532056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f432c06beb0 0x7f432c144e80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f431c009130 tx=0x7f431c004750 comp rx=0 tx=0).stop 2024-05-08T21:22:22.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.733+0000 7f4330ff6700 1 -- 172.21.15.3:0/276532056 shutdown_connections 2024-05-08T21:22:22.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.733+0000 7f4330ff6700 1 --2- 172.21.15.3:0/276532056 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f432c15bbb0 0x7f432c06e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.733+0000 7f4330ff6700 1 --2- 172.21.15.3:0/276532056 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f430c06ac90 0x7f430c06d150 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.733+0000 7f4330ff6700 1 --2- 172.21.15.3:0/276532056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f432c06beb0 0x7f432c144e80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.733+0000 7f4330ff6700 1 --2- 172.21.15.3:0/276532056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432c158130 0x7f432c0ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.733+0000 7f4330ff6700 1 -- 172.21.15.3:0/276532056 >> 172.21.15.3:0/276532056 conn(0x7f432c071350 msgr2=0x7f432c14e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:22.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.733+0000 7f4330ff6700 1 -- 172.21.15.3:0/276532056 shutdown_connections 2024-05-08T21:22:22.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.733+0000 7f4330ff6700 1 -- 172.21.15.3:0/276532056 wait complete. 2024-05-08T21:22:22.748 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-08T21:21:49.638998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:21:51.024632+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:22:22.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:22:22.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.886+0000 7f6a3fc3a700 1 Processor -- start 2024-05-08T21:22:22.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.886+0000 7f6a3fc3a700 1 -- start start 2024-05-08T21:22:22.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.887+0000 7f6a3fc3a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6a380af8e0 0x7f6a3815aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.887+0000 7f6a3fc3a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a380b3d00 0x7f6a380b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.887+0000 7f6a3fc3a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a380b46d0 con 0x7f6a3815a7c0 2024-05-08T21:22:22.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.887+0000 7f6a3fc3a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a380aecf0 con 0x7f6a380af8e0 2024-05-08T21:22:22.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.887+0000 7f6a3fc3a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a380b4950 con 0x7f6a380b3d00 2024-05-08T21:22:22.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.887+0000 7f6a3d1d5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6a380af8e0 0x7f6a3815aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.887+0000 7f6a3d9d6700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6a3815a7c0 0x7f6a380af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59414/0 (socket says 172.21.15.3:59414) 2024-05-08T21:22:22.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.887+0000 7f6a3d9d6700 1 -- 172.21.15.3:0/2736917237 learned_addr learned my addr 172.21.15.3:0/2736917237 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:22.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.888+0000 7f6a3e1d7700 1 --2- 172.21.15.3:0/2736917237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a380b3d00 0x7f6a380b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.888+0000 7f6a3d1d5700 1 -- 172.21.15.3:0/2736917237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a380b3d00 msgr2=0x7f6a380b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:22.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.888+0000 7f6a3d1d5700 1 --2- 172.21.15.3:0/2736917237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a380b3d00 0x7f6a380b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.888+0000 7f6a3c9d4700 1 -- 172.21.15.3:0/2736917237 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724687322 0 0) 0x7f6a380b46d0 con 0x7f6a3815a7c0 2024-05-08T21:22:22.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.888+0000 7f6a3d1d5700 1 -- 172.21.15.3:0/2736917237 >> v1:172.21.15.3:6789/0 conn(0x7f6a3815a7c0 legacy=0x7f6a380af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:22.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.888+0000 7f6a3d1d5700 1 -- 172.21.15.3:0/2736917237 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3806de70 con 0x7f6a380af8e0 2024-05-08T21:22:22.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.888+0000 7f6a3d1d5700 1 --2- 172.21.15.3:0/2736917237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a380af8e0 0x7f6a3815aba0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6a28009840 tx=0x7f6a28009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a95c37e1255c268 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:22.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.888+0000 7f6a3c9d4700 1 -- 172.21.15.3:0/2736917237 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a28011830 con 0x7f6a380af8e0 2024-05-08T21:22:22.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.888+0000 7f6a3c9d4700 1 -- 172.21.15.3:0/2736917237 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a28011990 con 0x7f6a380af8e0 2024-05-08T21:22:22.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.888+0000 7f6a3c9d4700 1 -- 172.21.15.3:0/2736917237 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a28011d20 con 0x7f6a380af8e0 2024-05-08T21:22:22.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.889+0000 7f6a3fc3a700 1 -- 172.21.15.3:0/2736917237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a380af8e0 msgr2=0x7f6a3815aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:22.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.889+0000 7f6a3fc3a700 1 --2- 172.21.15.3:0/2736917237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a380af8e0 0x7f6a3815aba0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6a28009840 tx=0x7f6a28009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:22.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.889+0000 7f6a3fc3a700 1 -- 172.21.15.3:0/2736917237 shutdown_connections 2024-05-08T21:22:22.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.889+0000 7f6a3fc3a700 1 --2- 172.21.15.3:0/2736917237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a380b3d00 0x7f6a380b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.889+0000 7f6a3fc3a700 1 --2- 172.21.15.3:0/2736917237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a380af8e0 0x7f6a3815aba0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.889+0000 7f6a3fc3a700 1 -- 172.21.15.3:0/2736917237 >> 172.21.15.3:0/2736917237 conn(0x7f6a380b5810 msgr2=0x7f6a380b5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:22.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.889+0000 7f6a3fc3a700 1 -- 172.21.15.3:0/2736917237 shutdown_connections 2024-05-08T21:22:22.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.889+0000 7f6a3fc3a700 1 -- 172.21.15.3:0/2736917237 wait complete. 2024-05-08T21:22:22.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.890+0000 7f6a3fc3a700 1 Processor -- start 2024-05-08T21:22:22.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.891+0000 7f6a3fc3a700 1 -- start start 2024-05-08T21:22:22.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.891+0000 7f6a3fc3a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a380af8e0 0x7f6a381efa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.891+0000 7f6a3fc3a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a380b3d00 0x7f6a3814fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.891+0000 7f6a3fc3a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a3815a7c0 0x7f6a38150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.891+0000 7f6a3fc3a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a380b0670 con 0x7f6a380af8e0 2024-05-08T21:22:22.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.891+0000 7f6a3fc3a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a381466e0 con 0x7f6a380b3d00 2024-05-08T21:22:22.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.891+0000 7f6a3fc3a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a38150170 con 0x7f6a3815a7c0 2024-05-08T21:22:22.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.891+0000 7f6a3d9d6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a380af8e0 0x7f6a381efa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.891+0000 7f6a3d9d6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a380af8e0 0x7f6a381efa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43978/0 (socket says 172.21.15.3:43978) 2024-05-08T21:22:22.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.891+0000 7f6a3d9d6700 1 -- 172.21.15.3:0/2948208526 learned_addr learned my addr 172.21.15.3:0/2948208526 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:22.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.892+0000 7f6a3e1d7700 1 --2- 172.21.15.3:0/2948208526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a3815a7c0 0x7f6a38150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.892+0000 7f6a3d1d5700 1 --2- 172.21.15.3:0/2948208526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a380b3d00 0x7f6a3814fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.892+0000 7f6a3d9d6700 1 -- 172.21.15.3:0/2948208526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a3815a7c0 msgr2=0x7f6a38150b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:22.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.892+0000 7f6a3d9d6700 1 --2- 172.21.15.3:0/2948208526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a3815a7c0 0x7f6a38150b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.892+0000 7f6a3d9d6700 1 -- 172.21.15.3:0/2948208526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a380b3d00 msgr2=0x7f6a3814fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:22.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.892+0000 7f6a3d9d6700 1 --2- 172.21.15.3:0/2948208526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a380b3d00 0x7f6a3814fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:22.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.892+0000 7f6a3d9d6700 1 -- 172.21.15.3:0/2948208526 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a280094a0 con 0x7f6a380af8e0 2024-05-08T21:22:22.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.892+0000 7f6a3d9d6700 1 --2- 172.21.15.3:0/2948208526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a380af8e0 0x7f6a381efa20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400c550 tx=0x7f6a3400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:22.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.892+0000 7f6a2e7fc700 1 -- 172.21.15.3:0/2948208526 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a34004c80 con 0x7f6a380af8e0 2024-05-08T21:22:22.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.892+0000 7f6a2e7fc700 1 -- 172.21.15.3:0/2948208526 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a34004e00 con 0x7f6a380af8e0 2024-05-08T21:22:22.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.892+0000 7f6a3fc3a700 1 -- 172.21.15.3:0/2948208526 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a38150440 con 0x7f6a380af8e0 2024-05-08T21:22:22.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.892+0000 7f6a2e7fc700 1 -- 172.21.15.3:0/2948208526 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a34007310 con 0x7f6a380af8e0 2024-05-08T21:22:22.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.892+0000 7f6a3fc3a700 1 -- 172.21.15.3:0/2948208526 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a380b0290 con 0x7f6a380af8e0 2024-05-08T21:22:22.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.893+0000 7f6a3fc3a700 1 -- 172.21.15.3:0/2948208526 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a38074bc0 con 0x7f6a380af8e0 2024-05-08T21:22:22.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.894+0000 7f6a2e7fc700 1 -- 172.21.15.3:0/2948208526 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6a34007470 con 0x7f6a380af8e0 2024-05-08T21:22:22.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.894+0000 7f6a2e7fc700 1 --2- 172.21.15.3:0/2948208526 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6a1c06ac90 0x7f6a1c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:22.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.894+0000 7f6a2e7fc700 1 -- 172.21.15.3:0/2948208526 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a34088e90 con 0x7f6a380af8e0 2024-05-08T21:22:22.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.894+0000 7f6a3d1d5700 1 --2- 172.21.15.3:0/2948208526 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6a1c06ac90 0x7f6a1c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:22.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.895+0000 7f6a3d1d5700 1 --2- 172.21.15.3:0/2948208526 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6a1c06ac90 0x7f6a1c06d150 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6a280090b0 tx=0x7f6a2801e040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:22.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:22.899+0000 7f6a2e7fc700 1 -- 172.21.15.3:0/2948208526 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6a34054100 con 0x7f6a380af8e0 2024-05-08T21:22:23.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.116+0000 7f6a3fc3a700 1 -- 172.21.15.3:0/2948208526 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6a381576c0 con 0x7f6a380af8e0 2024-05-08T21:22:23.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.116+0000 7f6a2e7fc700 1 -- 172.21.15.3:0/2948208526 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6a34057720 con 0x7f6a380af8e0 2024-05-08T21:22:23.118 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:22:23.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.119+0000 7f6a3fc3a700 1 -- 172.21.15.3:0/2948208526 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6a1c06ac90 msgr2=0x7f6a1c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:23.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.119+0000 7f6a3fc3a700 1 --2- 172.21.15.3:0/2948208526 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6a1c06ac90 0x7f6a1c06d150 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6a280090b0 tx=0x7f6a2801e040 comp rx=0 tx=0).stop 2024-05-08T21:22:23.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.119+0000 7f6a3fc3a700 1 -- 172.21.15.3:0/2948208526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a380af8e0 msgr2=0x7f6a381efa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:23.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.119+0000 7f6a3fc3a700 1 --2- 172.21.15.3:0/2948208526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a380af8e0 0x7f6a381efa20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400c550 tx=0x7f6a3400c860 comp rx=0 tx=0).stop 2024-05-08T21:22:23.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.119+0000 7f6a3fc3a700 1 -- 172.21.15.3:0/2948208526 shutdown_connections 2024-05-08T21:22:23.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.119+0000 7f6a3fc3a700 1 --2- 172.21.15.3:0/2948208526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a3815a7c0 0x7f6a38150b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.119+0000 7f6a3fc3a700 1 --2- 172.21.15.3:0/2948208526 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6a1c06ac90 0x7f6a1c06d150 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.119+0000 7f6a3fc3a700 1 --2- 172.21.15.3:0/2948208526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a380af8e0 0x7f6a381efa20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.119+0000 7f6a3fc3a700 1 --2- 172.21.15.3:0/2948208526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a380b3d00 0x7f6a3814fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.119+0000 7f6a3fc3a700 1 -- 172.21.15.3:0/2948208526 >> 172.21.15.3:0/2948208526 conn(0x7f6a380b5810 msgr2=0x7f6a38156b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:23.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.119+0000 7f6a3fc3a700 1 -- 172.21.15.3:0/2948208526 shutdown_connections 2024-05-08T21:22:23.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.119+0000 7f6a3fc3a700 1 -- 172.21.15.3:0/2948208526 wait complete. 2024-05-08T21:22:23.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-08T21:22:23.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.271+0000 7f3cf52b4700 1 Processor -- start 2024-05-08T21:22:23.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.271+0000 7f3cf52b4700 1 -- start start 2024-05-08T21:22:23.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.272+0000 7f3cf52b4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3cf015dae0 0x7f3cf01614f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.272+0000 7f3cf52b4700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf0161e80 con 0x7f3cf015d700 2024-05-08T21:22:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.272+0000 7f3cf52b4700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf00b49b0 con 0x7f3cf0161a30 2024-05-08T21:22:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.272+0000 7f3cf52b4700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cf00b3da0 con 0x7f3cf015dae0 2024-05-08T21:22:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.272+0000 7f3cee7fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3cf015dae0 0x7f3cf01614f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.272+0000 7f3ceeffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3cf015d700 0x7f3cf00b3370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59430/0 (socket says 172.21.15.3:59430) 2024-05-08T21:22:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.272+0000 7f3ceeffd700 1 -- 172.21.15.3:0/558147259 learned_addr learned my addr 172.21.15.3:0/558147259 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.273+0000 7f3cee7fc700 1 -- 172.21.15.3:0/558147259 >> v1:172.21.15.186:6789/0 conn(0x7f3cf0161a30 legacy=0x7f3cf00b0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.273+0000 7f3cee7fc700 1 -- 172.21.15.3:0/558147259 >> v1:172.21.15.3:6789/0 conn(0x7f3cf015d700 legacy=0x7f3cf00b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.273+0000 7f3cedffb700 1 -- 172.21.15.3:0/558147259 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210889609 0 0) 0x7f3ce0001290 con 0x7f3cf015d700 2024-05-08T21:22:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.273+0000 7f3cee7fc700 1 -- 172.21.15.3:0/558147259 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf00b4080 con 0x7f3cf015dae0 2024-05-08T21:22:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.273+0000 7f3cee7fc700 1 --2- 172.21.15.3:0/558147259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cf015dae0 0x7f3cf01614f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4009840 tx=0x7f3ce4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f537b8c2e5aaff09 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.273+0000 7f3cedffb700 1 -- 172.21.15.3:0/558147259 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4011830 con 0x7f3cf015dae0 2024-05-08T21:22:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.273+0000 7f3cedffb700 1 -- 172.21.15.3:0/558147259 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce4011990 con 0x7f3cf015dae0 2024-05-08T21:22:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.273+0000 7f3cedffb700 1 -- 172.21.15.3:0/558147259 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4011d20 con 0x7f3cf015dae0 2024-05-08T21:22:23.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.274+0000 7f3cf52b4700 1 -- 172.21.15.3:0/558147259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cf015dae0 msgr2=0x7f3cf01614f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:23.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.274+0000 7f3cf52b4700 1 --2- 172.21.15.3:0/558147259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cf015dae0 0x7f3cf01614f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4009840 tx=0x7f3ce4009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:23.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.274+0000 7f3cf52b4700 1 -- 172.21.15.3:0/558147259 shutdown_connections 2024-05-08T21:22:23.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.274+0000 7f3cf52b4700 1 --2- 172.21.15.3:0/558147259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cf015dae0 0x7f3cf01614f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.274+0000 7f3cf52b4700 1 -- 172.21.15.3:0/558147259 >> 172.21.15.3:0/558147259 conn(0x7f3cf008afe0 msgr2=0x7f3cf00d3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:23.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.274+0000 7f3cf52b4700 1 -- 172.21.15.3:0/558147259 shutdown_connections 2024-05-08T21:22:23.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.274+0000 7f3cf52b4700 1 -- 172.21.15.3:0/558147259 wait complete. 2024-05-08T21:22:23.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.275+0000 7f3cf52b4700 1 Processor -- start 2024-05-08T21:22:23.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.275+0000 7f3cf52b4700 1 -- start start 2024-05-08T21:22:23.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3cf52b4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cf015d700 0x7f3cf0158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:23.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3cf52b4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cf015dae0 0x7f3cf00c7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:23.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3cf52b4700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cf0161a30 0x7f3cf00b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:23.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3cf52b4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf0159460 con 0x7f3cf015dae0 2024-05-08T21:22:23.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3cf52b4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf00c9df0 con 0x7f3cf015d700 2024-05-08T21:22:23.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3cf52b4700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf00c7680 con 0x7f3cf0161a30 2024-05-08T21:22:23.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3cee7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cf015dae0 0x7f3cf00c7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:23.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3cee7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cf015dae0 0x7f3cf00c7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43984/0 (socket says 172.21.15.3:43984) 2024-05-08T21:22:23.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3cee7fc700 1 -- 172.21.15.3:0/289673706 learned_addr learned my addr 172.21.15.3:0/289673706 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:23.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3ceeffd700 1 --2- 172.21.15.3:0/289673706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cf015d700 0x7f3cf0158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:23.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3cee7fc700 1 -- 172.21.15.3:0/289673706 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cf0161a30 msgr2=0x7f3cf00b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:23.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3cef7fe700 1 --2- 172.21.15.3:0/289673706 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cf0161a30 0x7f3cf00b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:23.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3cee7fc700 1 --2- 172.21.15.3:0/289673706 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cf0161a30 0x7f3cf00b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3cee7fc700 1 -- 172.21.15.3:0/289673706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cf015d700 msgr2=0x7f3cf0158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:23.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3cee7fc700 1 --2- 172.21.15.3:0/289673706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cf015d700 0x7f3cf0158f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3cee7fc700 1 -- 172.21.15.3:0/289673706 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce40094a0 con 0x7f3cf015dae0 2024-05-08T21:22:23.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3ceeffd700 1 --2- 172.21.15.3:0/289673706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cf015d700 0x7f3cf0158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:23.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.276+0000 7f3cef7fe700 1 --2- 172.21.15.3:0/289673706 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cf0161a30 0x7f3cf00b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:23.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.277+0000 7f3cee7fc700 1 --2- 172.21.15.3:0/289673706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cf015dae0 0x7f3cf00c7140 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40090b0 tx=0x7f3ce4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:23.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.277+0000 7f3cd77fe700 1 -- 172.21.15.3:0/289673706 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce40036a0 con 0x7f3cf015dae0 2024-05-08T21:22:23.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.277+0000 7f3cd77fe700 1 -- 172.21.15.3:0/289673706 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce401f070 con 0x7f3cf015dae0 2024-05-08T21:22:23.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.277+0000 7f3cf52b4700 1 -- 172.21.15.3:0/289673706 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf00c7920 con 0x7f3cf015dae0 2024-05-08T21:22:23.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.277+0000 7f3cd77fe700 1 -- 172.21.15.3:0/289673706 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4003a00 con 0x7f3cf015dae0 2024-05-08T21:22:23.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.277+0000 7f3cf52b4700 1 -- 172.21.15.3:0/289673706 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf00c7ee0 con 0x7f3cf015dae0 2024-05-08T21:22:23.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.278+0000 7f3cf52b4700 1 -- 172.21.15.3:0/289673706 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf00b3a00 con 0x7f3cf015dae0 2024-05-08T21:22:23.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.278+0000 7f3cd77fe700 1 -- 172.21.15.3:0/289673706 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3ce4003b60 con 0x7f3cf015dae0 2024-05-08T21:22:23.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.279+0000 7f3cd77fe700 1 --2- 172.21.15.3:0/289673706 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3cdc06ac90 0x7f3cdc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:23.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.279+0000 7f3cd77fe700 1 -- 172.21.15.3:0/289673706 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ce4021070 con 0x7f3cf015dae0 2024-05-08T21:22:23.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.279+0000 7f3ceeffd700 1 --2- 172.21.15.3:0/289673706 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3cdc06ac90 0x7f3cdc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:23.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.280+0000 7f3ceeffd700 1 --2- 172.21.15.3:0/289673706 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3cdc06ac90 0x7f3cdc06d150 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0001760 tx=0x7f3ce0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:23.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.284+0000 7f3cd77fe700 1 -- 172.21.15.3:0/289673706 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f3ce40559a0 con 0x7f3cf015dae0 2024-05-08T21:22:23.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.501+0000 7f3cf52b4700 1 -- 172.21.15.3:0/289673706 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3cf0074bc0 con 0x7f3cf015dae0 2024-05-08T21:22:23.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.501+0000 7f3cd77fe700 1 -- 172.21.15.3:0/289673706 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3ce40150a0 con 0x7f3cf015dae0 2024-05-08T21:22:23.503 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:22:23.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.503+0000 7f3cf52b4700 1 -- 172.21.15.3:0/289673706 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3cdc06ac90 msgr2=0x7f3cdc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:23.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.503+0000 7f3cf52b4700 1 --2- 172.21.15.3:0/289673706 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3cdc06ac90 0x7f3cdc06d150 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0001760 tx=0x7f3ce0009040 comp rx=0 tx=0).stop 2024-05-08T21:22:23.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.503+0000 7f3cf52b4700 1 -- 172.21.15.3:0/289673706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cf015dae0 msgr2=0x7f3cf00c7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:23.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.503+0000 7f3cf52b4700 1 --2- 172.21.15.3:0/289673706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cf015dae0 0x7f3cf00c7140 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40090b0 tx=0x7f3ce4004750 comp rx=0 tx=0).stop 2024-05-08T21:22:23.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.503+0000 7f3cf52b4700 1 -- 172.21.15.3:0/289673706 shutdown_connections 2024-05-08T21:22:23.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.503+0000 7f3cf52b4700 1 --2- 172.21.15.3:0/289673706 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cf0161a30 0x7f3cf00b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.503+0000 7f3cf52b4700 1 --2- 172.21.15.3:0/289673706 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3cdc06ac90 0x7f3cdc06d150 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.503+0000 7f3cf52b4700 1 --2- 172.21.15.3:0/289673706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cf015d700 0x7f3cf0158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.503+0000 7f3cf52b4700 1 --2- 172.21.15.3:0/289673706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cf015dae0 0x7f3cf00c7140 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.503+0000 7f3cf52b4700 1 -- 172.21.15.3:0/289673706 >> 172.21.15.3:0/289673706 conn(0x7f3cf008afe0 msgr2=0x7f3cf00cf660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:23.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.503+0000 7f3cf52b4700 1 -- 172.21.15.3:0/289673706 shutdown_connections 2024-05-08T21:22:23.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.503+0000 7f3cf52b4700 1 -- 172.21.15.3:0/289673706 wait complete. 2024-05-08T21:22:23.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:22:23.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.657+0000 7fb4d10f6700 1 Processor -- start 2024-05-08T21:22:23.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.657+0000 7fb4d10f6700 1 -- start start 2024-05-08T21:22:23.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.658+0000 7fb4d10f6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4cc15da60 0x7fb4cc0d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:23.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.658+0000 7fb4d10f6700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4cc06d010 con 0x7fb4cc09a160 2024-05-08T21:22:23.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.658+0000 7fb4d10f6700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4cc15de40 con 0x7fb4cc09a790 2024-05-08T21:22:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.658+0000 7fb4d10f6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc0d0880 con 0x7fb4cc15da60 2024-05-08T21:22:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.658+0000 7fb4cb59e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb4cc09a160 0x7fb4cc0cc720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59446/0 (socket says 172.21.15.3:59446) 2024-05-08T21:22:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.658+0000 7fb4cb59e700 1 -- 172.21.15.3:0/4191225452 learned_addr learned my addr 172.21.15.3:0/4191225452 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.658+0000 7fb4cad9d700 1 --2- 172.21.15.3:0/4191225452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4cc15da60 0x7fb4cc0d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.658+0000 7fb4c9d9b700 1 -- 172.21.15.3:0/4191225452 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729915599 0 0) 0x7fb4cc06d010 con 0x7fb4cc09a160 2024-05-08T21:22:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.658+0000 7fb4c9d9b700 1 -- 172.21.15.3:0/4191225452 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4b40035b0 con 0x7fb4cc09a160 2024-05-08T21:22:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4c9d9b700 1 -- 172.21.15.3:0/4191225452 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 747185465 0 0) 0x7fb4b40035b0 con 0x7fb4cc09a160 2024-05-08T21:22:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4c9d9b700 1 -- 172.21.15.3:0/4191225452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4cc15da60 msgr2=0x7fb4cc0d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4c9d9b700 1 --2- 172.21.15.3:0/4191225452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4cc15da60 0x7fb4cc0d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4c9d9b700 1 -- 172.21.15.3:0/4191225452 >> v1:172.21.15.186:6789/0 conn(0x7fb4cc09a790 legacy=0x7fb4cc099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4c9d9b700 1 -- 172.21.15.3:0/4191225452 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc0d0b20 con 0x7fb4cc09a160 2024-05-08T21:22:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4c9d9b700 1 -- 172.21.15.3:0/4191225452 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7fb4c40019f0 con 0x7fb4cc09a160 2024-05-08T21:22:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4c9d9b700 1 -- 172.21.15.3:0/4191225452 >> v1:172.21.15.3:6789/0 conn(0x7fb4cc09a160 legacy=0x7fb4cc0cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4c9d9b700 1 --2- 172.21.15.3:0/4191225452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4cc15da60 0x7fb4b4003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:23.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4c9d9b700 1 --2- 172.21.15.3:0/4191225452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4b4003f70 0x7fb4b4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:23.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4c9d9b700 1 --2- 172.21.15.3:0/4191225452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4b4004d90 0x7fb4b4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:23.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4c9d9b700 1 -- 172.21.15.3:0/4191225452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc0d0880 con 0x7fb4b4003f70 2024-05-08T21:22:23.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4c9d9b700 1 -- 172.21.15.3:0/4191225452 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b400ba70 con 0x7fb4b4004d90 2024-05-08T21:22:23.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4c9d9b700 1 -- 172.21.15.3:0/4191225452 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b400bc30 con 0x7fb4cc15da60 2024-05-08T21:22:23.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4ca59c700 1 --2- 172.21.15.3:0/4191225452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4b4003f70 0x7fb4b4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:23.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4cad9d700 1 --2- 172.21.15.3:0/4191225452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4cc15da60 0x7fb4b4003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:23.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4cb59e700 1 --2- 172.21.15.3:0/4191225452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4b4004d90 0x7fb4b4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:23.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4ca59c700 1 -- 172.21.15.3:0/4191225452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4cc15da60 msgr2=0x7fb4b4003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:23.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4ca59c700 1 --2- 172.21.15.3:0/4191225452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4cc15da60 0x7fb4b4003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4ca59c700 1 -- 172.21.15.3:0/4191225452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4b4004d90 msgr2=0x7fb4b4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4ca59c700 1 --2- 172.21.15.3:0/4191225452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4b4004d90 0x7fb4b4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4ca59c700 1 -- 172.21.15.3:0/4191225452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b40035b0 con 0x7fb4b4003f70 2024-05-08T21:22:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4ca59c700 1 --2- 172.21.15.3:0/4191225452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4b4003f70 0x7fb4b4004850 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00018d0 tx=0x7fb4b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea272efbbc9e0b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.659+0000 7fb4c9d9b700 1 -- 172.21.15.3:0/4191225452 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0012620 con 0x7fb4b4003f70 2024-05-08T21:22:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.660+0000 7fb4c9d9b700 1 -- 172.21.15.3:0/4191225452 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c0012780 con 0x7fb4b4003f70 2024-05-08T21:22:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.660+0000 7fb4c9d9b700 1 -- 172.21.15.3:0/4191225452 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0012b30 con 0x7fb4b4003f70 2024-05-08T21:22:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.660+0000 7fb4d10f6700 1 -- 172.21.15.3:0/4191225452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4b4003f70 msgr2=0x7fb4b4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.660+0000 7fb4d10f6700 1 --2- 172.21.15.3:0/4191225452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4b4003f70 0x7fb4b4004850 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00018d0 tx=0x7fb4b4005880 comp rx=0 tx=0).stop 2024-05-08T21:22:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.660+0000 7fb4cb59e700 1 -- 172.21.15.3:0/4191225452 reap_dead start 2024-05-08T21:22:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.660+0000 7fb4d10f6700 1 -- 172.21.15.3:0/4191225452 shutdown_connections 2024-05-08T21:22:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.660+0000 7fb4d10f6700 1 -- 172.21.15.3:0/4191225452 >> 172.21.15.3:0/4191225452 conn(0x7fb4cc0b2f60 msgr2=0x7fb4cc0b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.660+0000 7fb4d10f6700 1 -- 172.21.15.3:0/4191225452 shutdown_connections 2024-05-08T21:22:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.660+0000 7fb4d10f6700 1 -- 172.21.15.3:0/4191225452 wait complete. 2024-05-08T21:22:23.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.662+0000 7fb4d10f6700 1 Processor -- start 2024-05-08T21:22:23.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.662+0000 7fb4d10f6700 1 -- start start 2024-05-08T21:22:23.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4d10f6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4b4003f70 0x7fb4cc15f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:23.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4d10f6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4cc0c6e70 0x7fb4cc0c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:23.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4d10f6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4cc1f4270 0x7fb4cc1f4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:23.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4d10f6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc15f9f0 con 0x7fb4b4003f70 2024-05-08T21:22:23.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4d10f6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc15e410 con 0x7fb4cc0c6e70 2024-05-08T21:22:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4d10f6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc15e6f0 con 0x7fb4cc1f4270 2024-05-08T21:22:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4cad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4b4003f70 0x7fb4cc15f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4cad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4b4003f70 0x7fb4cc15f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43996/0 (socket says 172.21.15.3:43996) 2024-05-08T21:22:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4cad9d700 1 -- 172.21.15.3:0/3734098740 learned_addr learned my addr 172.21.15.3:0/3734098740 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4ca59c700 1 --2- 172.21.15.3:0/3734098740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4cc0c6e70 0x7fb4cc0c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4cb59e700 1 --2- 172.21.15.3:0/3734098740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4cc1f4270 0x7fb4cc1f4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4cad9d700 1 -- 172.21.15.3:0/3734098740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4cc1f4270 msgr2=0x7fb4cc1f4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4cad9d700 1 --2- 172.21.15.3:0/3734098740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4cc1f4270 0x7fb4cc1f4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4cad9d700 1 -- 172.21.15.3:0/3734098740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4cc0c6e70 msgr2=0x7fb4cc0c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4cad9d700 1 --2- 172.21.15.3:0/3734098740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4cc0c6e70 0x7fb4cc0c72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4cad9d700 1 -- 172.21.15.3:0/3734098740 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc15e9d0 con 0x7fb4b4003f70 2024-05-08T21:22:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4cad9d700 1 --2- 172.21.15.3:0/3734098740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4b4003f70 0x7fb4cc15f4b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc011aa0 tx=0x7fb4bc011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.663+0000 7fb4b37fe700 1 -- 172.21.15.3:0/3734098740 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc011ed0 con 0x7fb4b4003f70 2024-05-08T21:22:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.664+0000 7fb4b37fe700 1 -- 172.21.15.3:0/3734098740 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4bc00f940 con 0x7fb4b4003f70 2024-05-08T21:22:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.664+0000 7fb4b37fe700 1 -- 172.21.15.3:0/3734098740 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc00fcf0 con 0x7fb4b4003f70 2024-05-08T21:22:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.664+0000 7fb4d10f6700 1 -- 172.21.15.3:0/3734098740 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4cc15ecb0 con 0x7fb4b4003f70 2024-05-08T21:22:23.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.664+0000 7fb4d10f6700 1 -- 172.21.15.3:0/3734098740 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4cc15e030 con 0x7fb4b4003f70 2024-05-08T21:22:23.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.665+0000 7fb4d10f6700 1 -- 172.21.15.3:0/3734098740 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4cc074bc0 con 0x7fb4b4003f70 2024-05-08T21:22:23.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.665+0000 7fb4b37fe700 1 -- 172.21.15.3:0/3734098740 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb4bc014440 con 0x7fb4b4003f70 2024-05-08T21:22:23.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.666+0000 7fb4b37fe700 1 --2- 172.21.15.3:0/3734098740 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4b806ac90 0x7fb4b806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:23.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.666+0000 7fb4b37fe700 1 -- 172.21.15.3:0/3734098740 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4bc089930 con 0x7fb4b4003f70 2024-05-08T21:22:23.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.666+0000 7fb4ca59c700 1 --2- 172.21.15.3:0/3734098740 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4b806ac90 0x7fb4b806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:23.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.667+0000 7fb4ca59c700 1 --2- 172.21.15.3:0/3734098740 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4b806ac90 0x7fb4b806d150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0000c00 tx=0x7fb4c0005040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:23.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.670+0000 7fb4b37fe700 1 -- 172.21.15.3:0/3734098740 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb4bc054bd0 con 0x7fb4b4003f70 2024-05-08T21:22:23.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.889+0000 7fb4d10f6700 1 -- 172.21.15.3:0/3734098740 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb4cc0d0310 con 0x7fb4b4003f70 2024-05-08T21:22:23.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.889+0000 7fb4b37fe700 1 -- 172.21.15.3:0/3734098740 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb4bc022020 con 0x7fb4b4003f70 2024-05-08T21:22:23.891 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:22:23.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.892+0000 7fb4d10f6700 1 -- 172.21.15.3:0/3734098740 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4b806ac90 msgr2=0x7fb4b806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:23.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.892+0000 7fb4d10f6700 1 --2- 172.21.15.3:0/3734098740 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4b806ac90 0x7fb4b806d150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0000c00 tx=0x7fb4c0005040 comp rx=0 tx=0).stop 2024-05-08T21:22:23.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.892+0000 7fb4d10f6700 1 -- 172.21.15.3:0/3734098740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4b4003f70 msgr2=0x7fb4cc15f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:23.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.892+0000 7fb4d10f6700 1 --2- 172.21.15.3:0/3734098740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4b4003f70 0x7fb4cc15f4b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc011aa0 tx=0x7fb4bc011db0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.892+0000 7fb4d10f6700 1 -- 172.21.15.3:0/3734098740 shutdown_connections 2024-05-08T21:22:23.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.892+0000 7fb4d10f6700 1 --2- 172.21.15.3:0/3734098740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4cc1f4270 0x7fb4cc1f4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.892+0000 7fb4d10f6700 1 --2- 172.21.15.3:0/3734098740 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4b806ac90 0x7fb4b806d150 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.892+0000 7fb4d10f6700 1 --2- 172.21.15.3:0/3734098740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4b4003f70 0x7fb4cc15f4b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.892+0000 7fb4d10f6700 1 --2- 172.21.15.3:0/3734098740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4cc0c6e70 0x7fb4cc0c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:23.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.892+0000 7fb4d10f6700 1 -- 172.21.15.3:0/3734098740 >> 172.21.15.3:0/3734098740 conn(0x7fb4cc0b2f60 msgr2=0x7fb4cc0cdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:23.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.892+0000 7fb4d10f6700 1 -- 172.21.15.3:0/3734098740 shutdown_connections 2024-05-08T21:22:23.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:23.892+0000 7fb4d10f6700 1 -- 172.21.15.3:0/3734098740 wait complete. 2024-05-08T21:22:23.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple' 2024-05-08T21:22:24.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.041+0000 7f189e0fb700 1 Processor -- start 2024-05-08T21:22:24.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.041+0000 7f189e0fb700 1 -- start start 2024-05-08T21:22:24.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.042+0000 7f189e0fb700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f189815f230 0x7f18980cc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.042+0000 7f189e0fb700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18980cd3b0 con 0x7f18980aef20 2024-05-08T21:22:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.042+0000 7f189e0fb700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18980cce90 con 0x7f18980cca70 2024-05-08T21:22:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.042+0000 7f189e0fb700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1898141df0 con 0x7f189815f230 2024-05-08T21:22:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.042+0000 7f18977fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f18980aef20 0x7f189815c670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59462/0 (socket says 172.21.15.3:59462) 2024-05-08T21:22:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.042+0000 7f18977fe700 1 -- 172.21.15.3:0/2148282471 learned_addr learned my addr 172.21.15.3:0/2148282471 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f1896ffd700 1 --2- 172.21.15.3:0/2148282471 >> v2:172.21.15.186:3300/0 conn(0x7f189815f230 0x7f18980cc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f18967fc700 1 -- 172.21.15.3:0/2148282471 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935326150 0 0) 0x7f18980cd3b0 con 0x7f18980aef20 2024-05-08T21:22:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f18967fc700 1 -- 172.21.15.3:0/2148282471 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18800035b0 con 0x7f18980aef20 2024-05-08T21:22:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f18967fc700 1 -- 172.21.15.3:0/2148282471 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 339061593 0 0) 0x7f18800035b0 con 0x7f18980aef20 2024-05-08T21:22:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f18967fc700 1 -- 172.21.15.3:0/2148282471 >> v2:172.21.15.186:3300/0 conn(0x7f189815f230 msgr2=0x7f18980cc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f18967fc700 1 --2- 172.21.15.3:0/2148282471 >> v2:172.21.15.186:3300/0 conn(0x7f189815f230 0x7f18980cc530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f18967fc700 1 -- 172.21.15.3:0/2148282471 >> v1:172.21.15.186:6789/0 conn(0x7f18980cca70 legacy=0x7f189806ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f18967fc700 1 -- 172.21.15.3:0/2148282471 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18980af300 con 0x7f18980aef20 2024-05-08T21:22:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f18967fc700 1 -- 172.21.15.3:0/2148282471 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f1888001670 con 0x7f18980aef20 2024-05-08T21:22:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f18967fc700 1 -- 172.21.15.3:0/2148282471 >> v1:172.21.15.3:6789/0 conn(0x7f18980aef20 legacy=0x7f189815c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f18967fc700 1 --2- 172.21.15.3:0/2148282471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1880003a00 0x7f1880003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f18967fc700 1 --2- 172.21.15.3:0/2148282471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1880004850 0x7f1880004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f18967fc700 1 --2- 172.21.15.3:0/2148282471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1880005210 0x7f188000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f18967fc700 1 -- 172.21.15.3:0/2148282471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1898141df0 con 0x7f1880003a00 2024-05-08T21:22:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f18967fc700 1 -- 172.21.15.3:0/2148282471 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188000c1b0 con 0x7f1880005210 2024-05-08T21:22:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f18967fc700 1 -- 172.21.15.3:0/2148282471 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f188000c370 con 0x7f1880004850 2024-05-08T21:22:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f18977fe700 1 --2- 172.21.15.3:0/2148282471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1880003a00 0x7f1880003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f1896ffd700 1 --2- 172.21.15.3:0/2148282471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1880004850 0x7f1880004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.043+0000 7f1897fff700 1 --2- 172.21.15.3:0/2148282471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1880005210 0x7f188000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.044+0000 7f18977fe700 1 -- 172.21.15.3:0/2148282471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1880004850 msgr2=0x7f1880004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.044+0000 7f18977fe700 1 --2- 172.21.15.3:0/2148282471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1880004850 0x7f1880004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.044+0000 7f18977fe700 1 -- 172.21.15.3:0/2148282471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1880005210 msgr2=0x7f188000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.044+0000 7f18977fe700 1 --2- 172.21.15.3:0/2148282471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1880005210 0x7f188000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.044+0000 7f18977fe700 1 -- 172.21.15.3:0/2148282471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18800035b0 con 0x7f1880003a00 2024-05-08T21:22:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.044+0000 7f1897fff700 1 -- 172.21.15.3:0/2148282471 reap_dead start 2024-05-08T21:22:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.044+0000 7f18977fe700 1 --2- 172.21.15.3:0/2148282471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1880003a00 0x7f1880003e20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f188800c240 tx=0x7f1880005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f869b3bae58a7600 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.044+0000 7f18967fc700 1 -- 172.21.15.3:0/2148282471 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1888007c10 con 0x7f1880003a00 2024-05-08T21:22:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.044+0000 7f18967fc700 1 -- 172.21.15.3:0/2148282471 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188800ce00 con 0x7f1880003a00 2024-05-08T21:22:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.044+0000 7f18967fc700 1 -- 172.21.15.3:0/2148282471 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1888005ba0 con 0x7f1880003a00 2024-05-08T21:22:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.044+0000 7f189e0fb700 1 -- 172.21.15.3:0/2148282471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1880003a00 msgr2=0x7f1880003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.044+0000 7f189e0fb700 1 --2- 172.21.15.3:0/2148282471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1880003a00 0x7f1880003e20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f188800c240 tx=0x7f1880005880 comp rx=0 tx=0).stop 2024-05-08T21:22:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.045+0000 7f189e0fb700 1 -- 172.21.15.3:0/2148282471 shutdown_connections 2024-05-08T21:22:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.045+0000 7f189e0fb700 1 --2- 172.21.15.3:0/2148282471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1880003a00 0x7f1880003e20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.045+0000 7f189e0fb700 1 -- 172.21.15.3:0/2148282471 >> 172.21.15.3:0/2148282471 conn(0x7f18980cfce0 msgr2=0x7f18980d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.045+0000 7f189e0fb700 1 -- 172.21.15.3:0/2148282471 shutdown_connections 2024-05-08T21:22:24.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.045+0000 7f189e0fb700 1 -- 172.21.15.3:0/2148282471 wait complete. 2024-05-08T21:22:24.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.046+0000 7f189e0fb700 1 Processor -- start 2024-05-08T21:22:24.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.046+0000 7f189e0fb700 1 -- start start 2024-05-08T21:22:24.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f189e0fb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1880003a00 0x7f1898138a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f189e0fb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1898155d60 0x7f18981f6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f189e0fb700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18981f7400 0x7f189806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f189e0fb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1898139680 con 0x7f1898155d60 2024-05-08T21:22:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f189e0fb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189813d5b0 con 0x7f1880003a00 2024-05-08T21:22:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f189e0fb700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1898138f40 con 0x7f18981f7400 2024-05-08T21:22:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f1896ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1898155d60 0x7f18981f6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f1896ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1898155d60 0x7f18981f6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44008/0 (socket says 172.21.15.3:44008) 2024-05-08T21:22:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f1896ffd700 1 -- 172.21.15.3:0/857020271 learned_addr learned my addr 172.21.15.3:0/857020271 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f18977fe700 1 --2- 172.21.15.3:0/857020271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1880003a00 0x7f1898138a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f1897fff700 1 --2- 172.21.15.3:0/857020271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18981f7400 0x7f189806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f1896ffd700 1 -- 172.21.15.3:0/857020271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18981f7400 msgr2=0x7f189806ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f1896ffd700 1 --2- 172.21.15.3:0/857020271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18981f7400 0x7f189806ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f1896ffd700 1 -- 172.21.15.3:0/857020271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1880003a00 msgr2=0x7f1898138a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f1896ffd700 1 --2- 172.21.15.3:0/857020271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1880003a00 0x7f1898138a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f1896ffd700 1 -- 172.21.15.3:0/857020271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18981f7880 con 0x7f1898155d60 2024-05-08T21:22:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.047+0000 7f1897fff700 1 --2- 172.21.15.3:0/857020271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18981f7400 0x7f189806ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.048+0000 7f1896ffd700 1 --2- 172.21.15.3:0/857020271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1898155d60 0x7f18981f6ec0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f188c011aa0 tx=0x7f188c011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.048+0000 7f187ffff700 1 -- 172.21.15.3:0/857020271 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c00f9e0 con 0x7f1898155d60 2024-05-08T21:22:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.048+0000 7f187ffff700 1 -- 172.21.15.3:0/857020271 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188c00fb40 con 0x7f1898155d60 2024-05-08T21:22:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.048+0000 7f187ffff700 1 -- 172.21.15.3:0/857020271 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c014440 con 0x7f1898155d60 2024-05-08T21:22:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.048+0000 7f189e0fb700 1 -- 172.21.15.3:0/857020271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18981f7b60 con 0x7f1898155d60 2024-05-08T21:22:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.048+0000 7f189e0fb700 1 -- 172.21.15.3:0/857020271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f189806f380 con 0x7f1898155d60 2024-05-08T21:22:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.049+0000 7f189e0fb700 1 -- 172.21.15.3:0/857020271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18981f7da0 con 0x7f1898155d60 2024-05-08T21:22:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.049+0000 7f187ffff700 1 -- 172.21.15.3:0/857020271 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f188c0145a0 con 0x7f1898155d60 2024-05-08T21:22:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.050+0000 7f187ffff700 1 --2- 172.21.15.3:0/857020271 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f188406ac90 0x7f188406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.050+0000 7f187ffff700 1 -- 172.21.15.3:0/857020271 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f188c089ab0 con 0x7f1898155d60 2024-05-08T21:22:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.050+0000 7f18977fe700 1 --2- 172.21.15.3:0/857020271 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f188406ac90 0x7f188406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.051+0000 7f18977fe700 1 --2- 172.21.15.3:0/857020271 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f188406ac90 0x7f188406d150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1888000c00 tx=0x7f1888007000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:24.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.054+0000 7f187ffff700 1 -- 172.21.15.3:0/857020271 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f18981f7da0 con 0x7f1898155d60 2024-05-08T21:22:24.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.368+0000 7f189e0fb700 1 -- 172.21.15.3:0/857020271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple"]} v 0) v1 -- 0x7f18981f7da0 con 0x7f1898155d60 2024-05-08T21:22:24.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.623+0000 7f187ffff700 1 -- 172.21.15.3:0/857020271 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple"]}]=0 v53) v1 ==== 150+0+0 (secure 0 0 0) 0x7f18981f7da0 con 0x7f1898155d60 2024-05-08T21:22:24.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.626+0000 7f189e0fb700 1 -- 172.21.15.3:0/857020271 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f188406ac90 msgr2=0x7f188406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:24.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.626+0000 7f189e0fb700 1 --2- 172.21.15.3:0/857020271 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f188406ac90 0x7f188406d150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1888000c00 tx=0x7f1888007000 comp rx=0 tx=0).stop 2024-05-08T21:22:24.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.626+0000 7f189e0fb700 1 -- 172.21.15.3:0/857020271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1898155d60 msgr2=0x7f18981f6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:24.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.626+0000 7f189e0fb700 1 --2- 172.21.15.3:0/857020271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1898155d60 0x7f18981f6ec0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f188c011aa0 tx=0x7f188c011e60 comp rx=0 tx=0).stop 2024-05-08T21:22:24.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.626+0000 7f189e0fb700 1 -- 172.21.15.3:0/857020271 shutdown_connections 2024-05-08T21:22:24.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.626+0000 7f189e0fb700 1 --2- 172.21.15.3:0/857020271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18981f7400 0x7f189806ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:24.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.626+0000 7f189e0fb700 1 --2- 172.21.15.3:0/857020271 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f188406ac90 0x7f188406d150 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:24.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.626+0000 7f189e0fb700 1 --2- 172.21.15.3:0/857020271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1880003a00 0x7f1898138a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:24.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.626+0000 7f189e0fb700 1 --2- 172.21.15.3:0/857020271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1898155d60 0x7f18981f6ec0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:24.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.627+0000 7f189e0fb700 1 -- 172.21.15.3:0/857020271 >> 172.21.15.3:0/857020271 conn(0x7f18980cfce0 msgr2=0x7f189813fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:24.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.627+0000 7f189e0fb700 1 -- 172.21.15.3:0/857020271 shutdown_connections 2024-05-08T21:22:24.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.627+0000 7f189e0fb700 1 -- 172.21.15.3:0/857020271 wait complete. 2024-05-08T21:22:24.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-08T21:22:24.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.785+0000 7f1a8e5e3700 1 Processor -- start 2024-05-08T21:22:24.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.786+0000 7f1a8e5e3700 1 -- start start 2024-05-08T21:22:24.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.786+0000 7f1a8e5e3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1a8815b4e0 0x7f1a880b1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:24.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.786+0000 7f1a8e5e3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a880d1910 0x7f1a880cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:24.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.786+0000 7f1a8e5e3700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a880cf560 con 0x7f1a8815b8c0 2024-05-08T21:22:24.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.786+0000 7f1a8e5e3700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a880cefe0 con 0x7f1a8815b4e0 2024-05-08T21:22:24.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.786+0000 7f1a8e5e3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a880ccff0 con 0x7f1a880d1910 2024-05-08T21:22:24.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.786+0000 7f1a87fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1a8815b4e0 0x7f1a880b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:24.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.787+0000 7f1a877fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1a8815b8c0 0x7f1a880d1200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59464/0 (socket says 172.21.15.3:59464) 2024-05-08T21:22:24.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.787+0000 7f1a877fe700 1 -- 172.21.15.3:0/860128823 learned_addr learned my addr 172.21.15.3:0/860128823 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:24.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.787+0000 7f1a8cb80700 1 --2- 172.21.15.3:0/860128823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a880d1910 0x7f1a880cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:24.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.787+0000 7f1a87fff700 1 -- 172.21.15.3:0/860128823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a880d1910 msgr2=0x7f1a880cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:24.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.787+0000 7f1a87fff700 1 --2- 172.21.15.3:0/860128823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a880d1910 0x7f1a880cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:24.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.787+0000 7f1a87fff700 1 -- 172.21.15.3:0/860128823 >> v1:172.21.15.3:6789/0 conn(0x7f1a8815b8c0 legacy=0x7f1a880d1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:24.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.787+0000 7f1a87fff700 1 -- 172.21.15.3:0/860128823 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a880cd2d0 con 0x7f1a8815b4e0 2024-05-08T21:22:24.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.787+0000 7f1a8cb80700 1 --2- 172.21.15.3:0/860128823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a880d1910 0x7f1a880cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:24.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.787+0000 7f1a87fff700 1 --2- 172.21.15.3:0/860128823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a8815b4e0 0x7f1a880b1200 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1a78009840 tx=0x7f1a78009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e4618dd1f35959 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:24.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.787+0000 7f1a86ffd700 1 -- 172.21.15.3:0/860128823 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a78011830 con 0x7f1a8815b4e0 2024-05-08T21:22:24.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.787+0000 7f1a86ffd700 1 -- 172.21.15.3:0/860128823 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a78011990 con 0x7f1a8815b4e0 2024-05-08T21:22:24.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.787+0000 7f1a86ffd700 1 -- 172.21.15.3:0/860128823 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a78011d20 con 0x7f1a8815b4e0 2024-05-08T21:22:24.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.788+0000 7f1a8e5e3700 1 -- 172.21.15.3:0/860128823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a8815b4e0 msgr2=0x7f1a880b1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:24.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.788+0000 7f1a8e5e3700 1 --2- 172.21.15.3:0/860128823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a8815b4e0 0x7f1a880b1200 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1a78009840 tx=0x7f1a78009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:24.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.788+0000 7f1a8e5e3700 1 -- 172.21.15.3:0/860128823 shutdown_connections 2024-05-08T21:22:24.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.788+0000 7f1a8e5e3700 1 --2- 172.21.15.3:0/860128823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a880d1910 0x7f1a880cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:24.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.788+0000 7f1a8e5e3700 1 --2- 172.21.15.3:0/860128823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a8815b4e0 0x7f1a880b1200 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:24.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.788+0000 7f1a8e5e3700 1 -- 172.21.15.3:0/860128823 >> 172.21.15.3:0/860128823 conn(0x7f1a8808afe0 msgr2=0x7f1a8808b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:24.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.788+0000 7f1a8e5e3700 1 -- 172.21.15.3:0/860128823 shutdown_connections 2024-05-08T21:22:24.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.788+0000 7f1a8e5e3700 1 -- 172.21.15.3:0/860128823 wait complete. 2024-05-08T21:22:24.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.789+0000 7f1a8e5e3700 1 Processor -- start 2024-05-08T21:22:24.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.789+0000 7f1a8e5e3700 1 -- start start 2024-05-08T21:22:24.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.790+0000 7f1a8e5e3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a880d1910 0x7f1a881568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:24.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.790+0000 7f1a8e5e3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a8815b4e0 0x7f1a881f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:24.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.790+0000 7f1a8e5e3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a8815b8c0 0x7f1a88158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:24.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.790+0000 7f1a8e5e3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8806ba70 con 0x7f1a8815b8c0 2024-05-08T21:22:24.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.790+0000 7f1a8e5e3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a88156e20 con 0x7f1a8815b4e0 2024-05-08T21:22:24.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.790+0000 7f1a8e5e3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a881f3cc0 con 0x7f1a880d1910 2024-05-08T21:22:24.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.790+0000 7f1a8cb80700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a8815b8c0 0x7f1a88158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:24.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.790+0000 7f1a8cb80700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a8815b8c0 0x7f1a88158ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44024/0 (socket says 172.21.15.3:44024) 2024-05-08T21:22:24.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.790+0000 7f1a8cb80700 1 -- 172.21.15.3:0/4077035874 learned_addr learned my addr 172.21.15.3:0/4077035874 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:24.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.790+0000 7f1a87fff700 1 --2- 172.21.15.3:0/4077035874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a880d1910 0x7f1a881568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:24.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.790+0000 7f1a877fe700 1 --2- 172.21.15.3:0/4077035874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a8815b4e0 0x7f1a881f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:24.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.791+0000 7f1a8cb80700 1 -- 172.21.15.3:0/4077035874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a880d1910 msgr2=0x7f1a881568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:24.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.791+0000 7f1a8cb80700 1 --2- 172.21.15.3:0/4077035874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a880d1910 0x7f1a881568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:24.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.791+0000 7f1a8cb80700 1 -- 172.21.15.3:0/4077035874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a8815b4e0 msgr2=0x7f1a881f3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:24.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.791+0000 7f1a8cb80700 1 --2- 172.21.15.3:0/4077035874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a8815b4e0 0x7f1a881f3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:24.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.791+0000 7f1a8cb80700 1 -- 172.21.15.3:0/4077035874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a780094a0 con 0x7f1a8815b8c0 2024-05-08T21:22:24.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.791+0000 7f1a8cb80700 1 --2- 172.21.15.3:0/4077035874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a8815b8c0 0x7f1a88158ed0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009fa0 tx=0x7f1a8000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:24.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.791+0000 7f1a84ff9700 1 -- 172.21.15.3:0/4077035874 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8000ddb0 con 0x7f1a8815b8c0 2024-05-08T21:22:24.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.791+0000 7f1a84ff9700 1 -- 172.21.15.3:0/4077035874 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8000f040 con 0x7f1a8815b8c0 2024-05-08T21:22:24.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.791+0000 7f1a8e5e3700 1 -- 172.21.15.3:0/4077035874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a881f3fa0 con 0x7f1a8815b8c0 2024-05-08T21:22:24.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.791+0000 7f1a84ff9700 1 -- 172.21.15.3:0/4077035874 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a80003c60 con 0x7f1a8815b8c0 2024-05-08T21:22:24.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.791+0000 7f1a8e5e3700 1 -- 172.21.15.3:0/4077035874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a8806bc80 con 0x7f1a8815b8c0 2024-05-08T21:22:24.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.792+0000 7f1a8e5e3700 1 -- 172.21.15.3:0/4077035874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a88074bc0 con 0x7f1a8815b8c0 2024-05-08T21:22:24.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.793+0000 7f1a84ff9700 1 -- 172.21.15.3:0/4077035874 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1a80004030 con 0x7f1a8815b8c0 2024-05-08T21:22:24.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.793+0000 7f1a84ff9700 1 --2- 172.21.15.3:0/4077035874 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1a7406ac90 0x7f1a7406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:24.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.793+0000 7f1a84ff9700 1 -- 172.21.15.3:0/4077035874 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a80089350 con 0x7f1a8815b8c0 2024-05-08T21:22:24.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.793+0000 7f1a87fff700 1 --2- 172.21.15.3:0/4077035874 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1a7406ac90 0x7f1a7406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:24.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.794+0000 7f1a87fff700 1 --2- 172.21.15.3:0/4077035874 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1a7406ac90 0x7f1a7406d150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1a780090b0 tx=0x7f1a78008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:24.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:24.798+0000 7f1a84ff9700 1 -- 172.21.15.3:0/4077035874 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f1a800545c0 con 0x7f1a8815b8c0 2024-05-08T21:22:25.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.016+0000 7f1a8e5e3700 1 -- 172.21.15.3:0/4077035874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1a880b0fe0 con 0x7f1a8815b8c0 2024-05-08T21:22:25.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.019+0000 7f1a84ff9700 1 -- 172.21.15.3:0/4077035874 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1a80057be0 con 0x7f1a8815b8c0 2024-05-08T21:22:25.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.022+0000 7f1a8e5e3700 1 -- 172.21.15.3:0/4077035874 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1a7406ac90 msgr2=0x7f1a7406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:25.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.022+0000 7f1a8e5e3700 1 --2- 172.21.15.3:0/4077035874 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1a7406ac90 0x7f1a7406d150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1a780090b0 tx=0x7f1a78008030 comp rx=0 tx=0).stop 2024-05-08T21:22:25.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.022+0000 7f1a8e5e3700 1 -- 172.21.15.3:0/4077035874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a8815b8c0 msgr2=0x7f1a88158ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:25.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.022+0000 7f1a8e5e3700 1 --2- 172.21.15.3:0/4077035874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a8815b8c0 0x7f1a88158ed0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009fa0 tx=0x7f1a8000d3b0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.022+0000 7f1a8e5e3700 1 -- 172.21.15.3:0/4077035874 shutdown_connections 2024-05-08T21:22:25.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.022+0000 7f1a8e5e3700 1 --2- 172.21.15.3:0/4077035874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a880d1910 0x7f1a881568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.022+0000 7f1a8e5e3700 1 --2- 172.21.15.3:0/4077035874 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1a7406ac90 0x7f1a7406d150 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.022+0000 7f1a8e5e3700 1 --2- 172.21.15.3:0/4077035874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a8815b8c0 0x7f1a88158ed0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.022+0000 7f1a8e5e3700 1 --2- 172.21.15.3:0/4077035874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a8815b4e0 0x7f1a881f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.022+0000 7f1a8e5e3700 1 -- 172.21.15.3:0/4077035874 >> 172.21.15.3:0/4077035874 conn(0x7f1a8808afe0 msgr2=0x7f1a880cdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:25.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.022+0000 7f1a8e5e3700 1 -- 172.21.15.3:0/4077035874 shutdown_connections 2024-05-08T21:22:25.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.022+0000 7f1a8e5e3700 1 -- 172.21.15.3:0/4077035874 wait complete. 2024-05-08T21:22:25.038 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-08T21:22:25.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-08T21:22:25.085 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-08T21:22:25.134 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-08T21:22:25.134 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:22:25.134 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:25.155 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:22:25.155 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:25.217 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:22:25.217 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:25.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:25.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.464+0000 7fac26c10700 1 Processor -- start 2024-05-08T21:22:25.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.464+0000 7fac26c10700 1 -- start start 2024-05-08T21:22:25.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.464+0000 7fac26c10700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fac200aef80 0x7fac200b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:25.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.464+0000 7fac26c10700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac200af360 0x7fac2015e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:25.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.464+0000 7fac26c10700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac200b3e60 con 0x7fac2006df00 2024-05-08T21:22:25.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.465+0000 7fac26c10700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac2015ec50 con 0x7fac200aef80 2024-05-08T21:22:25.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.465+0000 7fac26c10700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2006b970 con 0x7fac200af360 2024-05-08T21:22:25.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.465+0000 7fac249ac700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fac200aef80 0x7fac200b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:25.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.465+0000 7fac249ac700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fac200aef80 0x7fac200b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44040/0 (socket says 172.21.15.3:44040) 2024-05-08T21:22:25.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.465+0000 7fac251ad700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fac2006df00 0x7fac20153d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59470/0 (socket says 172.21.15.3:59470) 2024-05-08T21:22:25.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.465+0000 7fac249ac700 1 -- 172.21.15.3:0/1293776202 learned_addr learned my addr 172.21.15.3:0/1293776202 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:25.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.465+0000 7fac1ffff700 1 --2- 172.21.15.3:0/1293776202 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac200af360 0x7fac2015e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:25.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.465+0000 7fac249ac700 1 -- 172.21.15.3:0/1293776202 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac200af360 msgr2=0x7fac2015e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:25.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.465+0000 7fac249ac700 1 --2- 172.21.15.3:0/1293776202 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac200af360 0x7fac2015e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.465+0000 7fac249ac700 1 -- 172.21.15.3:0/1293776202 >> v1:172.21.15.3:6789/0 conn(0x7fac2006df00 legacy=0x7fac20153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:25.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.465+0000 7fac249ac700 1 -- 172.21.15.3:0/1293776202 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac200b4140 con 0x7fac200aef80 2024-05-08T21:22:25.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.465+0000 7fac1f7fe700 1 -- 172.21.15.3:0/1293776202 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615015621 0 0) 0x7fac200b3e60 con 0x7fac2006df00 2024-05-08T21:22:25.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.465+0000 7fac249ac700 1 --2- 172.21.15.3:0/1293776202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200aef80 0x7fac200b37d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fac140098f0 tx=0x7fac14009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5272101e35872e server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:25.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.465+0000 7fac1f7fe700 1 -- 172.21.15.3:0/1293776202 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1400b8a0 con 0x7fac200aef80 2024-05-08T21:22:25.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.465+0000 7fac1f7fe700 1 -- 172.21.15.3:0/1293776202 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1400ba00 con 0x7fac200aef80 2024-05-08T21:22:25.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.466+0000 7fac1f7fe700 1 -- 172.21.15.3:0/1293776202 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1400bd90 con 0x7fac200aef80 2024-05-08T21:22:25.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.466+0000 7fac26c10700 1 -- 172.21.15.3:0/1293776202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200aef80 msgr2=0x7fac200b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:25.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.466+0000 7fac26c10700 1 --2- 172.21.15.3:0/1293776202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200aef80 0x7fac200b37d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fac140098f0 tx=0x7fac14009c00 comp rx=0 tx=0).stop 2024-05-08T21:22:25.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.466+0000 7fac26c10700 1 -- 172.21.15.3:0/1293776202 shutdown_connections 2024-05-08T21:22:25.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.466+0000 7fac26c10700 1 --2- 172.21.15.3:0/1293776202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200aef80 0x7fac200b37d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.466+0000 7fac26c10700 1 --2- 172.21.15.3:0/1293776202 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac200af360 0x7fac2015e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.466+0000 7fac26c10700 1 -- 172.21.15.3:0/1293776202 >> 172.21.15.3:0/1293776202 conn(0x7fac200cfce0 msgr2=0x7fac200d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:25.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.466+0000 7fac26c10700 1 -- 172.21.15.3:0/1293776202 shutdown_connections 2024-05-08T21:22:25.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.466+0000 7fac26c10700 1 -- 172.21.15.3:0/1293776202 wait complete. 2024-05-08T21:22:25.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.467+0000 7fac26c10700 1 Processor -- start 2024-05-08T21:22:25.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.467+0000 7fac26c10700 1 -- start start 2024-05-08T21:22:25.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.468+0000 7fac26c10700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac2006df00 0x7fac201540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:25.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.468+0000 7fac26c10700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac200aef80 0x7fac20158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:25.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.468+0000 7fac26c10700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200af360 0x7fac20158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:25.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.468+0000 7fac26c10700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac20154c80 con 0x7fac200af360 2024-05-08T21:22:25.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.468+0000 7fac26c10700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2015afa0 con 0x7fac200aef80 2024-05-08T21:22:25.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.468+0000 7fac26c10700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2015b280 con 0x7fac2006df00 2024-05-08T21:22:25.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.469+0000 7fac251ad700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200af360 0x7fac20158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:25.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.469+0000 7fac251ad700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200af360 0x7fac20158300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44046/0 (socket says 172.21.15.3:44046) 2024-05-08T21:22:25.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.469+0000 7fac251ad700 1 -- 172.21.15.3:0/2803137077 learned_addr learned my addr 172.21.15.3:0/2803137077 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:25.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.469+0000 7fac249ac700 1 --2- 172.21.15.3:0/2803137077 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac2006df00 0x7fac201540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:25.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.469+0000 7fac1ffff700 1 --2- 172.21.15.3:0/2803137077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac200aef80 0x7fac20158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.469+0000 7fac251ad700 1 -- 172.21.15.3:0/2803137077 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac2006df00 msgr2=0x7fac201540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.469+0000 7fac251ad700 1 --2- 172.21.15.3:0/2803137077 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac2006df00 0x7fac201540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.469+0000 7fac251ad700 1 -- 172.21.15.3:0/2803137077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac200aef80 msgr2=0x7fac20158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.469+0000 7fac251ad700 1 --2- 172.21.15.3:0/2803137077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac200aef80 0x7fac20158fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.469+0000 7fac251ad700 1 -- 172.21.15.3:0/2803137077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac14009550 con 0x7fac200af360 2024-05-08T21:22:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.469+0000 7fac251ad700 1 --2- 172.21.15.3:0/2803137077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200af360 0x7fac20158300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fac18002230 tx=0x7fac1800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.469+0000 7fac1d7fa700 1 -- 172.21.15.3:0/2803137077 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1800ac20 con 0x7fac200af360 2024-05-08T21:22:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.469+0000 7fac1d7fa700 1 -- 172.21.15.3:0/2803137077 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1800ada0 con 0x7fac200af360 2024-05-08T21:22:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.469+0000 7fac1d7fa700 1 -- 172.21.15.3:0/2803137077 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18007310 con 0x7fac200af360 2024-05-08T21:22:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.469+0000 7fac26c10700 1 -- 172.21.15.3:0/2803137077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2015b560 con 0x7fac200af360 2024-05-08T21:22:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.469+0000 7fac26c10700 1 -- 172.21.15.3:0/2803137077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2015bb20 con 0x7fac200af360 2024-05-08T21:22:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.470+0000 7fac26c10700 1 -- 172.21.15.3:0/2803137077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac2015bd30 con 0x7fac200af360 2024-05-08T21:22:25.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.471+0000 7fac1d7fa700 1 -- 172.21.15.3:0/2803137077 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fac18007470 con 0x7fac200af360 2024-05-08T21:22:25.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.472+0000 7fac1d7fa700 1 --2- 172.21.15.3:0/2803137077 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fac0406ac90 0x7fac0406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:25.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.472+0000 7fac1d7fa700 1 -- 172.21.15.3:0/2803137077 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac18089100 con 0x7fac200af360 2024-05-08T21:22:25.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.472+0000 7fac249ac700 1 --2- 172.21.15.3:0/2803137077 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fac0406ac90 0x7fac0406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:25.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.472+0000 7fac249ac700 1 --2- 172.21.15.3:0/2803137077 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fac0406ac90 0x7fac0406d150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fac14000c00 tx=0x7fac14008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:25.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.476+0000 7fac1d7fa700 1 -- 172.21.15.3:0/2803137077 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fac180543d0 con 0x7fac200af360 2024-05-08T21:22:25.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.695+0000 7fac26c10700 1 -- 172.21.15.3:0/2803137077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac20068000 con 0x7fac200af360 2024-05-08T21:22:25.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.696+0000 7fac1d7fa700 1 -- 172.21.15.3:0/2803137077 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15989 (secure 0 0 0) 0x7fac180579f0 con 0x7fac200af360 2024-05-08T21:22:25.698 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:25.698 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:53.064342+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:25.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.698+0000 7fac26c10700 1 -- 172.21.15.3:0/2803137077 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fac0406ac90 msgr2=0x7fac0406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:25.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.698+0000 7fac26c10700 1 --2- 172.21.15.3:0/2803137077 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fac0406ac90 0x7fac0406d150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fac14000c00 tx=0x7fac14008030 comp rx=0 tx=0).stop 2024-05-08T21:22:25.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.698+0000 7fac26c10700 1 -- 172.21.15.3:0/2803137077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200af360 msgr2=0x7fac20158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:25.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.698+0000 7fac26c10700 1 --2- 172.21.15.3:0/2803137077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200af360 0x7fac20158300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fac18002230 tx=0x7fac1800a060 comp rx=0 tx=0).stop 2024-05-08T21:22:25.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.698+0000 7fac26c10700 1 -- 172.21.15.3:0/2803137077 shutdown_connections 2024-05-08T21:22:25.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.698+0000 7fac26c10700 1 --2- 172.21.15.3:0/2803137077 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac2006df00 0x7fac201540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.698+0000 7fac26c10700 1 --2- 172.21.15.3:0/2803137077 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fac0406ac90 0x7fac0406d150 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.698+0000 7fac26c10700 1 --2- 172.21.15.3:0/2803137077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200af360 0x7fac20158300 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.698+0000 7fac26c10700 1 --2- 172.21.15.3:0/2803137077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac200aef80 0x7fac20158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.698+0000 7fac26c10700 1 -- 172.21.15.3:0/2803137077 >> 172.21.15.3:0/2803137077 conn(0x7fac200cfce0 msgr2=0x7fac200afb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:25.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.698+0000 7fac26c10700 1 -- 172.21.15.3:0/2803137077 shutdown_connections 2024-05-08T21:22:25.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.699+0000 7fac26c10700 1 -- 172.21.15.3:0/2803137077 wait complete. 2024-05-08T21:22:25.714 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-08T21:21:49.638998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:21:51.024632+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:22:25.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:22:25.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.851+0000 7f52dd37f700 1 Processor -- start 2024-05-08T21:22:25.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.851+0000 7f52dd37f700 1 -- start start 2024-05-08T21:22:25.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.852+0000 7f52dd37f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52d81591e0 0x7f52d80aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:25.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.852+0000 7f52dd37f700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f52d815c6a0 0x7f52d806a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:25.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.852+0000 7f52dd37f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d806b2d0 con 0x7f52d806aea0 2024-05-08T21:22:25.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.852+0000 7f52dd37f700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d806b850 con 0x7f52d815c6a0 2024-05-08T21:22:25.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.852+0000 7f52dd37f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d815cb20 con 0x7f52d81591e0 2024-05-08T21:22:25.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.852+0000 7f52d77fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f52d806aea0 0x7f52d815f210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59478/0 (socket says 172.21.15.3:59478) 2024-05-08T21:22:25.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.852+0000 7f52d77fe700 1 -- 172.21.15.3:0/107492239 learned_addr learned my addr 172.21.15.3:0/107492239 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:25.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.853+0000 7f52d6ffd700 1 --2- 172.21.15.3:0/107492239 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52d81591e0 0x7f52d80aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.853+0000 7f52d67fc700 1 --2- 172.21.15.3:0/107492239 >> v2:172.21.15.186:3300/0 conn(0x7f52d815c6a0 0x7f52d806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.853+0000 7f52d5ffb700 1 -- 172.21.15.3:0/107492239 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134523400 0 0) 0x7f52d806b2d0 con 0x7f52d806aea0 2024-05-08T21:22:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.853+0000 7f52d5ffb700 1 -- 172.21.15.3:0/107492239 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52c00035b0 con 0x7f52d806aea0 2024-05-08T21:22:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.853+0000 7f52d5ffb700 1 -- 172.21.15.3:0/107492239 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 833418972 0 0) 0x7f52c00035b0 con 0x7f52d806aea0 2024-05-08T21:22:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.853+0000 7f52d6ffd700 1 -- 172.21.15.3:0/107492239 >> v2:172.21.15.186:3300/0 conn(0x7f52d815c6a0 msgr2=0x7f52d806a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.853+0000 7f52d6ffd700 1 --2- 172.21.15.3:0/107492239 >> v2:172.21.15.186:3300/0 conn(0x7f52d815c6a0 0x7f52d806a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.853+0000 7f52d6ffd700 1 -- 172.21.15.3:0/107492239 >> v1:172.21.15.3:6789/0 conn(0x7f52d806aea0 legacy=0x7f52d815f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.853+0000 7f52d6ffd700 1 -- 172.21.15.3:0/107492239 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d80af0b0 con 0x7f52d81591e0 2024-05-08T21:22:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.853+0000 7f52d67fc700 1 --2- 172.21.15.3:0/107492239 >> v2:172.21.15.186:3300/0 conn(0x7f52d815c6a0 0x7f52d806a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.853+0000 7f52d6ffd700 1 --2- 172.21.15.3:0/107492239 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52d81591e0 0x7f52d80aea40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f52c8009840 tx=0x7f52c8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dc0f3e0969364af server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.854+0000 7f52d5ffb700 1 -- 172.21.15.3:0/107492239 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d00019f0 con 0x7f52d81591e0 2024-05-08T21:22:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.854+0000 7f52d5ffb700 1 -- 172.21.15.3:0/107492239 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c8004030 con 0x7f52d81591e0 2024-05-08T21:22:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.854+0000 7f52d5ffb700 1 -- 172.21.15.3:0/107492239 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c80043c0 con 0x7f52d81591e0 2024-05-08T21:22:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.854+0000 7f52dd37f700 1 -- 172.21.15.3:0/107492239 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52d81591e0 msgr2=0x7f52d80aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:25.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.854+0000 7f52dd37f700 1 --2- 172.21.15.3:0/107492239 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52d81591e0 0x7f52d80aea40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f52c8009840 tx=0x7f52c8009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:25.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.854+0000 7f52dd37f700 1 -- 172.21.15.3:0/107492239 shutdown_connections 2024-05-08T21:22:25.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.854+0000 7f52dd37f700 1 --2- 172.21.15.3:0/107492239 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52d81591e0 0x7f52d80aea40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.854+0000 7f52dd37f700 1 --2- 172.21.15.3:0/107492239 >> v2:172.21.15.186:3300/0 conn(0x7f52d815c6a0 0x7f52d806a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.854+0000 7f52dd37f700 1 -- 172.21.15.3:0/107492239 >> 172.21.15.3:0/107492239 conn(0x7f52d80b5810 msgr2=0x7f52d80b5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:25.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.854+0000 7f52dd37f700 1 -- 172.21.15.3:0/107492239 shutdown_connections 2024-05-08T21:22:25.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.854+0000 7f52dd37f700 1 -- 172.21.15.3:0/107492239 wait complete. 2024-05-08T21:22:25.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.856+0000 7f52dd37f700 1 Processor -- start 2024-05-08T21:22:25.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.856+0000 7f52dd37f700 1 -- start start 2024-05-08T21:22:25.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.856+0000 7f52dd37f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52d806aea0 0x7f52d81f6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:25.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.856+0000 7f52dd37f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52d81591e0 0x7f52d8150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:25.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.856+0000 7f52dd37f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52d815c6a0 0x7f52d80c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:25.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52dd37f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d81513b0 con 0x7f52d806aea0 2024-05-08T21:22:25.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52dd37f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d8152340 con 0x7f52d815c6a0 2024-05-08T21:22:25.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52dd37f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d8151730 con 0x7f52d81591e0 2024-05-08T21:22:25.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52d6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52d806aea0 0x7f52d81f6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:25.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52d6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52d806aea0 0x7f52d81f6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44054/0 (socket says 172.21.15.3:44054) 2024-05-08T21:22:25.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52d6ffd700 1 -- 172.21.15.3:0/3563415520 learned_addr learned my addr 172.21.15.3:0/3563415520 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:25.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52d67fc700 1 --2- 172.21.15.3:0/3563415520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52d81591e0 0x7f52d8150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:25.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52d77fe700 1 --2- 172.21.15.3:0/3563415520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52d815c6a0 0x7f52d80c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:25.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52d6ffd700 1 -- 172.21.15.3:0/3563415520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52d81591e0 msgr2=0x7f52d8150e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:25.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52d6ffd700 1 --2- 172.21.15.3:0/3563415520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52d81591e0 0x7f52d8150e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52d6ffd700 1 -- 172.21.15.3:0/3563415520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52d815c6a0 msgr2=0x7f52d80c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:25.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52d6ffd700 1 --2- 172.21.15.3:0/3563415520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52d815c6a0 0x7f52d80c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:25.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52d6ffd700 1 -- 172.21.15.3:0/3563415520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc0094a0 con 0x7f52d806aea0 2024-05-08T21:22:25.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52d6ffd700 1 --2- 172.21.15.3:0/3563415520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52d806aea0 0x7f52d81f6350 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f52d80676a0 tx=0x7f52c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:25.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52bf7fe700 1 -- 172.21.15.3:0/3563415520 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8003c70 con 0x7f52d806aea0 2024-05-08T21:22:25.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52bf7fe700 1 -- 172.21.15.3:0/3563415520 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c801f070 con 0x7f52d806aea0 2024-05-08T21:22:25.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52dd37f700 1 -- 172.21.15.3:0/3563415520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c80094a0 con 0x7f52d806aea0 2024-05-08T21:22:25.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52bf7fe700 1 -- 172.21.15.3:0/3563415520 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8011630 con 0x7f52d806aea0 2024-05-08T21:22:25.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.857+0000 7f52dd37f700 1 -- 172.21.15.3:0/3563415520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d8151cf0 con 0x7f52d806aea0 2024-05-08T21:22:25.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.859+0000 7f52dd37f700 1 -- 172.21.15.3:0/3563415520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d8074bc0 con 0x7f52d806aea0 2024-05-08T21:22:25.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.859+0000 7f52bf7fe700 1 -- 172.21.15.3:0/3563415520 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f52c8011790 con 0x7f52d806aea0 2024-05-08T21:22:25.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.860+0000 7f52bf7fe700 1 --2- 172.21.15.3:0/3563415520 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f52c406ac90 0x7f52c406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:25.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.860+0000 7f52bf7fe700 1 -- 172.21.15.3:0/3563415520 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52c8021070 con 0x7f52d806aea0 2024-05-08T21:22:25.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.860+0000 7f52d67fc700 1 --2- 172.21.15.3:0/3563415520 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f52c406ac90 0x7f52c406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:25.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.860+0000 7f52d67fc700 1 --2- 172.21.15.3:0/3563415520 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f52c406ac90 0x7f52c406d150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f52d81f73e0 tx=0x7f52cc009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:25.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:25.864+0000 7f52bf7fe700 1 -- 172.21.15.3:0/3563415520 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f52c8055d40 con 0x7f52d806aea0 2024-05-08T21:22:26.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.085+0000 7f52dd37f700 1 -- 172.21.15.3:0/3563415520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f52d806ab00 con 0x7f52d806aea0 2024-05-08T21:22:26.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.086+0000 7f52bf7fe700 1 -- 172.21.15.3:0/3563415520 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f52c802b020 con 0x7f52d806aea0 2024-05-08T21:22:26.087 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:22:26.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.088+0000 7f52dd37f700 1 -- 172.21.15.3:0/3563415520 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f52c406ac90 msgr2=0x7f52c406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:26.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.088+0000 7f52dd37f700 1 --2- 172.21.15.3:0/3563415520 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f52c406ac90 0x7f52c406d150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f52d81f73e0 tx=0x7f52cc009210 comp rx=0 tx=0).stop 2024-05-08T21:22:26.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.088+0000 7f52dd37f700 1 -- 172.21.15.3:0/3563415520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52d806aea0 msgr2=0x7f52d81f6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:26.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.088+0000 7f52dd37f700 1 --2- 172.21.15.3:0/3563415520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52d806aea0 0x7f52d81f6350 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f52d80676a0 tx=0x7f52c8004750 comp rx=0 tx=0).stop 2024-05-08T21:22:26.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.088+0000 7f52dd37f700 1 -- 172.21.15.3:0/3563415520 shutdown_connections 2024-05-08T21:22:26.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.088+0000 7f52dd37f700 1 --2- 172.21.15.3:0/3563415520 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f52c406ac90 0x7f52c406d150 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.088+0000 7f52dd37f700 1 --2- 172.21.15.3:0/3563415520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52d815c6a0 0x7f52d80c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.088+0000 7f52dd37f700 1 --2- 172.21.15.3:0/3563415520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52d806aea0 0x7f52d81f6350 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.088+0000 7f52dd37f700 1 --2- 172.21.15.3:0/3563415520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52d81591e0 0x7f52d8150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.088+0000 7f52dd37f700 1 -- 172.21.15.3:0/3563415520 >> 172.21.15.3:0/3563415520 conn(0x7f52d80b5810 msgr2=0x7f52d8156b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:26.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.088+0000 7f52dd37f700 1 -- 172.21.15.3:0/3563415520 shutdown_connections 2024-05-08T21:22:26.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.088+0000 7f52dd37f700 1 -- 172.21.15.3:0/3563415520 wait complete. 2024-05-08T21:22:26.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-08T21:22:26.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.240+0000 7f52cc36e700 1 Processor -- start 2024-05-08T21:22:26.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.240+0000 7f52cc36e700 1 -- start start 2024-05-08T21:22:26.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.241+0000 7f52cc36e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f52c406cb90 0x7f52c415d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:26.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.241+0000 7f52cc36e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c40cfbb0 con 0x7f52c40cf6d0 2024-05-08T21:22:26.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.241+0000 7f52cc36e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c40cfe90 con 0x7f52c40b0a80 2024-05-08T21:22:26.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.241+0000 7f52cc36e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c40d0c50 con 0x7f52c406cb90 2024-05-08T21:22:26.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.241+0000 7f52ca90b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f52c406cb90 0x7f52c415d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:26.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.241+0000 7f52c9909700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f52c40cf6d0 0x7f52c406c480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59486/0 (socket says 172.21.15.3:59486) 2024-05-08T21:22:26.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.241+0000 7f52c9909700 1 -- 172.21.15.3:0/3904825397 learned_addr learned my addr 172.21.15.3:0/3904825397 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:26.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.242+0000 7f52ca90b700 1 -- 172.21.15.3:0/3904825397 >> v1:172.21.15.186:6789/0 conn(0x7f52c40b0a80 legacy=0x7f52c409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:26.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.242+0000 7f52ca90b700 1 -- 172.21.15.3:0/3904825397 >> v1:172.21.15.3:6789/0 conn(0x7f52c40cf6d0 legacy=0x7f52c406c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:26.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.242+0000 7f52ca90b700 1 -- 172.21.15.3:0/3904825397 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c40d0170 con 0x7f52c406cb90 2024-05-08T21:22:26.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.242+0000 7f52c9108700 1 -- 172.21.15.3:0/3904825397 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180328673 0 0) 0x7f52c40cfbb0 con 0x7f52c40cf6d0 2024-05-08T21:22:26.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.242+0000 7f52ca90b700 1 --2- 172.21.15.3:0/3904825397 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52c406cb90 0x7f52c415d100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009d90 tx=0x7f52bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a5816b33114c710 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:26.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.242+0000 7f52c9108700 1 -- 172.21.15.3:0/3904825397 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc00e070 con 0x7f52c406cb90 2024-05-08T21:22:26.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.242+0000 7f52c9108700 1 -- 172.21.15.3:0/3904825397 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52bc00bbd0 con 0x7f52c406cb90 2024-05-08T21:22:26.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.242+0000 7f52c9108700 1 -- 172.21.15.3:0/3904825397 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc012430 con 0x7f52c406cb90 2024-05-08T21:22:26.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.243+0000 7f52cc36e700 1 -- 172.21.15.3:0/3904825397 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52c406cb90 msgr2=0x7f52c415d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:26.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.243+0000 7f52cc36e700 1 --2- 172.21.15.3:0/3904825397 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52c406cb90 0x7f52c415d100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009d90 tx=0x7f52bc00b330 comp rx=0 tx=0).stop 2024-05-08T21:22:26.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.243+0000 7f52cc36e700 1 -- 172.21.15.3:0/3904825397 shutdown_connections 2024-05-08T21:22:26.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.243+0000 7f52cc36e700 1 --2- 172.21.15.3:0/3904825397 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52c406cb90 0x7f52c415d100 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.243+0000 7f52cc36e700 1 -- 172.21.15.3:0/3904825397 >> 172.21.15.3:0/3904825397 conn(0x7f52c40b2f80 msgr2=0x7f52c40b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:26.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.243+0000 7f52cc36e700 1 -- 172.21.15.3:0/3904825397 shutdown_connections 2024-05-08T21:22:26.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.243+0000 7f52cc36e700 1 -- 172.21.15.3:0/3904825397 wait complete. 2024-05-08T21:22:26.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.244+0000 7f52cc36e700 1 Processor -- start 2024-05-08T21:22:26.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.244+0000 7f52cc36e700 1 -- start start 2024-05-08T21:22:26.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52cc36e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52c406cb90 0x7f52c41da760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:26.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52cc36e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52c40b0a80 0x7f52c40b3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:26.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52cc36e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52c40cf6d0 0x7f52c40ce610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:26.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52cc36e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c40c72e0 con 0x7f52c40b0a80 2024-05-08T21:22:26.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52cc36e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c40c6710 con 0x7f52c406cb90 2024-05-08T21:22:26.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52cc36e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c40c69b0 con 0x7f52c40cf6d0 2024-05-08T21:22:26.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52c9909700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52c40b0a80 0x7f52c40b3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:26.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52c9909700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52c40b0a80 0x7f52c40b3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44064/0 (socket says 172.21.15.3:44064) 2024-05-08T21:22:26.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52c9909700 1 -- 172.21.15.3:0/318406698 learned_addr learned my addr 172.21.15.3:0/318406698 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:26.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52ca10a700 1 --2- 172.21.15.3:0/318406698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52c406cb90 0x7f52c41da760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:26.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52ca90b700 1 --2- 172.21.15.3:0/318406698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52c40cf6d0 0x7f52c40ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:26.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52c9909700 1 -- 172.21.15.3:0/318406698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52c40cf6d0 msgr2=0x7f52c40ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:26.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52c9909700 1 --2- 172.21.15.3:0/318406698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52c40cf6d0 0x7f52c40ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52c9909700 1 -- 172.21.15.3:0/318406698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52c406cb90 msgr2=0x7f52c41da760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:26.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52c9909700 1 --2- 172.21.15.3:0/318406698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52c406cb90 0x7f52c41da760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52c9909700 1 -- 172.21.15.3:0/318406698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52bc009a70 con 0x7f52c40b0a80 2024-05-08T21:22:26.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52ca90b700 1 --2- 172.21.15.3:0/318406698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52c40cf6d0 0x7f52c40ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:26.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.245+0000 7f52ca10a700 1 --2- 172.21.15.3:0/318406698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52c406cb90 0x7f52c41da760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:26.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.246+0000 7f52c9909700 1 --2- 172.21.15.3:0/318406698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52c40b0a80 0x7f52c40b3e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f52b400c520 tx=0x7f52b400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:26.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.246+0000 7f52baffd700 1 -- 172.21.15.3:0/318406698 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b4004c80 con 0x7f52c40b0a80 2024-05-08T21:22:26.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.246+0000 7f52baffd700 1 -- 172.21.15.3:0/318406698 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b4004e00 con 0x7f52c40b0a80 2024-05-08T21:22:26.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.246+0000 7f52cc36e700 1 -- 172.21.15.3:0/318406698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c40c6c90 con 0x7f52c40b0a80 2024-05-08T21:22:26.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.246+0000 7f52baffd700 1 -- 172.21.15.3:0/318406698 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b4007310 con 0x7f52c40b0a80 2024-05-08T21:22:26.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.246+0000 7f52cc36e700 1 -- 172.21.15.3:0/318406698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c40b43a0 con 0x7f52c40b0a80 2024-05-08T21:22:26.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.247+0000 7f52cc36e700 1 -- 172.21.15.3:0/318406698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c40b25f0 con 0x7f52c40b0a80 2024-05-08T21:22:26.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.247+0000 7f52baffd700 1 -- 172.21.15.3:0/318406698 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f52b4007470 con 0x7f52c40b0a80 2024-05-08T21:22:26.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.248+0000 7f52baffd700 1 --2- 172.21.15.3:0/318406698 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f52a806ac90 0x7f52a806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:26.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.248+0000 7f52baffd700 1 -- 172.21.15.3:0/318406698 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52b4089160 con 0x7f52c40b0a80 2024-05-08T21:22:26.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.248+0000 7f52ca10a700 1 --2- 172.21.15.3:0/318406698 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f52a806ac90 0x7f52a806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:26.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.249+0000 7f52ca10a700 1 --2- 172.21.15.3:0/318406698 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f52a806ac90 0x7f52a806d150 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f52c0000f60 tx=0x7f52c0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:26.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.252+0000 7f52baffd700 1 -- 172.21.15.3:0/318406698 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f52b40542a0 con 0x7f52c40b0a80 2024-05-08T21:22:26.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.466+0000 7f52cc36e700 1 -- 172.21.15.3:0/318406698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f52c4067960 con 0x7f52c40b0a80 2024-05-08T21:22:26.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.467+0000 7f52baffd700 1 -- 172.21.15.3:0/318406698 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f52b40578c0 con 0x7f52c40b0a80 2024-05-08T21:22:26.469 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:22:26.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.469+0000 7f52cc36e700 1 -- 172.21.15.3:0/318406698 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f52a806ac90 msgr2=0x7f52a806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:26.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.469+0000 7f52cc36e700 1 --2- 172.21.15.3:0/318406698 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f52a806ac90 0x7f52a806d150 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f52c0000f60 tx=0x7f52c0009040 comp rx=0 tx=0).stop 2024-05-08T21:22:26.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.469+0000 7f52cc36e700 1 -- 172.21.15.3:0/318406698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52c40b0a80 msgr2=0x7f52c40b3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:26.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.469+0000 7f52cc36e700 1 --2- 172.21.15.3:0/318406698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52c40b0a80 0x7f52c40b3e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f52b400c520 tx=0x7f52b400c8e0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.469+0000 7f52cc36e700 1 -- 172.21.15.3:0/318406698 shutdown_connections 2024-05-08T21:22:26.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.469+0000 7f52cc36e700 1 --2- 172.21.15.3:0/318406698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52c40cf6d0 0x7f52c40ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.469+0000 7f52cc36e700 1 --2- 172.21.15.3:0/318406698 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f52a806ac90 0x7f52a806d150 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.469+0000 7f52cc36e700 1 --2- 172.21.15.3:0/318406698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52c406cb90 0x7f52c41da760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.469+0000 7f52cc36e700 1 --2- 172.21.15.3:0/318406698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52c40b0a80 0x7f52c40b3e60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.469+0000 7f52cc36e700 1 -- 172.21.15.3:0/318406698 >> 172.21.15.3:0/318406698 conn(0x7f52c40b2f80 msgr2=0x7f52c40cdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:26.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.470+0000 7f52cc36e700 1 -- 172.21.15.3:0/318406698 shutdown_connections 2024-05-08T21:22:26.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.470+0000 7f52cc36e700 1 -- 172.21.15.3:0/318406698 wait complete. 2024-05-08T21:22:26.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:22:26.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.622+0000 7f95bb3af700 1 Processor -- start 2024-05-08T21:22:26.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.622+0000 7f95bb3af700 1 -- start start 2024-05-08T21:22:26.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.623+0000 7f95bb3af700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f95b40aea70 0x7f95b40b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.623+0000 7f95bb3af700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95b40d0a60 0x7f95b40b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.623+0000 7f95bb3af700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95b415add0 con 0x7f95b4154df0 2024-05-08T21:22:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.623+0000 7f95bb3af700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95b415abe0 con 0x7f95b40aea70 2024-05-08T21:22:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.623+0000 7f95bb3af700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95b415b070 con 0x7f95b40d0a60 2024-05-08T21:22:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.623+0000 7f95b894a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f95b40aea70 0x7f95b40b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.623+0000 7f95b914b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f95b4154df0 0x7f95b40d0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59494/0 (socket says 172.21.15.3:59494) 2024-05-08T21:22:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.623+0000 7f95b894a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f95b40aea70 0x7f95b40b3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44066/0 (socket says 172.21.15.3:44066) 2024-05-08T21:22:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.623+0000 7f95b914b700 1 -- 172.21.15.3:0/1753019573 learned_addr learned my addr 172.21.15.3:0/1753019573 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.623+0000 7f95b994c700 1 --2- 172.21.15.3:0/1753019573 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95b40d0a60 0x7f95b40b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.623+0000 7f95b894a700 1 -- 172.21.15.3:0/1753019573 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95b40d0a60 msgr2=0x7f95b40b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.623+0000 7f95b894a700 1 --2- 172.21.15.3:0/1753019573 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95b40d0a60 0x7f95b40b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.623+0000 7f95b894a700 1 -- 172.21.15.3:0/1753019573 >> v1:172.21.15.3:6789/0 conn(0x7f95b4154df0 legacy=0x7f95b40d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.623+0000 7f95abfff700 1 -- 172.21.15.3:0/1753019573 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215173447 0 0) 0x7f95b415add0 con 0x7f95b4154df0 2024-05-08T21:22:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.623+0000 7f95b894a700 1 -- 172.21.15.3:0/1753019573 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b415b350 con 0x7f95b40aea70 2024-05-08T21:22:26.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.624+0000 7f95b894a700 1 --2- 172.21.15.3:0/1753019573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95b40aea70 0x7f95b40b3e70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f95a400bae0 tx=0x7f95a400bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7e4918a9e8fc1f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:26.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.624+0000 7f95abfff700 1 -- 172.21.15.3:0/1753019573 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a400dd30 con 0x7f95b40aea70 2024-05-08T21:22:26.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.624+0000 7f95abfff700 1 -- 172.21.15.3:0/1753019573 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a400de90 con 0x7f95b40aea70 2024-05-08T21:22:26.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.624+0000 7f95abfff700 1 -- 172.21.15.3:0/1753019573 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a4010690 con 0x7f95b40aea70 2024-05-08T21:22:26.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.624+0000 7f95bb3af700 1 -- 172.21.15.3:0/1753019573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95b40aea70 msgr2=0x7f95b40b3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:26.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.624+0000 7f95bb3af700 1 --2- 172.21.15.3:0/1753019573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95b40aea70 0x7f95b40b3e70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f95a400bae0 tx=0x7f95a400bdf0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.625+0000 7f95bb3af700 1 -- 172.21.15.3:0/1753019573 shutdown_connections 2024-05-08T21:22:26.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.625+0000 7f95bb3af700 1 --2- 172.21.15.3:0/1753019573 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95b40d0a60 0x7f95b40b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.625+0000 7f95bb3af700 1 --2- 172.21.15.3:0/1753019573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95b40aea70 0x7f95b40b3e70 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.625+0000 7f95bb3af700 1 -- 172.21.15.3:0/1753019573 >> 172.21.15.3:0/1753019573 conn(0x7f95b4099a50 msgr2=0x7f95b40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:26.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.625+0000 7f95bb3af700 1 -- 172.21.15.3:0/1753019573 shutdown_connections 2024-05-08T21:22:26.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.625+0000 7f95bb3af700 1 -- 172.21.15.3:0/1753019573 wait complete. 2024-05-08T21:22:26.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.626+0000 7f95bb3af700 1 Processor -- start 2024-05-08T21:22:26.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.626+0000 7f95bb3af700 1 -- start start 2024-05-08T21:22:26.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.627+0000 7f95bb3af700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95b40aea70 0x7f95b41ee6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:26.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.627+0000 7f95bb3af700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95b40d0a60 0x7f95b4140480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:26.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.627+0000 7f95bb3af700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95b4154df0 0x7f95b4141b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:26.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.627+0000 7f95bb3af700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b409b660 con 0x7f95b4154df0 2024-05-08T21:22:26.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.627+0000 7f95bb3af700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b41eec30 con 0x7f95b40aea70 2024-05-08T21:22:26.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.627+0000 7f95bb3af700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95b4140e40 con 0x7f95b40d0a60 2024-05-08T21:22:26.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.627+0000 7f95b994c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95b4154df0 0x7f95b4141b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:26.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.628+0000 7f95b994c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95b4154df0 0x7f95b4141b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44080/0 (socket says 172.21.15.3:44080) 2024-05-08T21:22:26.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.628+0000 7f95b994c700 1 -- 172.21.15.3:0/1631344393 learned_addr learned my addr 172.21.15.3:0/1631344393 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:26.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.628+0000 7f95b894a700 1 --2- 172.21.15.3:0/1631344393 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95b40d0a60 0x7f95b4140480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:26.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.628+0000 7f95b914b700 1 --2- 172.21.15.3:0/1631344393 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95b40aea70 0x7f95b41ee6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:26.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.628+0000 7f95b994c700 1 -- 172.21.15.3:0/1631344393 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95b40d0a60 msgr2=0x7f95b4140480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:26.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.628+0000 7f95b994c700 1 --2- 172.21.15.3:0/1631344393 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95b40d0a60 0x7f95b4140480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.628+0000 7f95b994c700 1 -- 172.21.15.3:0/1631344393 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95b40aea70 msgr2=0x7f95b41ee6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:26.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.628+0000 7f95b994c700 1 --2- 172.21.15.3:0/1631344393 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95b40aea70 0x7f95b41ee6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.628+0000 7f95b994c700 1 -- 172.21.15.3:0/1631344393 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a400b740 con 0x7f95b4154df0 2024-05-08T21:22:26.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.628+0000 7f95b994c700 1 --2- 172.21.15.3:0/1631344393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95b4154df0 0x7f95b4141b50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f95ac007e40 tx=0x7f95ac00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.629+0000 7f95a9ffb700 1 -- 172.21.15.3:0/1631344393 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac00ed30 con 0x7f95b4154df0 2024-05-08T21:22:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.629+0000 7f95bb3af700 1 -- 172.21.15.3:0/1631344393 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b4141120 con 0x7f95b4154df0 2024-05-08T21:22:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.629+0000 7f95a9ffb700 1 -- 172.21.15.3:0/1631344393 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95ac00ee90 con 0x7f95b4154df0 2024-05-08T21:22:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.629+0000 7f95a9ffb700 1 -- 172.21.15.3:0/1631344393 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac01e660 con 0x7f95b4154df0 2024-05-08T21:22:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.630+0000 7f95bb3af700 1 -- 172.21.15.3:0/1631344393 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b41416e0 con 0x7f95b4154df0 2024-05-08T21:22:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.631+0000 7f95bb3af700 1 -- 172.21.15.3:0/1631344393 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95b409b7d0 con 0x7f95b4154df0 2024-05-08T21:22:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.631+0000 7f95a9ffb700 1 -- 172.21.15.3:0/1631344393 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f95ac01e7c0 con 0x7f95b4154df0 2024-05-08T21:22:26.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.631+0000 7f95a9ffb700 1 --2- 172.21.15.3:0/1631344393 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f959806ac90 0x7f959806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:26.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.632+0000 7f95a9ffb700 1 -- 172.21.15.3:0/1631344393 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f95ac089790 con 0x7f95b4154df0 2024-05-08T21:22:26.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.632+0000 7f95b914b700 1 --2- 172.21.15.3:0/1631344393 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f959806ac90 0x7f959806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:26.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.632+0000 7f95b914b700 1 --2- 172.21.15.3:0/1631344393 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f959806ac90 0x7f959806d150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f95b0000f60 tx=0x7f95b0007e70 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:26.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.636+0000 7f95a9ffb700 1 -- 172.21.15.3:0/1631344393 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f95ac054980 con 0x7f95b4154df0 2024-05-08T21:22:26.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.853+0000 7f95bb3af700 1 -- 172.21.15.3:0/1631344393 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f95b40b4e70 con 0x7f95b4154df0 2024-05-08T21:22:26.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.853+0000 7f95a9ffb700 1 -- 172.21.15.3:0/1631344393 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f95ac057fa0 con 0x7f95b4154df0 2024-05-08T21:22:26.855 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:22:26.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.856+0000 7f95bb3af700 1 -- 172.21.15.3:0/1631344393 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f959806ac90 msgr2=0x7f959806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:26.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.856+0000 7f95bb3af700 1 --2- 172.21.15.3:0/1631344393 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f959806ac90 0x7f959806d150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f95b0000f60 tx=0x7f95b0007e70 comp rx=0 tx=0).stop 2024-05-08T21:22:26.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.856+0000 7f95bb3af700 1 -- 172.21.15.3:0/1631344393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95b4154df0 msgr2=0x7f95b4141b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:26.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.856+0000 7f95bb3af700 1 --2- 172.21.15.3:0/1631344393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95b4154df0 0x7f95b4141b50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f95ac007e40 tx=0x7f95ac00e3c0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.856+0000 7f95bb3af700 1 -- 172.21.15.3:0/1631344393 shutdown_connections 2024-05-08T21:22:26.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.856+0000 7f95bb3af700 1 --2- 172.21.15.3:0/1631344393 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f959806ac90 0x7f959806d150 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.856+0000 7f95bb3af700 1 --2- 172.21.15.3:0/1631344393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95b4154df0 0x7f95b4141b50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.856+0000 7f95bb3af700 1 --2- 172.21.15.3:0/1631344393 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95b40aea70 0x7f95b41ee6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.856+0000 7f95bb3af700 1 --2- 172.21.15.3:0/1631344393 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95b40d0a60 0x7f95b4140480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:26.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.856+0000 7f95bb3af700 1 -- 172.21.15.3:0/1631344393 >> 172.21.15.3:0/1631344393 conn(0x7f95b4099a50 msgr2=0x7f95b40b23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:26.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.856+0000 7f95bb3af700 1 -- 172.21.15.3:0/1631344393 shutdown_connections 2024-05-08T21:22:26.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:26.856+0000 7f95bb3af700 1 -- 172.21.15.3:0/1631344393 wait complete. 2024-05-08T21:22:26.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-08T21:22:27.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.008+0000 7fe7f082f700 1 Processor -- start 2024-05-08T21:22:27.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.008+0000 7fe7f082f700 1 -- start start 2024-05-08T21:22:27.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.009+0000 7fe7f082f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe7ec0c6920 0x7fe7ec0cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:27.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.009+0000 7fe7f082f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7ec0b14c0 0x7fe7ec0b1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:27.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.009+0000 7fe7f082f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ec0b1e50 con 0x7fe7ec0b0cf0 2024-05-08T21:22:27.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.009+0000 7fe7f082f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7ec0b10d0 con 0x7fe7ec0c6920 2024-05-08T21:22:27.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.009+0000 7fe7f082f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec0c6da0 con 0x7fe7ec0b14c0 2024-05-08T21:22:27.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.009+0000 7fe7e9d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe7ec0c6920 0x7fe7ec0cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:27.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.009+0000 7fe7ea59c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe7ec0b0cf0 0x7fe7ec0caa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59508/0 (socket says 172.21.15.3:59508) 2024-05-08T21:22:27.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.009+0000 7fe7ea59c700 1 -- 172.21.15.3:0/626239959 learned_addr learned my addr 172.21.15.3:0/626239959 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:27.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.009+0000 7fe7ead9d700 1 --2- 172.21.15.3:0/626239959 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7ec0b14c0 0x7fe7ec0b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:27.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.009+0000 7fe7e9d9b700 1 -- 172.21.15.3:0/626239959 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7ec0b14c0 msgr2=0x7fe7ec0b1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.009+0000 7fe7e9d9b700 1 --2- 172.21.15.3:0/626239959 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7ec0b14c0 0x7fe7ec0b1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.009+0000 7fe7e9d9b700 1 -- 172.21.15.3:0/626239959 >> v1:172.21.15.3:6789/0 conn(0x7fe7ec0b0cf0 legacy=0x7fe7ec0caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.009+0000 7fe7e9d9b700 1 -- 172.21.15.3:0/626239959 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ec15dee0 con 0x7fe7ec0c6920 2024-05-08T21:22:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.009+0000 7fe7e959a700 1 -- 172.21.15.3:0/626239959 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416723594 0 0) 0x7fe7dc001110 con 0x7fe7ec0b0cf0 2024-05-08T21:22:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.009+0000 7fe7ead9d700 1 --2- 172.21.15.3:0/626239959 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7ec0b14c0 0x7fe7ec0b1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.009+0000 7fe7e9d9b700 1 --2- 172.21.15.3:0/626239959 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7ec0c6920 0x7fe7ec0cd680 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0009840 tx=0x7fe7e0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee8db4af78715a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.010+0000 7fe7e959a700 1 -- 172.21.15.3:0/626239959 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0011830 con 0x7fe7ec0c6920 2024-05-08T21:22:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.010+0000 7fe7e959a700 1 -- 172.21.15.3:0/626239959 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e0011990 con 0x7fe7ec0c6920 2024-05-08T21:22:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.010+0000 7fe7e959a700 1 -- 172.21.15.3:0/626239959 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0011d20 con 0x7fe7ec0c6920 2024-05-08T21:22:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.010+0000 7fe7f082f700 1 -- 172.21.15.3:0/626239959 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7ec0c6920 msgr2=0x7fe7ec0cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.010+0000 7fe7f082f700 1 --2- 172.21.15.3:0/626239959 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7ec0c6920 0x7fe7ec0cd680 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0009840 tx=0x7fe7e0009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.010+0000 7fe7f082f700 1 -- 172.21.15.3:0/626239959 shutdown_connections 2024-05-08T21:22:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.010+0000 7fe7f082f700 1 --2- 172.21.15.3:0/626239959 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7ec0b14c0 0x7fe7ec0b1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.010+0000 7fe7f082f700 1 --2- 172.21.15.3:0/626239959 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7ec0c6920 0x7fe7ec0cd680 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.010+0000 7fe7f082f700 1 -- 172.21.15.3:0/626239959 >> 172.21.15.3:0/626239959 conn(0x7fe7ec0b1ff0 msgr2=0x7fe7ec0b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:27.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.010+0000 7fe7f082f700 1 -- 172.21.15.3:0/626239959 shutdown_connections 2024-05-08T21:22:27.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.010+0000 7fe7f082f700 1 -- 172.21.15.3:0/626239959 wait complete. 2024-05-08T21:22:27.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.012+0000 7fe7f082f700 1 Processor -- start 2024-05-08T21:22:27.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.012+0000 7fe7f082f700 1 -- start start 2024-05-08T21:22:27.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7f082f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7ec0b0cf0 0x7fe7ec14f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:27.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7f082f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7ec0b14c0 0x7fe7ec152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:27.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7f082f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7ec0c6920 0x7fe7ec0b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:27.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7f082f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec14f7d0 con 0x7fe7ec0c6920 2024-05-08T21:22:27.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7f082f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec152a40 con 0x7fe7ec0b14c0 2024-05-08T21:22:27.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7f082f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec0c7830 con 0x7fe7ec0b0cf0 2024-05-08T21:22:27.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7ead9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7ec0c6920 0x7fe7ec0b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:27.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7ead9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7ec0c6920 0x7fe7ec0b4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44102/0 (socket says 172.21.15.3:44102) 2024-05-08T21:22:27.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7ead9d700 1 -- 172.21.15.3:0/1556460578 learned_addr learned my addr 172.21.15.3:0/1556460578 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:27.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7ea59c700 1 --2- 172.21.15.3:0/1556460578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7ec0b0cf0 0x7fe7ec14f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:27.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7e9d9b700 1 --2- 172.21.15.3:0/1556460578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7ec0b14c0 0x7fe7ec152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:27.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7ead9d700 1 -- 172.21.15.3:0/1556460578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7ec0b0cf0 msgr2=0x7fe7ec14f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:27.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7ead9d700 1 --2- 172.21.15.3:0/1556460578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7ec0b0cf0 0x7fe7ec14f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7ead9d700 1 -- 172.21.15.3:0/1556460578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7ec0b14c0 msgr2=0x7fe7ec152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:27.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7ead9d700 1 --2- 172.21.15.3:0/1556460578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7ec0b14c0 0x7fe7ec152500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7ead9d700 1 -- 172.21.15.3:0/1556460578 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e00094a0 con 0x7fe7ec0c6920 2024-05-08T21:22:27.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.013+0000 7fe7ead9d700 1 --2- 172.21.15.3:0/1556460578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7ec0c6920 0x7fe7ec0b4c60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe7e400cef0 tx=0x7fe7e400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:27.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.014+0000 7fe7daffd700 1 -- 172.21.15.3:0/1556460578 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e400ae50 con 0x7fe7ec0c6920 2024-05-08T21:22:27.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.014+0000 7fe7daffd700 1 -- 172.21.15.3:0/1556460578 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e4015440 con 0x7fe7ec0c6920 2024-05-08T21:22:27.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.014+0000 7fe7f082f700 1 -- 172.21.15.3:0/1556460578 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7ec14fb00 con 0x7fe7ec0c6920 2024-05-08T21:22:27.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.014+0000 7fe7daffd700 1 -- 172.21.15.3:0/1556460578 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e40157f0 con 0x7fe7ec0c6920 2024-05-08T21:22:27.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.014+0000 7fe7f082f700 1 -- 172.21.15.3:0/1556460578 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ec1500c0 con 0x7fe7ec0c6920 2024-05-08T21:22:27.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.015+0000 7fe7f082f700 1 -- 172.21.15.3:0/1556460578 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ec1502b0 con 0x7fe7ec0c6920 2024-05-08T21:22:27.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.015+0000 7fe7daffd700 1 -- 172.21.15.3:0/1556460578 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe7e400d430 con 0x7fe7ec0c6920 2024-05-08T21:22:27.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.016+0000 7fe7daffd700 1 --2- 172.21.15.3:0/1556460578 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe7cc06ac90 0x7fe7cc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:27.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.016+0000 7fe7daffd700 1 -- 172.21.15.3:0/1556460578 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7e4089980 con 0x7fe7ec0c6920 2024-05-08T21:22:27.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.016+0000 7fe7ea59c700 1 --2- 172.21.15.3:0/1556460578 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe7cc06ac90 0x7fe7cc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:27.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.016+0000 7fe7ea59c700 1 --2- 172.21.15.3:0/1556460578 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe7cc06ac90 0x7fe7cc06d150 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc000f60 tx=0x7fe7dc007a40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:27.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.020+0000 7fe7daffd700 1 -- 172.21.15.3:0/1556460578 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe7e4054b50 con 0x7fe7ec0c6920 2024-05-08T21:22:27.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.254+0000 7fe7f082f700 1 -- 172.21.15.3:0/1556460578 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe7ec068000 con 0x7fe7ec0c6920 2024-05-08T21:22:27.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.254+0000 7fe7daffd700 1 -- 172.21.15.3:0/1556460578 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe7e401a020 con 0x7fe7ec0c6920 2024-05-08T21:22:27.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.257+0000 7fe7f082f700 1 -- 172.21.15.3:0/1556460578 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe7cc06ac90 msgr2=0x7fe7cc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:27.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.257+0000 7fe7f082f700 1 --2- 172.21.15.3:0/1556460578 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe7cc06ac90 0x7fe7cc06d150 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc000f60 tx=0x7fe7dc007a40 comp rx=0 tx=0).stop 2024-05-08T21:22:27.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.257+0000 7fe7f082f700 1 -- 172.21.15.3:0/1556460578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7ec0c6920 msgr2=0x7fe7ec0b4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:27.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.257+0000 7fe7f082f700 1 --2- 172.21.15.3:0/1556460578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7ec0c6920 0x7fe7ec0b4c60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe7e400cef0 tx=0x7fe7e400a340 comp rx=0 tx=0).stop 2024-05-08T21:22:27.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.257+0000 7fe7f082f700 1 -- 172.21.15.3:0/1556460578 shutdown_connections 2024-05-08T21:22:27.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.257+0000 7fe7f082f700 1 --2- 172.21.15.3:0/1556460578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7ec0b0cf0 0x7fe7ec14f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.257+0000 7fe7f082f700 1 --2- 172.21.15.3:0/1556460578 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe7cc06ac90 0x7fe7cc06d150 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.257+0000 7fe7f082f700 1 --2- 172.21.15.3:0/1556460578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7ec0c6920 0x7fe7ec0b4c60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.257+0000 7fe7f082f700 1 --2- 172.21.15.3:0/1556460578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7ec0b14c0 0x7fe7ec152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.257+0000 7fe7f082f700 1 -- 172.21.15.3:0/1556460578 >> 172.21.15.3:0/1556460578 conn(0x7fe7ec0b1ff0 msgr2=0x7fe7ec0d0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:27.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.257+0000 7fe7f082f700 1 -- 172.21.15.3:0/1556460578 shutdown_connections 2024-05-08T21:22:27.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.257+0000 7fe7f082f700 1 -- 172.21.15.3:0/1556460578 wait complete. 2024-05-08T21:22:27.260 INFO:teuthology.orchestra.run.smithi003.stderr:listed 0 entries 2024-05-08T21:22:27.272 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-08T21:22:27.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:27.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-05-08T21:22:27.315 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi003.front.sepia.ceph.com 2024-05-08T21:22:27.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:27.315 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-08T21:22:27.387 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:22:27.387 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:27.409 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:22:27.409 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:27.469 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:22:27.469 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:27.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:27.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.673+0000 7f6404284700 1 Processor -- start 2024-05-08T21:22:27.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.673+0000 7f6404284700 1 -- start start 2024-05-08T21:22:27.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f6404284700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f63fc15b7f0 0x7f63fc0cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:27.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f6404284700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63fc09aa90 0x7f63fc0cff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:27.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f6404284700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63fc09a7e0 con 0x7f63fc0cb0d0 2024-05-08T21:22:27.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f6404284700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63fc09a500 con 0x7f63fc15b7f0 2024-05-08T21:22:27.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f6404284700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc0c85e0 con 0x7f63fc09aa90 2024-05-08T21:22:27.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f640181f700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f63fc0cb0d0 0x7f63fc0cbef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59510/0 (socket says 172.21.15.3:59510) 2024-05-08T21:22:27.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f640181f700 1 -- 172.21.15.3:0/3316735347 learned_addr learned my addr 172.21.15.3:0/3316735347 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:27.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f6402020700 1 --2- 172.21.15.3:0/3316735347 >> v2:172.21.15.186:3300/0 conn(0x7f63fc15b7f0 0x7f63fc0cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:27.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f6402821700 1 --2- 172.21.15.3:0/3316735347 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63fc09aa90 0x7f63fc0cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:27.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f640101e700 1 -- 172.21.15.3:0/3316735347 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459911389 0 0) 0x7f63fc09a7e0 con 0x7f63fc0cb0d0 2024-05-08T21:22:27.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f640101e700 1 -- 172.21.15.3:0/3316735347 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63e80035b0 con 0x7f63fc0cb0d0 2024-05-08T21:22:27.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f640101e700 1 -- 172.21.15.3:0/3316735347 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 625153688 0 0) 0x7f63e80035b0 con 0x7f63fc0cb0d0 2024-05-08T21:22:27.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f640101e700 1 -- 172.21.15.3:0/3316735347 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63fc09aa90 msgr2=0x7f63fc0cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:27.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f640101e700 1 --2- 172.21.15.3:0/3316735347 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63fc09aa90 0x7f63fc0cff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f640101e700 1 -- 172.21.15.3:0/3316735347 >> v2:172.21.15.186:3300/0 conn(0x7f63fc15b7f0 msgr2=0x7f63fc0cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:27.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f640101e700 1 --2- 172.21.15.3:0/3316735347 >> v2:172.21.15.186:3300/0 conn(0x7f63fc15b7f0 0x7f63fc0cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f640101e700 1 -- 172.21.15.3:0/3316735347 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63fc0c88c0 con 0x7f63fc0cb0d0 2024-05-08T21:22:27.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.674+0000 7f640101e700 1 -- 172.21.15.3:0/3316735347 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f63ec001670 con 0x7f63fc0cb0d0 2024-05-08T21:22:27.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640101e700 1 -- 172.21.15.3:0/3316735347 >> v1:172.21.15.3:6789/0 conn(0x7f63fc0cb0d0 legacy=0x7f63fc0cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:27.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640101e700 1 --2- 172.21.15.3:0/3316735347 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63e8003a00 0x7f63e8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:27.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640101e700 1 --2- 172.21.15.3:0/3316735347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63e8004850 0x7f63e8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:27.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640101e700 1 --2- 172.21.15.3:0/3316735347 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63fc09aa90 0x7f63e8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:27.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640101e700 1 -- 172.21.15.3:0/3316735347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc09a500 con 0x7f63e8004850 2024-05-08T21:22:27.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640101e700 1 -- 172.21.15.3:0/3316735347 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc0c85e0 con 0x7f63e8003a00 2024-05-08T21:22:27.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640101e700 1 -- 172.21.15.3:0/3316735347 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e800ba10 con 0x7f63fc09aa90 2024-05-08T21:22:27.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640181f700 1 --2- 172.21.15.3:0/3316735347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63e8004850 0x7f63e8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:27.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f6402020700 1 --2- 172.21.15.3:0/3316735347 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63e8003a00 0x7f63e8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:27.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f6402821700 1 --2- 172.21.15.3:0/3316735347 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63fc09aa90 0x7f63e8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:27.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640181f700 1 -- 172.21.15.3:0/3316735347 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63fc09aa90 msgr2=0x7f63e8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:27.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640181f700 1 --2- 172.21.15.3:0/3316735347 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63fc09aa90 0x7f63e8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640181f700 1 -- 172.21.15.3:0/3316735347 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63e8003a00 msgr2=0x7f63e8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:27.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640181f700 1 --2- 172.21.15.3:0/3316735347 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63e8003a00 0x7f63e8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640181f700 1 -- 172.21.15.3:0/3316735347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e80035b0 con 0x7f63e8004850 2024-05-08T21:22:27.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f6402020700 1 --2- 172.21.15.3:0/3316735347 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63e8003a00 0x7f63e8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:27.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640181f700 1 --2- 172.21.15.3:0/3316735347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63e8004850 0x7f63e8004cd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00c320 tx=0x7f63e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=428555d8f579e4e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:27.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640101e700 1 -- 172.21.15.3:0/3316735347 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec00ce20 con 0x7f63e8004850 2024-05-08T21:22:27.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640101e700 1 -- 172.21.15.3:0/3316735347 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ec005960 con 0x7f63e8004850 2024-05-08T21:22:27.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.675+0000 7f640101e700 1 -- 172.21.15.3:0/3316735347 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec005d10 con 0x7f63e8004850 2024-05-08T21:22:27.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.676+0000 7f6404284700 1 -- 172.21.15.3:0/3316735347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63e8004850 msgr2=0x7f63e8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:27.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.676+0000 7f6404284700 1 --2- 172.21.15.3:0/3316735347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63e8004850 0x7f63e8004cd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00c320 tx=0x7f63e8005880 comp rx=0 tx=0).stop 2024-05-08T21:22:27.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.676+0000 7f6402821700 1 -- 172.21.15.3:0/3316735347 reap_dead start 2024-05-08T21:22:27.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.676+0000 7f6404284700 1 -- 172.21.15.3:0/3316735347 shutdown_connections 2024-05-08T21:22:27.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.676+0000 7f6404284700 1 -- 172.21.15.3:0/3316735347 >> 172.21.15.3:0/3316735347 conn(0x7f63fc0b2ed0 msgr2=0x7f63fc0b32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:27.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.676+0000 7f6404284700 1 -- 172.21.15.3:0/3316735347 shutdown_connections 2024-05-08T21:22:27.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.676+0000 7f6404284700 1 -- 172.21.15.3:0/3316735347 wait complete. 2024-05-08T21:22:27.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.677+0000 7f6404284700 1 Processor -- start 2024-05-08T21:22:27.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.677+0000 7f6404284700 1 -- start start 2024-05-08T21:22:27.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.678+0000 7f6404284700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e8004850 0x7f63fc06f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:27.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.678+0000 7f6404284700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63fc0b6050 0x7f63fc155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:27.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.678+0000 7f6404284700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63fc0b64d0 0x7f63fc155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:27.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.678+0000 7f6404284700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc09b1d0 con 0x7f63fc0b6050 2024-05-08T21:22:27.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.678+0000 7f6404284700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc09b9a0 con 0x7f63fc0b64d0 2024-05-08T21:22:27.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.678+0000 7f6404284700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc09d8a0 con 0x7f63e8004850 2024-05-08T21:22:27.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.678+0000 7f640181f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63fc0b6050 0x7f63fc155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:27.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.678+0000 7f640181f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63fc0b6050 0x7f63fc155520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44114/0 (socket says 172.21.15.3:44114) 2024-05-08T21:22:27.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.678+0000 7f640181f700 1 -- 172.21.15.3:0/2660247700 learned_addr learned my addr 172.21.15.3:0/2660247700 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:27.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.679+0000 7f6402821700 1 --2- 172.21.15.3:0/2660247700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63fc0b64d0 0x7f63fc155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:27.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.679+0000 7f6402020700 1 --2- 172.21.15.3:0/2660247700 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e8004850 0x7f63fc06f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:27.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.679+0000 7f640181f700 1 -- 172.21.15.3:0/2660247700 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e8004850 msgr2=0x7f63fc06f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:27.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.679+0000 7f640181f700 1 --2- 172.21.15.3:0/2660247700 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e8004850 0x7f63fc06f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.679+0000 7f640181f700 1 -- 172.21.15.3:0/2660247700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63fc0b64d0 msgr2=0x7f63fc155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:27.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.679+0000 7f640181f700 1 --2- 172.21.15.3:0/2660247700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63fc0b64d0 0x7f63fc155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.679+0000 7f640181f700 1 -- 172.21.15.3:0/2660247700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63fc0cd330 con 0x7f63fc0b6050 2024-05-08T21:22:27.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.679+0000 7f6402020700 1 --2- 172.21.15.3:0/2660247700 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e8004850 0x7f63fc06f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:27.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.679+0000 7f640181f700 1 --2- 172.21.15.3:0/2660247700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63fc0b6050 0x7f63fc155520 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f63ec008fd0 tx=0x7f63ec001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:27.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.679+0000 7f63f2ffd700 1 -- 172.21.15.3:0/2660247700 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec004870 con 0x7f63fc0b6050 2024-05-08T21:22:27.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.679+0000 7f63f2ffd700 1 -- 172.21.15.3:0/2660247700 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ec0049d0 con 0x7f63fc0b6050 2024-05-08T21:22:27.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.679+0000 7f63f2ffd700 1 -- 172.21.15.3:0/2660247700 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec004d80 con 0x7f63fc0b6050 2024-05-08T21:22:27.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.679+0000 7f6404284700 1 -- 172.21.15.3:0/2660247700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63fc09b3e0 con 0x7f63fc0b6050 2024-05-08T21:22:27.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.679+0000 7f6404284700 1 -- 172.21.15.3:0/2660247700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63fc09dab0 con 0x7f63fc0b6050 2024-05-08T21:22:27.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.680+0000 7f6404284700 1 -- 172.21.15.3:0/2660247700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63fc09e100 con 0x7f63fc0b6050 2024-05-08T21:22:27.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.681+0000 7f63f2ffd700 1 -- 172.21.15.3:0/2660247700 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f63ec0070d0 con 0x7f63fc0b6050 2024-05-08T21:22:27.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.681+0000 7f63f2ffd700 1 --2- 172.21.15.3:0/2660247700 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f63e006ac90 0x7f63e006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:27.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.681+0000 7f63f2ffd700 1 -- 172.21.15.3:0/2660247700 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63ec08a7a0 con 0x7f63fc0b6050 2024-05-08T21:22:27.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.681+0000 7f6402020700 1 --2- 172.21.15.3:0/2660247700 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f63e006ac90 0x7f63e006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:27.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.682+0000 7f6402020700 1 --2- 172.21.15.3:0/2660247700 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f63e006ac90 0x7f63e006d150 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009d00 tx=0x7f63f8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:27.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.686+0000 7f63f2ffd700 1 -- 172.21.15.3:0/2660247700 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f63ec055a10 con 0x7f63fc0b6050 2024-05-08T21:22:27.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.905+0000 7f6404284700 1 -- 172.21.15.3:0/2660247700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63fc158290 con 0x7f63fc0b6050 2024-05-08T21:22:27.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.906+0000 7f63f2ffd700 1 -- 172.21.15.3:0/2660247700 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15989 (secure 0 0 0) 0x7f63ec007dc0 con 0x7f63fc0b6050 2024-05-08T21:22:27.908 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:27.908 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:53.064342+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:27.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.909+0000 7f6404284700 1 -- 172.21.15.3:0/2660247700 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f63e006ac90 msgr2=0x7f63e006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:27.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.909+0000 7f6404284700 1 --2- 172.21.15.3:0/2660247700 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f63e006ac90 0x7f63e006d150 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009d00 tx=0x7f63f8008040 comp rx=0 tx=0).stop 2024-05-08T21:22:27.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.909+0000 7f6404284700 1 -- 172.21.15.3:0/2660247700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63fc0b6050 msgr2=0x7f63fc155520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:27.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.909+0000 7f6404284700 1 --2- 172.21.15.3:0/2660247700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63fc0b6050 0x7f63fc155520 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f63ec008fd0 tx=0x7f63ec001a00 comp rx=0 tx=0).stop 2024-05-08T21:22:27.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.909+0000 7f6404284700 1 -- 172.21.15.3:0/2660247700 shutdown_connections 2024-05-08T21:22:27.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.909+0000 7f6404284700 1 --2- 172.21.15.3:0/2660247700 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e8004850 0x7f63fc06f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.909+0000 7f6404284700 1 --2- 172.21.15.3:0/2660247700 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f63e006ac90 0x7f63e006d150 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.909+0000 7f6404284700 1 --2- 172.21.15.3:0/2660247700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63fc0b64d0 0x7f63fc155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.909+0000 7f6404284700 1 --2- 172.21.15.3:0/2660247700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63fc0b6050 0x7f63fc155520 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:27.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.909+0000 7f6404284700 1 -- 172.21.15.3:0/2660247700 >> 172.21.15.3:0/2660247700 conn(0x7f63fc0b2ed0 msgr2=0x7f63fc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:27.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.909+0000 7f6404284700 1 -- 172.21.15.3:0/2660247700 shutdown_connections 2024-05-08T21:22:27.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:27.909+0000 7f6404284700 1 -- 172.21.15.3:0/2660247700 wait complete. 2024-05-08T21:22:27.924 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-08T21:21:49.638998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:21:51.024632+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:22:27.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:22:28.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.062+0000 7f7d9876f700 1 Processor -- start 2024-05-08T21:22:28.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.062+0000 7f7d9876f700 1 -- start start 2024-05-08T21:22:28.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.063+0000 7f7d9876f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7d90166550 0x7f7d90168940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:28.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.063+0000 7f7d9876f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9015ed00 con 0x7f7d9015d910 2024-05-08T21:22:28.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.063+0000 7f7d9876f700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d900d10f0 con 0x7f7d90161da0 2024-05-08T21:22:28.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.063+0000 7f7d9876f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d9006b780 con 0x7f7d90166550 2024-05-08T21:22:28.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.064+0000 7f7d96d0c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7d90166550 0x7f7d90168940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:28.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.064+0000 7f7d9650b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7d9015d910 0x7f7d9006a960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59516/0 (socket says 172.21.15.3:59516) 2024-05-08T21:22:28.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.064+0000 7f7d9650b700 1 -- 172.21.15.3:0/1440090430 learned_addr learned my addr 172.21.15.3:0/1440090430 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:28.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.064+0000 7f7d96d0c700 1 -- 172.21.15.3:0/1440090430 >> v1:172.21.15.186:6789/0 conn(0x7f7d90161da0 legacy=0x7f7d9006b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.064+0000 7f7d96d0c700 1 -- 172.21.15.3:0/1440090430 >> v1:172.21.15.3:6789/0 conn(0x7f7d9015d910 legacy=0x7f7d9006a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.064+0000 7f7d96d0c700 1 -- 172.21.15.3:0/1440090430 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d900b2ba0 con 0x7f7d90166550 2024-05-08T21:22:28.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.064+0000 7f7d95509700 1 -- 172.21.15.3:0/1440090430 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340289299 0 0) 0x7f7d9015ed00 con 0x7f7d9015d910 2024-05-08T21:22:28.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.064+0000 7f7d96d0c700 1 --2- 172.21.15.3:0/1440090430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d90166550 0x7f7d90168940 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7d88009d90 tx=0x7f7d8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a934be7ee6bcc877 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:28.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.064+0000 7f7d95509700 1 -- 172.21.15.3:0/1440090430 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8800e070 con 0x7f7d90166550 2024-05-08T21:22:28.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.064+0000 7f7d95509700 1 -- 172.21.15.3:0/1440090430 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8800bbd0 con 0x7f7d90166550 2024-05-08T21:22:28.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.064+0000 7f7d95509700 1 -- 172.21.15.3:0/1440090430 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d88012430 con 0x7f7d90166550 2024-05-08T21:22:28.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.065+0000 7f7d9876f700 1 -- 172.21.15.3:0/1440090430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d90166550 msgr2=0x7f7d90168940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.065+0000 7f7d9876f700 1 --2- 172.21.15.3:0/1440090430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d90166550 0x7f7d90168940 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7d88009d90 tx=0x7f7d8800b330 comp rx=0 tx=0).stop 2024-05-08T21:22:28.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.065+0000 7f7d9876f700 1 -- 172.21.15.3:0/1440090430 shutdown_connections 2024-05-08T21:22:28.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.065+0000 7f7d9876f700 1 --2- 172.21.15.3:0/1440090430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d90166550 0x7f7d90168940 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.065+0000 7f7d9876f700 1 -- 172.21.15.3:0/1440090430 >> 172.21.15.3:0/1440090430 conn(0x7f7d900d03a0 msgr2=0x7f7d900d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:28.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.065+0000 7f7d9876f700 1 -- 172.21.15.3:0/1440090430 shutdown_connections 2024-05-08T21:22:28.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.065+0000 7f7d9876f700 1 -- 172.21.15.3:0/1440090430 wait complete. 2024-05-08T21:22:28.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.066+0000 7f7d9876f700 1 Processor -- start 2024-05-08T21:22:28.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.066+0000 7f7d9876f700 1 -- start start 2024-05-08T21:22:28.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.067+0000 7f7d9876f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d9015d910 0x7f7d9014fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:28.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.067+0000 7f7d9876f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d90161da0 0x7f7d9014abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:28.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.067+0000 7f7d9876f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d90166550 0x7f7d90150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:28.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.067+0000 7f7d9876f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d901581b0 con 0x7f7d90166550 2024-05-08T21:22:28.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.067+0000 7f7d9876f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d90158420 con 0x7f7d9015d910 2024-05-08T21:22:28.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.067+0000 7f7d9876f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d90158700 con 0x7f7d90161da0 2024-05-08T21:22:28.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.067+0000 7f7d96d0c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d90166550 0x7f7d90150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:28.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.067+0000 7f7d96d0c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d90166550 0x7f7d90150bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44122/0 (socket says 172.21.15.3:44122) 2024-05-08T21:22:28.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.067+0000 7f7d96d0c700 1 -- 172.21.15.3:0/1705275380 learned_addr learned my addr 172.21.15.3:0/1705275380 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:28.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.068+0000 7f7d95d0a700 1 --2- 172.21.15.3:0/1705275380 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d90161da0 0x7f7d9014abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:28.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.068+0000 7f7d9650b700 1 --2- 172.21.15.3:0/1705275380 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d9015d910 0x7f7d9014fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:28.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.068+0000 7f7d96d0c700 1 -- 172.21.15.3:0/1705275380 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d90161da0 msgr2=0x7f7d9014abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.068+0000 7f7d96d0c700 1 --2- 172.21.15.3:0/1705275380 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d90161da0 0x7f7d9014abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.068+0000 7f7d96d0c700 1 -- 172.21.15.3:0/1705275380 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d9015d910 msgr2=0x7f7d9014fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.068+0000 7f7d96d0c700 1 --2- 172.21.15.3:0/1705275380 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d9015d910 0x7f7d9014fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.068+0000 7f7d96d0c700 1 -- 172.21.15.3:0/1705275380 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d88009a70 con 0x7f7d90166550 2024-05-08T21:22:28.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.068+0000 7f7d96d0c700 1 --2- 172.21.15.3:0/1705275380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d90166550 0x7f7d90150bc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7d88014040 tx=0x7f7d88004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:28.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.068+0000 7f7d82ffd700 1 -- 172.21.15.3:0/1705275380 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8801b810 con 0x7f7d90166550 2024-05-08T21:22:28.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.068+0000 7f7d82ffd700 1 -- 172.21.15.3:0/1705275380 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d88020070 con 0x7f7d90166550 2024-05-08T21:22:28.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.068+0000 7f7d9876f700 1 -- 172.21.15.3:0/1705275380 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d901589e0 con 0x7f7d90166550 2024-05-08T21:22:28.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.068+0000 7f7d82ffd700 1 -- 172.21.15.3:0/1705275380 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8801bbb0 con 0x7f7d90166550 2024-05-08T21:22:28.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.068+0000 7f7d9876f700 1 -- 172.21.15.3:0/1705275380 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d90151100 con 0x7f7d90166550 2024-05-08T21:22:28.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.069+0000 7f7d9876f700 1 -- 172.21.15.3:0/1705275380 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d90158bd0 con 0x7f7d90166550 2024-05-08T21:22:28.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.069+0000 7f7d82ffd700 1 -- 172.21.15.3:0/1705275380 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7d8801bd10 con 0x7f7d90166550 2024-05-08T21:22:28.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.070+0000 7f7d82ffd700 1 --2- 172.21.15.3:0/1705275380 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7d7406ac90 0x7f7d7406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:28.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.070+0000 7f7d82ffd700 1 -- 172.21.15.3:0/1705275380 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d88022070 con 0x7f7d90166550 2024-05-08T21:22:28.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.070+0000 7f7d9650b700 1 --2- 172.21.15.3:0/1705275380 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7d7406ac90 0x7f7d7406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:28.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.071+0000 7f7d9650b700 1 --2- 172.21.15.3:0/1705275380 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7d7406ac90 0x7f7d7406d150 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c007fd0 tx=0x7f7d8c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:28.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.075+0000 7f7d82ffd700 1 -- 172.21.15.3:0/1705275380 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f7d8805a3c0 con 0x7f7d90166550 2024-05-08T21:22:28.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.294+0000 7f7d9876f700 1 -- 172.21.15.3:0/1705275380 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d90074bc0 con 0x7f7d90166550 2024-05-08T21:22:28.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.294+0000 7f7d82ffd700 1 -- 172.21.15.3:0/1705275380 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7d88059f50 con 0x7f7d90166550 2024-05-08T21:22:28.297 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:22:28.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.296+0000 7f7d9876f700 1 -- 172.21.15.3:0/1705275380 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7d7406ac90 msgr2=0x7f7d7406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.296+0000 7f7d9876f700 1 --2- 172.21.15.3:0/1705275380 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7d7406ac90 0x7f7d7406d150 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c007fd0 tx=0x7f7d8c009040 comp rx=0 tx=0).stop 2024-05-08T21:22:28.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.296+0000 7f7d9876f700 1 -- 172.21.15.3:0/1705275380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d90166550 msgr2=0x7f7d90150bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.296+0000 7f7d9876f700 1 --2- 172.21.15.3:0/1705275380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d90166550 0x7f7d90150bc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7d88014040 tx=0x7f7d88004ad0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.296+0000 7f7d9876f700 1 -- 172.21.15.3:0/1705275380 shutdown_connections 2024-05-08T21:22:28.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.296+0000 7f7d9876f700 1 --2- 172.21.15.3:0/1705275380 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7d7406ac90 0x7f7d7406d150 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.296+0000 7f7d9876f700 1 --2- 172.21.15.3:0/1705275380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d90166550 0x7f7d90150bc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.296+0000 7f7d9876f700 1 --2- 172.21.15.3:0/1705275380 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d9015d910 0x7f7d9014fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.296+0000 7f7d9876f700 1 --2- 172.21.15.3:0/1705275380 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d90161da0 0x7f7d9014abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.297+0000 7f7d9876f700 1 -- 172.21.15.3:0/1705275380 >> 172.21.15.3:0/1705275380 conn(0x7f7d900d03a0 msgr2=0x7f7d9006f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:28.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.297+0000 7f7d9876f700 1 -- 172.21.15.3:0/1705275380 shutdown_connections 2024-05-08T21:22:28.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.297+0000 7f7d9876f700 1 -- 172.21.15.3:0/1705275380 wait complete. 2024-05-08T21:22:28.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-08T21:22:28.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.449+0000 7f773322f700 1 Processor -- start 2024-05-08T21:22:28.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.449+0000 7f773322f700 1 -- start start 2024-05-08T21:22:28.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.450+0000 7f773322f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f772c06b070 0x7f772c15e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:28.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.450+0000 7f773322f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f772c0cf530 con 0x7f772c0cc470 2024-05-08T21:22:28.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.450+0000 7f773322f700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f772c0cf760 con 0x7f772c06f920 2024-05-08T21:22:28.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.450+0000 7f773322f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f772c0cfa40 con 0x7f772c06b070 2024-05-08T21:22:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.450+0000 7f77317cc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f772c06b070 0x7f772c15e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.450+0000 7f772bfff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f772c0cc470 0x7f772c06a960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59524/0 (socket says 172.21.15.3:59524) 2024-05-08T21:22:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.450+0000 7f772bfff700 1 -- 172.21.15.3:0/636917907 learned_addr learned my addr 172.21.15.3:0/636917907 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.450+0000 7f77317cc700 1 -- 172.21.15.3:0/636917907 >> v1:172.21.15.186:6789/0 conn(0x7f772c06f920 legacy=0x7f772c0cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.450+0000 7f77317cc700 1 -- 172.21.15.3:0/636917907 >> v1:172.21.15.3:6789/0 conn(0x7f772c0cc470 legacy=0x7f772c06a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.450+0000 7f77317cc700 1 -- 172.21.15.3:0/636917907 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772c0cfd20 con 0x7f772c06b070 2024-05-08T21:22:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.450+0000 7f772b7fe700 1 -- 172.21.15.3:0/636917907 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572723457 0 0) 0x7f772c0cf530 con 0x7f772c0cc470 2024-05-08T21:22:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.451+0000 7f77317cc700 1 --2- 172.21.15.3:0/636917907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f772c06b070 0x7f772c15e1f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7724009d90 tx=0x7f772400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104056741ed6abda server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.451+0000 7f772b7fe700 1 -- 172.21.15.3:0/636917907 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772400e070 con 0x7f772c06b070 2024-05-08T21:22:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.451+0000 7f772b7fe700 1 -- 172.21.15.3:0/636917907 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772400bb90 con 0x7f772c06b070 2024-05-08T21:22:28.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.451+0000 7f772b7fe700 1 -- 172.21.15.3:0/636917907 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7724003a20 con 0x7f772c06b070 2024-05-08T21:22:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.451+0000 7f773322f700 1 -- 172.21.15.3:0/636917907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f772c06b070 msgr2=0x7f772c15e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.451+0000 7f773322f700 1 --2- 172.21.15.3:0/636917907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f772c06b070 0x7f772c15e1f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7724009d90 tx=0x7f772400b330 comp rx=0 tx=0).stop 2024-05-08T21:22:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.451+0000 7f773322f700 1 -- 172.21.15.3:0/636917907 shutdown_connections 2024-05-08T21:22:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.451+0000 7f773322f700 1 --2- 172.21.15.3:0/636917907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f772c06b070 0x7f772c15e1f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.451+0000 7f773322f700 1 -- 172.21.15.3:0/636917907 >> 172.21.15.3:0/636917907 conn(0x7f772c0b1e10 msgr2=0x7f772c0b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.451+0000 7f773322f700 1 -- 172.21.15.3:0/636917907 shutdown_connections 2024-05-08T21:22:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.451+0000 7f773322f700 1 -- 172.21.15.3:0/636917907 wait complete. 2024-05-08T21:22:28.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.453+0000 7f773322f700 1 Processor -- start 2024-05-08T21:22:28.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.453+0000 7f773322f700 1 -- start start 2024-05-08T21:22:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f773322f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f772c06b070 0x7f772c14b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f773322f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f772c06f920 0x7f772c0b0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f773322f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f772c0cc470 0x7f772c0b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f773322f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772c14beb0 con 0x7f772c06f920 2024-05-08T21:22:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f773322f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772c0c7470 con 0x7f772c0cc470 2024-05-08T21:22:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f773322f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f772c0c7710 con 0x7f772c06b070 2024-05-08T21:22:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f772bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f772c06f920 0x7f772c0b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f772bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f772c06f920 0x7f772c0b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44150/0 (socket says 172.21.15.3:44150) 2024-05-08T21:22:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f772bfff700 1 -- 172.21.15.3:0/1232929410 learned_addr learned my addr 172.21.15.3:0/1232929410 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f77317cc700 1 --2- 172.21.15.3:0/1232929410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f772c0cc470 0x7f772c0b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f7730fcb700 1 --2- 172.21.15.3:0/1232929410 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f772c06b070 0x7f772c14b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f772bfff700 1 -- 172.21.15.3:0/1232929410 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f772c06b070 msgr2=0x7f772c14b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f772bfff700 1 --2- 172.21.15.3:0/1232929410 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f772c06b070 0x7f772c14b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f772bfff700 1 -- 172.21.15.3:0/1232929410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f772c0cc470 msgr2=0x7f772c0b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f772bfff700 1 --2- 172.21.15.3:0/1232929410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f772c0cc470 0x7f772c0b0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f772bfff700 1 -- 172.21.15.3:0/1232929410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7724009a70 con 0x7f772c06f920 2024-05-08T21:22:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f77317cc700 1 --2- 172.21.15.3:0/1232929410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f772c0cc470 0x7f772c0b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f772bfff700 1 --2- 172.21.15.3:0/1232929410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f772c06f920 0x7f772c0b0880 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f771800c540 tx=0x7f771800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.454+0000 7f77297fa700 1 -- 172.21.15.3:0/1232929410 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7718004c80 con 0x7f772c06f920 2024-05-08T21:22:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.455+0000 7f77297fa700 1 -- 172.21.15.3:0/1232929410 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7718004de0 con 0x7f772c06f920 2024-05-08T21:22:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.455+0000 7f773322f700 1 -- 172.21.15.3:0/1232929410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f772c0c79f0 con 0x7f772c06f920 2024-05-08T21:22:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.455+0000 7f77297fa700 1 -- 172.21.15.3:0/1232929410 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7718007310 con 0x7f772c06f920 2024-05-08T21:22:28.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.455+0000 7f773322f700 1 -- 172.21.15.3:0/1232929410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f772c0c9c10 con 0x7f772c06f920 2024-05-08T21:22:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.456+0000 7f773322f700 1 -- 172.21.15.3:0/1232929410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772c0c7f10 con 0x7f772c06f920 2024-05-08T21:22:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.456+0000 7f77297fa700 1 -- 172.21.15.3:0/1232929410 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7718007470 con 0x7f772c06f920 2024-05-08T21:22:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.457+0000 7f77297fa700 1 --2- 172.21.15.3:0/1232929410 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f771006ac90 0x7f771006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.457+0000 7f77297fa700 1 -- 172.21.15.3:0/1232929410 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7718089100 con 0x7f772c06f920 2024-05-08T21:22:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.457+0000 7f7730fcb700 1 --2- 172.21.15.3:0/1232929410 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f771006ac90 0x7f771006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:28.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.457+0000 7f7730fcb700 1 --2- 172.21.15.3:0/1232929410 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f771006ac90 0x7f771006d150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f772c09e440 tx=0x7f7720009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.462+0000 7f77297fa700 1 -- 172.21.15.3:0/1232929410 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f772c0c7f10 con 0x7f772c06f920 2024-05-08T21:22:28.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.677+0000 7f773322f700 1 -- 172.21.15.3:0/1232929410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f772c0c7f10 con 0x7f772c06f920 2024-05-08T21:22:28.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.677+0000 7f77297fa700 1 -- 172.21.15.3:0/1232929410 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f772c0c7f10 con 0x7f772c06f920 2024-05-08T21:22:28.679 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:22:28.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.680+0000 7f773322f700 1 -- 172.21.15.3:0/1232929410 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f771006ac90 msgr2=0x7f771006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.680+0000 7f773322f700 1 --2- 172.21.15.3:0/1232929410 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f771006ac90 0x7f771006d150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f772c09e440 tx=0x7f7720009040 comp rx=0 tx=0).stop 2024-05-08T21:22:28.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.680+0000 7f773322f700 1 -- 172.21.15.3:0/1232929410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f772c06f920 msgr2=0x7f772c0b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.680+0000 7f773322f700 1 --2- 172.21.15.3:0/1232929410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f772c06f920 0x7f772c0b0880 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f771800c540 tx=0x7f771800c900 comp rx=0 tx=0).stop 2024-05-08T21:22:28.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.680+0000 7f773322f700 1 -- 172.21.15.3:0/1232929410 shutdown_connections 2024-05-08T21:22:28.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.680+0000 7f773322f700 1 --2- 172.21.15.3:0/1232929410 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f772c06b070 0x7f772c14b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.680+0000 7f773322f700 1 --2- 172.21.15.3:0/1232929410 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f771006ac90 0x7f771006d150 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.680+0000 7f773322f700 1 --2- 172.21.15.3:0/1232929410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f772c0cc470 0x7f772c0b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.680+0000 7f773322f700 1 --2- 172.21.15.3:0/1232929410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f772c06f920 0x7f772c0b0880 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.680+0000 7f773322f700 1 -- 172.21.15.3:0/1232929410 >> 172.21.15.3:0/1232929410 conn(0x7f772c0b1e10 msgr2=0x7f772c06a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:28.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.680+0000 7f773322f700 1 -- 172.21.15.3:0/1232929410 shutdown_connections 2024-05-08T21:22:28.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.680+0000 7f773322f700 1 -- 172.21.15.3:0/1232929410 wait complete. 2024-05-08T21:22:28.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:22:28.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.832+0000 7f0a16d69700 1 Processor -- start 2024-05-08T21:22:28.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.832+0000 7f0a16d69700 1 -- start start 2024-05-08T21:22:28.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.833+0000 7f0a16d69700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0a100b0d80 0x7f0a100b3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:28.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.833+0000 7f0a16d69700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a10145040 con 0x7f0a100b4d50 2024-05-08T21:22:28.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.833+0000 7f0a16d69700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a100b38c0 con 0x7f0a1006b070 2024-05-08T21:22:28.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.833+0000 7f0a16d69700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a100b16e0 con 0x7f0a100b0d80 2024-05-08T21:22:28.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.833+0000 7f0a14b05700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0a100b0d80 0x7f0a100b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:28.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.833+0000 7f0a0ffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0a100b4d50 0x7f0a1006a960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59526/0 (socket says 172.21.15.3:59526) 2024-05-08T21:22:28.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.833+0000 7f0a0ffff700 1 -- 172.21.15.3:0/2630556009 learned_addr learned my addr 172.21.15.3:0/2630556009 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:28.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.834+0000 7f0a14b05700 1 -- 172.21.15.3:0/2630556009 >> v1:172.21.15.186:6789/0 conn(0x7f0a1006b070 legacy=0x7f0a101454d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.834+0000 7f0a14b05700 1 -- 172.21.15.3:0/2630556009 >> v1:172.21.15.3:6789/0 conn(0x7f0a100b4d50 legacy=0x7f0a1006a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.834+0000 7f0a14b05700 1 -- 172.21.15.3:0/2630556009 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a100b19c0 con 0x7f0a100b0d80 2024-05-08T21:22:28.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.834+0000 7f0a14b05700 1 --2- 172.21.15.3:0/2630556009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a100b0d80 0x7f0a100b3380 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009840 tx=0x7f0a00009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452436ed04947507 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:28.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.834+0000 7f0a0f7fe700 1 -- 172.21.15.3:0/2630556009 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a00011830 con 0x7f0a100b0d80 2024-05-08T21:22:28.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.834+0000 7f0a0f7fe700 1 -- 172.21.15.3:0/2630556009 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a00011990 con 0x7f0a100b0d80 2024-05-08T21:22:28.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.834+0000 7f0a0f7fe700 1 -- 172.21.15.3:0/2630556009 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a00011d20 con 0x7f0a100b0d80 2024-05-08T21:22:28.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.835+0000 7f0a16d69700 1 -- 172.21.15.3:0/2630556009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a100b0d80 msgr2=0x7f0a100b3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.835+0000 7f0a16d69700 1 --2- 172.21.15.3:0/2630556009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a100b0d80 0x7f0a100b3380 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009840 tx=0x7f0a00009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:28.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.835+0000 7f0a16d69700 1 -- 172.21.15.3:0/2630556009 shutdown_connections 2024-05-08T21:22:28.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.835+0000 7f0a16d69700 1 --2- 172.21.15.3:0/2630556009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a100b0d80 0x7f0a100b3380 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.835+0000 7f0a16d69700 1 -- 172.21.15.3:0/2630556009 >> 172.21.15.3:0/2630556009 conn(0x7f0a10099a50 msgr2=0x7f0a100b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:28.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.835+0000 7f0a16d69700 1 -- 172.21.15.3:0/2630556009 shutdown_connections 2024-05-08T21:22:28.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.835+0000 7f0a16d69700 1 -- 172.21.15.3:0/2630556009 wait complete. 2024-05-08T21:22:28.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.836+0000 7f0a16d69700 1 Processor -- start 2024-05-08T21:22:28.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.836+0000 7f0a16d69700 1 -- start start 2024-05-08T21:22:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a16d69700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a1006b070 0x7f0a1015d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a16d69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a100b0d80 0x7f0a100af430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a16d69700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0a100b4d50 0x7f0a100af970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a16d69700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1009d8a0 con 0x7f0a100b0d80 2024-05-08T21:22:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a16d69700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1015d650 con 0x7f0a1006b070 2024-05-08T21:22:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a16d69700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a101e7770 con 0x7f0a100b4d50 2024-05-08T21:22:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a0ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a100b0d80 0x7f0a100af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a0ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a100b0d80 0x7f0a100af430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44164/0 (socket says 172.21.15.3:44164) 2024-05-08T21:22:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a0ffff700 1 -- 172.21.15.3:0/330822471 learned_addr learned my addr 172.21.15.3:0/330822471 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a14b05700 1 --2- 172.21.15.3:0/330822471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a1006b070 0x7f0a1015d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a0ffff700 1 -- 172.21.15.3:0/330822471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0a100b4d50 msgr2=0x7f0a100af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a15306700 1 --2- 172.21.15.3:0/330822471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0a100b4d50 0x7f0a100af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:28.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a0ffff700 1 --2- 172.21.15.3:0/330822471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0a100b4d50 0x7f0a100af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a0ffff700 1 -- 172.21.15.3:0/330822471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a1006b070 msgr2=0x7f0a1015d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:28.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a0ffff700 1 --2- 172.21.15.3:0/330822471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a1006b070 0x7f0a1015d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:28.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a0ffff700 1 -- 172.21.15.3:0/330822471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a000094a0 con 0x7f0a100b0d80 2024-05-08T21:22:28.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.837+0000 7f0a14b05700 1 --2- 172.21.15.3:0/330822471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a1006b070 0x7f0a1015d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:28.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.838+0000 7f0a0ffff700 1 --2- 172.21.15.3:0/330822471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a100b0d80 0x7f0a100af430 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f09f800c4b0 tx=0x7f09f800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:28.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.838+0000 7f0a0d7fa700 1 -- 172.21.15.3:0/330822471 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f8005d70 con 0x7f0a100b0d80 2024-05-08T21:22:28.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.838+0000 7f0a0d7fa700 1 -- 172.21.15.3:0/330822471 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f80070d0 con 0x7f0a100b0d80 2024-05-08T21:22:28.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.838+0000 7f0a16d69700 1 -- 172.21.15.3:0/330822471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1009d000 con 0x7f0a100b0d80 2024-05-08T21:22:28.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.838+0000 7f0a0d7fa700 1 -- 172.21.15.3:0/330822471 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f8007440 con 0x7f0a100b0d80 2024-05-08T21:22:28.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.838+0000 7f0a16d69700 1 -- 172.21.15.3:0/330822471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1009d5c0 con 0x7f0a100b0d80 2024-05-08T21:22:28.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.839+0000 7f0a16d69700 1 -- 172.21.15.3:0/330822471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a10074bc0 con 0x7f0a100b0d80 2024-05-08T21:22:28.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.839+0000 7f0a0d7fa700 1 -- 172.21.15.3:0/330822471 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f09f80075a0 con 0x7f0a100b0d80 2024-05-08T21:22:28.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.840+0000 7f0a0d7fa700 1 --2- 172.21.15.3:0/330822471 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f09f006ac90 0x7f09f006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:28.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.840+0000 7f0a0d7fa700 1 -- 172.21.15.3:0/330822471 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09f80885d0 con 0x7f0a100b0d80 2024-05-08T21:22:28.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.840+0000 7f0a14b05700 1 --2- 172.21.15.3:0/330822471 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f09f006ac90 0x7f09f006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:28.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.841+0000 7f0a14b05700 1 --2- 172.21.15.3:0/330822471 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f09f006ac90 0x7f09f006d150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0a101e73d0 tx=0x7f0a00008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:28.845+0000 7f0a0d7fa700 1 -- 172.21.15.3:0/330822471 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f09f80538f0 con 0x7f0a100b0d80 2024-05-08T21:22:29.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.061+0000 7f0a16d69700 1 -- 172.21.15.3:0/330822471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0a1006aca0 con 0x7f0a100b0d80 2024-05-08T21:22:29.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.062+0000 7f0a0d7fa700 1 -- 172.21.15.3:0/330822471 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f09f8056f10 con 0x7f0a100b0d80 2024-05-08T21:22:29.064 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:22:29.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.064+0000 7f0a16d69700 1 -- 172.21.15.3:0/330822471 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f09f006ac90 msgr2=0x7f09f006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:29.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.064+0000 7f0a16d69700 1 --2- 172.21.15.3:0/330822471 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f09f006ac90 0x7f09f006d150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0a101e73d0 tx=0x7f0a00008030 comp rx=0 tx=0).stop 2024-05-08T21:22:29.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.064+0000 7f0a16d69700 1 -- 172.21.15.3:0/330822471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a100b0d80 msgr2=0x7f0a100af430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:29.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.064+0000 7f0a16d69700 1 --2- 172.21.15.3:0/330822471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a100b0d80 0x7f0a100af430 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f09f800c4b0 tx=0x7f09f800c870 comp rx=0 tx=0).stop 2024-05-08T21:22:29.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.064+0000 7f0a16d69700 1 -- 172.21.15.3:0/330822471 shutdown_connections 2024-05-08T21:22:29.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.064+0000 7f0a16d69700 1 --2- 172.21.15.3:0/330822471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0a100b4d50 0x7f0a100af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.064+0000 7f0a16d69700 1 --2- 172.21.15.3:0/330822471 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f09f006ac90 0x7f09f006d150 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.064+0000 7f0a16d69700 1 --2- 172.21.15.3:0/330822471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a1006b070 0x7f0a1015d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.064+0000 7f0a16d69700 1 --2- 172.21.15.3:0/330822471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a100b0d80 0x7f0a100af430 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.064+0000 7f0a16d69700 1 -- 172.21.15.3:0/330822471 >> 172.21.15.3:0/330822471 conn(0x7f0a10099a50 msgr2=0x7f0a100b1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:29.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.064+0000 7f0a16d69700 1 -- 172.21.15.3:0/330822471 shutdown_connections 2024-05-08T21:22:29.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.064+0000 7f0a16d69700 1 -- 172.21.15.3:0/330822471 wait complete. 2024-05-08T21:22:29.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-08T21:22:29.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.215+0000 7f590ae93700 1 Processor -- start 2024-05-08T21:22:29.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.215+0000 7f590ae93700 1 -- start start 2024-05-08T21:22:29.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.216+0000 7f590ae93700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59040cd4e0 0x7f590406f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:29.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.216+0000 7f590ae93700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f590406f130 0x7f59040ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:29.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.216+0000 7f590ae93700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f590415c280 con 0x7f59040ceb20 2024-05-08T21:22:29.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.216+0000 7f590ae93700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59040cefb0 con 0x7f590406f130 2024-05-08T21:22:29.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.216+0000 7f590ae93700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590406e780 con 0x7f59040cd4e0 2024-05-08T21:22:29.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.216+0000 7f5909430700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f59040ceb20 0x7f590415e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59536/0 (socket says 172.21.15.3:59536) 2024-05-08T21:22:29.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.216+0000 7f5909430700 1 -- 172.21.15.3:0/2890867551 learned_addr learned my addr 172.21.15.3:0/2890867551 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:29.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.216+0000 7f5903fff700 1 --2- 172.21.15.3:0/2890867551 >> v2:172.21.15.186:3300/0 conn(0x7f590406f130 0x7f59040ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:29.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.216+0000 7f5908c2f700 1 --2- 172.21.15.3:0/2890867551 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59040cd4e0 0x7f590406f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:29.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.216+0000 7f59037fe700 1 -- 172.21.15.3:0/2890867551 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936419173 0 0) 0x7f590415c280 con 0x7f59040ceb20 2024-05-08T21:22:29.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.216+0000 7f59037fe700 1 -- 172.21.15.3:0/2890867551 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58f40035b0 con 0x7f59040ceb20 2024-05-08T21:22:29.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.217+0000 7f59037fe700 1 -- 172.21.15.3:0/2890867551 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1526879348 0 0) 0x7f58f40035b0 con 0x7f59040ceb20 2024-05-08T21:22:29.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.217+0000 7f5903fff700 1 -- 172.21.15.3:0/2890867551 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59040cd4e0 msgr2=0x7f590406f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:29.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.217+0000 7f5903fff700 1 --2- 172.21.15.3:0/2890867551 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59040cd4e0 0x7f590406f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.217+0000 7f5903fff700 1 -- 172.21.15.3:0/2890867551 >> v1:172.21.15.3:6789/0 conn(0x7f59040ceb20 legacy=0x7f590415e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:29.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.217+0000 7f5903fff700 1 -- 172.21.15.3:0/2890867551 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f590406ea60 con 0x7f590406f130 2024-05-08T21:22:29.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.217+0000 7f5908c2f700 1 --2- 172.21.15.3:0/2890867551 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59040cd4e0 0x7f590406f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.217+0000 7f5903fff700 1 --2- 172.21.15.3:0/2890867551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f590406f130 0x7f59040ce5e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009840 tx=0x7f58f0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23987ba1d15808fe server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.217+0000 7f59037fe700 1 -- 172.21.15.3:0/2890867551 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc0019f0 con 0x7f590406f130 2024-05-08T21:22:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.217+0000 7f59037fe700 1 -- 172.21.15.3:0/2890867551 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f0004030 con 0x7f590406f130 2024-05-08T21:22:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.217+0000 7f59037fe700 1 -- 172.21.15.3:0/2890867551 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f00043a0 con 0x7f590406f130 2024-05-08T21:22:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.218+0000 7f590ae93700 1 -- 172.21.15.3:0/2890867551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f590406f130 msgr2=0x7f59040ce5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.218+0000 7f590ae93700 1 --2- 172.21.15.3:0/2890867551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f590406f130 0x7f59040ce5e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009840 tx=0x7f58f0009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.218+0000 7f590ae93700 1 -- 172.21.15.3:0/2890867551 shutdown_connections 2024-05-08T21:22:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.218+0000 7f590ae93700 1 --2- 172.21.15.3:0/2890867551 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59040cd4e0 0x7f590406f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.218+0000 7f590ae93700 1 --2- 172.21.15.3:0/2890867551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f590406f130 0x7f59040ce5e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.218+0000 7f590ae93700 1 -- 172.21.15.3:0/2890867551 >> 172.21.15.3:0/2890867551 conn(0x7f59040d0320 msgr2=0x7f59040d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.218+0000 7f590ae93700 1 -- 172.21.15.3:0/2890867551 shutdown_connections 2024-05-08T21:22:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.218+0000 7f590ae93700 1 -- 172.21.15.3:0/2890867551 wait complete. 2024-05-08T21:22:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.219+0000 7f590ae93700 1 Processor -- start 2024-05-08T21:22:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.219+0000 7f590ae93700 1 -- start start 2024-05-08T21:22:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.220+0000 7f590ae93700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f590406f130 0x7f59040c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.220+0000 7f590ae93700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59040cd4e0 0x7f59040b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.220+0000 7f590ae93700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59040ceb20 0x7f59040b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.220+0000 7f590ae93700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59040ca230 con 0x7f590406f130 2024-05-08T21:22:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.220+0000 7f590ae93700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59040c3460 con 0x7f59040ceb20 2024-05-08T21:22:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.220+0000 7f590ae93700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59040b36b0 con 0x7f59040cd4e0 2024-05-08T21:22:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.220+0000 7f5908c2f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f590406f130 0x7f59040c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.220+0000 7f5908c2f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f590406f130 0x7f59040c9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44178/0 (socket says 172.21.15.3:44178) 2024-05-08T21:22:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.220+0000 7f5908c2f700 1 -- 172.21.15.3:0/540476702 learned_addr learned my addr 172.21.15.3:0/540476702 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.220+0000 7f5909430700 1 --2- 172.21.15.3:0/540476702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59040ceb20 0x7f59040b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.221+0000 7f5903fff700 1 --2- 172.21.15.3:0/540476702 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59040cd4e0 0x7f59040b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.221+0000 7f5908c2f700 1 -- 172.21.15.3:0/540476702 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59040cd4e0 msgr2=0x7f59040b3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.221+0000 7f5908c2f700 1 --2- 172.21.15.3:0/540476702 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59040cd4e0 0x7f59040b3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.221+0000 7f5908c2f700 1 -- 172.21.15.3:0/540476702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59040ceb20 msgr2=0x7f59040b0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.221+0000 7f5908c2f700 1 --2- 172.21.15.3:0/540476702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59040ceb20 0x7f59040b0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.221+0000 7f5908c2f700 1 -- 172.21.15.3:0/540476702 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f80094a0 con 0x7f590406f130 2024-05-08T21:22:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.221+0000 7f5909430700 1 --2- 172.21.15.3:0/540476702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59040ceb20 0x7f59040b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.221+0000 7f5908c2f700 1 --2- 172.21.15.3:0/540476702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f590406f130 0x7f59040c9cf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f58f800ebe0 tx=0x7f58f8009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.221+0000 7f59017fa700 1 -- 172.21.15.3:0/540476702 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f800c910 con 0x7f590406f130 2024-05-08T21:22:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.221+0000 7f59017fa700 1 -- 172.21.15.3:0/540476702 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f800ca70 con 0x7f590406f130 2024-05-08T21:22:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.221+0000 7f590ae93700 1 -- 172.21.15.3:0/540476702 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f00094a0 con 0x7f590406f130 2024-05-08T21:22:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.221+0000 7f59017fa700 1 -- 172.21.15.3:0/540476702 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f800cde0 con 0x7f590406f130 2024-05-08T21:22:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.221+0000 7f590ae93700 1 -- 172.21.15.3:0/540476702 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59040b3c50 con 0x7f590406f130 2024-05-08T21:22:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.222+0000 7f590ae93700 1 -- 172.21.15.3:0/540476702 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5904068000 con 0x7f590406f130 2024-05-08T21:22:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.223+0000 7f59017fa700 1 -- 172.21.15.3:0/540476702 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f58f8013470 con 0x7f590406f130 2024-05-08T21:22:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.223+0000 7f59017fa700 1 --2- 172.21.15.3:0/540476702 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f58e806ac90 0x7f58e806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.223+0000 7f59017fa700 1 -- 172.21.15.3:0/540476702 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58f8089500 con 0x7f590406f130 2024-05-08T21:22:29.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.223+0000 7f5903fff700 1 --2- 172.21.15.3:0/540476702 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f58e806ac90 0x7f58e806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:29.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.224+0000 7f5903fff700 1 --2- 172.21.15.3:0/540476702 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f58e806ac90 0x7f58e806d150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f58f00090b0 tx=0x7f58f0008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:29.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.228+0000 7f59017fa700 1 -- 172.21.15.3:0/540476702 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f58f80546e0 con 0x7f590406f130 2024-05-08T21:22:29.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.462+0000 7f590ae93700 1 -- 172.21.15.3:0/540476702 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f590406d440 con 0x7f590406f130 2024-05-08T21:22:29.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.462+0000 7f59017fa700 1 -- 172.21.15.3:0/540476702 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f58f8057d00 con 0x7f590406f130 2024-05-08T21:22:29.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.464+0000 7f590ae93700 1 -- 172.21.15.3:0/540476702 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f58e806ac90 msgr2=0x7f58e806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:29.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.464+0000 7f590ae93700 1 --2- 172.21.15.3:0/540476702 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f58e806ac90 0x7f58e806d150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f58f00090b0 tx=0x7f58f0008030 comp rx=0 tx=0).stop 2024-05-08T21:22:29.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.464+0000 7f590ae93700 1 -- 172.21.15.3:0/540476702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f590406f130 msgr2=0x7f59040c9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:29.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.464+0000 7f590ae93700 1 --2- 172.21.15.3:0/540476702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f590406f130 0x7f59040c9cf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f58f800ebe0 tx=0x7f58f8009ec0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.464+0000 7f590ae93700 1 -- 172.21.15.3:0/540476702 shutdown_connections 2024-05-08T21:22:29.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.464+0000 7f590ae93700 1 --2- 172.21.15.3:0/540476702 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f58e806ac90 0x7f58e806d150 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.464+0000 7f590ae93700 1 --2- 172.21.15.3:0/540476702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59040ceb20 0x7f59040b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.464+0000 7f590ae93700 1 --2- 172.21.15.3:0/540476702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f590406f130 0x7f59040c9cf0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.464+0000 7f590ae93700 1 --2- 172.21.15.3:0/540476702 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59040cd4e0 0x7f59040b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.464+0000 7f590ae93700 1 -- 172.21.15.3:0/540476702 >> 172.21.15.3:0/540476702 conn(0x7f59040d0320 msgr2=0x7f59040cc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:29.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.464+0000 7f590ae93700 1 -- 172.21.15.3:0/540476702 shutdown_connections 2024-05-08T21:22:29.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.464+0000 7f590ae93700 1 -- 172.21.15.3:0/540476702 wait complete. 2024-05-08T21:22:29.467 INFO:teuthology.orchestra.run.smithi003.stderr:listed 0 entries 2024-05-08T21:22:29.479 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-05-08T21:22:29.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:29.480 DEBUG:teuthology.orchestra.run.smithi186:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-05-08T21:22:29.520 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi186.front.sepia.ceph.com 2024-05-08T21:22:29.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:29.520 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-08T21:22:29.591 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:22:29.591 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:29.613 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:22:29.613 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:29.675 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:22:29.675 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:29.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:29.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.921+0000 7fe3664f7700 1 Processor -- start 2024-05-08T21:22:29.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.921+0000 7fe3664f7700 1 -- start start 2024-05-08T21:22:29.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.922+0000 7fe3664f7700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe36006bd80 0x7fe3600d0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:29.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.922+0000 7fe3664f7700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3600cf990 con 0x7fe36006c3d0 2024-05-08T21:22:29.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.922+0000 7fe3664f7700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3600d04b0 con 0x7fe36015da10 2024-05-08T21:22:29.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.922+0000 7fe3664f7700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36009ab40 con 0x7fe36006bd80 2024-05-08T21:22:29.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.922+0000 7fe35f7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe36006bd80 0x7fe3600d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:29.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.922+0000 7fe364a94700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe36006c3d0 0x7fe3600cbb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59538/0 (socket says 172.21.15.3:59538) 2024-05-08T21:22:29.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.922+0000 7fe364a94700 1 -- 172.21.15.3:0/1422622524 learned_addr learned my addr 172.21.15.3:0/1422622524 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:29.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.922+0000 7fe35f7fe700 1 -- 172.21.15.3:0/1422622524 >> v1:172.21.15.186:6789/0 conn(0x7fe36015da10 legacy=0x7fe36009a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:29.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.922+0000 7fe35f7fe700 1 -- 172.21.15.3:0/1422622524 >> v1:172.21.15.3:6789/0 conn(0x7fe36006c3d0 legacy=0x7fe3600cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:29.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.922+0000 7fe35f7fe700 1 -- 172.21.15.3:0/1422622524 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36009ad60 con 0x7fe36006bd80 2024-05-08T21:22:29.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.922+0000 7fe35effd700 1 -- 172.21.15.3:0/1422622524 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292969576 0 0) 0x7fe3600cf990 con 0x7fe36006c3d0 2024-05-08T21:22:29.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.922+0000 7fe35f7fe700 1 --2- 172.21.15.3:0/1422622524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36006bd80 0x7fe3600d0940 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe354009840 tx=0x7fe354009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef51bc9601250cfa server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:29.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.923+0000 7fe35effd700 1 -- 172.21.15.3:0/1422622524 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe354004230 con 0x7fe36006bd80 2024-05-08T21:22:29.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.923+0000 7fe35effd700 1 -- 172.21.15.3:0/1422622524 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe354004390 con 0x7fe36006bd80 2024-05-08T21:22:29.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.923+0000 7fe35effd700 1 -- 172.21.15.3:0/1422622524 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3540036a0 con 0x7fe36006bd80 2024-05-08T21:22:29.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.923+0000 7fe3664f7700 1 -- 172.21.15.3:0/1422622524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36006bd80 msgr2=0x7fe3600d0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:29.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.923+0000 7fe3664f7700 1 --2- 172.21.15.3:0/1422622524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36006bd80 0x7fe3600d0940 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe354009840 tx=0x7fe354009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:29.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.923+0000 7fe3664f7700 1 -- 172.21.15.3:0/1422622524 shutdown_connections 2024-05-08T21:22:29.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.923+0000 7fe3664f7700 1 --2- 172.21.15.3:0/1422622524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36006bd80 0x7fe3600d0940 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.923+0000 7fe3664f7700 1 -- 172.21.15.3:0/1422622524 >> 172.21.15.3:0/1422622524 conn(0x7fe3600b3010 msgr2=0x7fe3600b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:29.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.923+0000 7fe3664f7700 1 -- 172.21.15.3:0/1422622524 shutdown_connections 2024-05-08T21:22:29.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.923+0000 7fe3664f7700 1 -- 172.21.15.3:0/1422622524 wait complete. 2024-05-08T21:22:29.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.925+0000 7fe3664f7700 1 Processor -- start 2024-05-08T21:22:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.925+0000 7fe3664f7700 1 -- start start 2024-05-08T21:22:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe3664f7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe36006bd80 0x7fe3600c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe3664f7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe36006c3d0 0x7fe36014a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe3664f7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36015da10 0x7fe3600cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe3664f7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3600b5db0 con 0x7fe36015da10 2024-05-08T21:22:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe3664f7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3600b46b0 con 0x7fe36006c3d0 2024-05-08T21:22:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe3664f7700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3600b4990 con 0x7fe36006bd80 2024-05-08T21:22:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe364a94700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36015da10 0x7fe3600cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe364a94700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36015da10 0x7fe3600cbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44194/0 (socket says 172.21.15.3:44194) 2024-05-08T21:22:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe364a94700 1 -- 172.21.15.3:0/113555006 learned_addr learned my addr 172.21.15.3:0/113555006 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe35f7fe700 1 --2- 172.21.15.3:0/113555006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe36006c3d0 0x7fe36014a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe35ffff700 1 --2- 172.21.15.3:0/113555006 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe36006bd80 0x7fe3600c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe364a94700 1 -- 172.21.15.3:0/113555006 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe36006bd80 msgr2=0x7fe3600c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe364a94700 1 --2- 172.21.15.3:0/113555006 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe36006bd80 0x7fe3600c7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe364a94700 1 -- 172.21.15.3:0/113555006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe36006c3d0 msgr2=0x7fe36014a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:29.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe364a94700 1 --2- 172.21.15.3:0/113555006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe36006c3d0 0x7fe36014a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:29.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe364a94700 1 -- 172.21.15.3:0/113555006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3540094a0 con 0x7fe36015da10 2024-05-08T21:22:29.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe364a94700 1 --2- 172.21.15.3:0/113555006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36015da10 0x7fe3600cbf50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe35800cca0 tx=0x7fe35800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:29.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.926+0000 7fe35cff9700 1 -- 172.21.15.3:0/113555006 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35800ab70 con 0x7fe36015da10 2024-05-08T21:22:29.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.927+0000 7fe35cff9700 1 -- 172.21.15.3:0/113555006 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35800acd0 con 0x7fe36015da10 2024-05-08T21:22:29.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.927+0000 7fe3664f7700 1 -- 172.21.15.3:0/113555006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3600b4c70 con 0x7fe36015da10 2024-05-08T21:22:29.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.927+0000 7fe35cff9700 1 -- 172.21.15.3:0/113555006 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358007310 con 0x7fe36015da10 2024-05-08T21:22:29.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.927+0000 7fe3664f7700 1 -- 172.21.15.3:0/113555006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3600b5230 con 0x7fe36015da10 2024-05-08T21:22:29.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.928+0000 7fe3664f7700 1 -- 172.21.15.3:0/113555006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3600c8e20 con 0x7fe36015da10 2024-05-08T21:22:29.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.928+0000 7fe35cff9700 1 -- 172.21.15.3:0/113555006 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe358007470 con 0x7fe36015da10 2024-05-08T21:22:29.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.928+0000 7fe35cff9700 1 --2- 172.21.15.3:0/113555006 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe34c06ac90 0x7fe34c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:29.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.929+0000 7fe35cff9700 1 -- 172.21.15.3:0/113555006 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe358089060 con 0x7fe36015da10 2024-05-08T21:22:29.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.929+0000 7fe35ffff700 1 --2- 172.21.15.3:0/113555006 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe34c06ac90 0x7fe34c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:29.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.929+0000 7fe35ffff700 1 --2- 172.21.15.3:0/113555006 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe34c06ac90 0x7fe34c06d150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe350009960 tx=0x7fe3500090e0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:29.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:29.933+0000 7fe35cff9700 1 -- 172.21.15.3:0/113555006 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe3580542d0 con 0x7fe36015da10 2024-05-08T21:22:30.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.153+0000 7fe3664f7700 1 -- 172.21.15.3:0/113555006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3600b5420 con 0x7fe36015da10 2024-05-08T21:22:30.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.154+0000 7fe35cff9700 1 -- 172.21.15.3:0/113555006 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15989 (secure 0 0 0) 0x7fe3580578f0 con 0x7fe36015da10 2024-05-08T21:22:30.156 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:30.156 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:53.064342+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:30.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.156+0000 7fe3664f7700 1 -- 172.21.15.3:0/113555006 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe34c06ac90 msgr2=0x7fe34c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:30.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.156+0000 7fe3664f7700 1 --2- 172.21.15.3:0/113555006 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe34c06ac90 0x7fe34c06d150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe350009960 tx=0x7fe3500090e0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.156+0000 7fe3664f7700 1 -- 172.21.15.3:0/113555006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36015da10 msgr2=0x7fe3600cbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:30.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.156+0000 7fe3664f7700 1 --2- 172.21.15.3:0/113555006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36015da10 0x7fe3600cbf50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe35800cca0 tx=0x7fe35800a060 comp rx=0 tx=0).stop 2024-05-08T21:22:30.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.157+0000 7fe3664f7700 1 -- 172.21.15.3:0/113555006 shutdown_connections 2024-05-08T21:22:30.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.157+0000 7fe3664f7700 1 --2- 172.21.15.3:0/113555006 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe36006bd80 0x7fe3600c7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.157+0000 7fe3664f7700 1 --2- 172.21.15.3:0/113555006 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe34c06ac90 0x7fe34c06d150 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.157+0000 7fe3664f7700 1 --2- 172.21.15.3:0/113555006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36015da10 0x7fe3600cbf50 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.157+0000 7fe3664f7700 1 --2- 172.21.15.3:0/113555006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe36006c3d0 0x7fe36014a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.157+0000 7fe3664f7700 1 -- 172.21.15.3:0/113555006 >> 172.21.15.3:0/113555006 conn(0x7fe3600b3010 msgr2=0x7fe36006ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:30.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.157+0000 7fe3664f7700 1 -- 172.21.15.3:0/113555006 shutdown_connections 2024-05-08T21:22:30.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.157+0000 7fe3664f7700 1 -- 172.21.15.3:0/113555006 wait complete. 2024-05-08T21:22:30.172 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-08T21:21:49.638998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:21:51.024632+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:22:30.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:22:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.310+0000 7fd3afd57700 1 Processor -- start 2024-05-08T21:22:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.311+0000 7fd3afd57700 1 -- start start 2024-05-08T21:22:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.312+0000 7fd3afd57700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3a815b4e0 0x7fd3a80b1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.312+0000 7fd3afd57700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd3a815b8c0 0x7fd3a80d1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.312+0000 7fd3afd57700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a80cf560 con 0x7fd3a80d1740 2024-05-08T21:22:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.312+0000 7fd3afd57700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a80cefe0 con 0x7fd3a815b8c0 2024-05-08T21:22:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.312+0000 7fd3afd57700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80d1bb0 con 0x7fd3a815b4e0 2024-05-08T21:22:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.312+0000 7fd3ae2f4700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd3a80d1740 0x7fd3a80cc2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59548/0 (socket says 172.21.15.3:59548) 2024-05-08T21:22:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.312+0000 7fd3ae2f4700 1 -- 172.21.15.3:0/3814971874 learned_addr learned my addr 172.21.15.3:0/3814971874 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.312+0000 7fd3adaf3700 1 --2- 172.21.15.3:0/3814971874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3a815b4e0 0x7fd3a80b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.312+0000 7fd3ad2f2700 1 --2- 172.21.15.3:0/3814971874 >> v2:172.21.15.186:3300/0 conn(0x7fd3a815b8c0 0x7fd3a80d1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.312+0000 7fd3acaf1700 1 -- 172.21.15.3:0/3814971874 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532662795 0 0) 0x7fd3a80cf560 con 0x7fd3a80d1740 2024-05-08T21:22:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.312+0000 7fd3acaf1700 1 -- 172.21.15.3:0/3814971874 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3940035b0 con 0x7fd3a80d1740 2024-05-08T21:22:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.313+0000 7fd3ad2f2700 1 -- 172.21.15.3:0/3814971874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3a815b4e0 msgr2=0x7fd3a80b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.313+0000 7fd3ad2f2700 1 --2- 172.21.15.3:0/3814971874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3a815b4e0 0x7fd3a80b1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.313+0000 7fd3ad2f2700 1 -- 172.21.15.3:0/3814971874 >> v1:172.21.15.3:6789/0 conn(0x7fd3a80d1740 legacy=0x7fd3a80cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.313+0000 7fd3ad2f2700 1 -- 172.21.15.3:0/3814971874 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80ccff0 con 0x7fd3a815b8c0 2024-05-08T21:22:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.313+0000 7fd3ad2f2700 1 --2- 172.21.15.3:0/3814971874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a815b8c0 0x7fd3a80d1200 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd398009840 tx=0x7fd398009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=150c3409d4288022 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.313+0000 7fd3acaf1700 1 -- 172.21.15.3:0/3814971874 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398011830 con 0x7fd3a815b8c0 2024-05-08T21:22:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.313+0000 7fd3acaf1700 1 -- 172.21.15.3:0/3814971874 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd398011990 con 0x7fd3a815b8c0 2024-05-08T21:22:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.313+0000 7fd3acaf1700 1 -- 172.21.15.3:0/3814971874 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398011d20 con 0x7fd3a815b8c0 2024-05-08T21:22:30.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.314+0000 7fd3afd57700 1 -- 172.21.15.3:0/3814971874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a815b8c0 msgr2=0x7fd3a80d1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:30.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.314+0000 7fd3afd57700 1 --2- 172.21.15.3:0/3814971874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a815b8c0 0x7fd3a80d1200 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd398009840 tx=0x7fd398009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:30.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.314+0000 7fd3afd57700 1 -- 172.21.15.3:0/3814971874 shutdown_connections 2024-05-08T21:22:30.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.314+0000 7fd3afd57700 1 --2- 172.21.15.3:0/3814971874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3a815b4e0 0x7fd3a80b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.314+0000 7fd3afd57700 1 --2- 172.21.15.3:0/3814971874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a815b8c0 0x7fd3a80d1200 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.314+0000 7fd3afd57700 1 -- 172.21.15.3:0/3814971874 >> 172.21.15.3:0/3814971874 conn(0x7fd3a808afe0 msgr2=0x7fd3a808b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.314+0000 7fd3afd57700 1 -- 172.21.15.3:0/3814971874 shutdown_connections 2024-05-08T21:22:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.314+0000 7fd3afd57700 1 -- 172.21.15.3:0/3814971874 wait complete. 2024-05-08T21:22:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.315+0000 7fd3afd57700 1 Processor -- start 2024-05-08T21:22:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.315+0000 7fd3afd57700 1 -- start start 2024-05-08T21:22:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3afd57700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3a80d1740 0x7fd3a80c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3afd57700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a815b4e0 0x7fd3a815e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3afd57700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3a815b8c0 0x7fd3a814aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3afd57700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8152e20 con 0x7fd3a80d1740 2024-05-08T21:22:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3afd57700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a814afa0 con 0x7fd3a815b4e0 2024-05-08T21:22:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3afd57700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80c9760 con 0x7fd3a815b8c0 2024-05-08T21:22:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3adaf3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3a80d1740 0x7fd3a80c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3adaf3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3a80d1740 0x7fd3a80c84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44210/0 (socket says 172.21.15.3:44210) 2024-05-08T21:22:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3adaf3700 1 -- 172.21.15.3:0/1995172403 learned_addr learned my addr 172.21.15.3:0/1995172403 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:30.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3ad2f2700 1 --2- 172.21.15.3:0/1995172403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a815b4e0 0x7fd3a815e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:30.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3ae2f4700 1 --2- 172.21.15.3:0/1995172403 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3a815b8c0 0x7fd3a814aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:30.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3adaf3700 1 -- 172.21.15.3:0/1995172403 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3a815b8c0 msgr2=0x7fd3a814aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:30.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3adaf3700 1 --2- 172.21.15.3:0/1995172403 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3a815b8c0 0x7fd3a814aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3adaf3700 1 -- 172.21.15.3:0/1995172403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a815b4e0 msgr2=0x7fd3a815e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:30.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3adaf3700 1 --2- 172.21.15.3:0/1995172403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a815b4e0 0x7fd3a815e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3adaf3700 1 -- 172.21.15.3:0/1995172403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3980094a0 con 0x7fd3a80d1740 2024-05-08T21:22:30.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.316+0000 7fd3ae2f4700 1 --2- 172.21.15.3:0/1995172403 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3a815b8c0 0x7fd3a814aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:30.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.317+0000 7fd3adaf3700 1 --2- 172.21.15.3:0/1995172403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3a80d1740 0x7fd3a80c84e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400e9b0 tx=0x7fd3a400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:30.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.317+0000 7fd39e7fc700 1 -- 172.21.15.3:0/1995172403 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4009710 con 0x7fd3a80d1740 2024-05-08T21:22:30.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.317+0000 7fd39e7fc700 1 -- 172.21.15.3:0/1995172403 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a400c850 con 0x7fd3a80d1740 2024-05-08T21:22:30.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.317+0000 7fd3afd57700 1 -- 172.21.15.3:0/1995172403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a80c9bf0 con 0x7fd3a80d1740 2024-05-08T21:22:30.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.317+0000 7fd39e7fc700 1 -- 172.21.15.3:0/1995172403 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a400cbc0 con 0x7fd3a80d1740 2024-05-08T21:22:30.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.317+0000 7fd3afd57700 1 -- 172.21.15.3:0/1995172403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a80c8e30 con 0x7fd3a80d1740 2024-05-08T21:22:30.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.318+0000 7fd3afd57700 1 -- 172.21.15.3:0/1995172403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a80c9a40 con 0x7fd3a80d1740 2024-05-08T21:22:30.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.318+0000 7fd39e7fc700 1 -- 172.21.15.3:0/1995172403 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd3a400cd20 con 0x7fd3a80d1740 2024-05-08T21:22:30.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.319+0000 7fd39e7fc700 1 --2- 172.21.15.3:0/1995172403 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd38c06ac90 0x7fd38c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:30.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.319+0000 7fd39e7fc700 1 -- 172.21.15.3:0/1995172403 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3a40892e0 con 0x7fd3a80d1740 2024-05-08T21:22:30.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.319+0000 7fd3ad2f2700 1 --2- 172.21.15.3:0/1995172403 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd38c06ac90 0x7fd38c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:30.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.319+0000 7fd3ad2f2700 1 --2- 172.21.15.3:0/1995172403 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd38c06ac90 0x7fd38c06d150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd398000c00 tx=0x7fd39801c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:30.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.323+0000 7fd39e7fc700 1 -- 172.21.15.3:0/1995172403 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd3a4054550 con 0x7fd3a80d1740 2024-05-08T21:22:30.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.540+0000 7fd3afd57700 1 -- 172.21.15.3:0/1995172403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd3a80cc9b0 con 0x7fd3a80d1740 2024-05-08T21:22:30.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.540+0000 7fd39e7fc700 1 -- 172.21.15.3:0/1995172403 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd3a80cc9b0 con 0x7fd3a80d1740 2024-05-08T21:22:30.542 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:22:30.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.542+0000 7fd3afd57700 1 -- 172.21.15.3:0/1995172403 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd38c06ac90 msgr2=0x7fd38c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:30.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.542+0000 7fd3afd57700 1 --2- 172.21.15.3:0/1995172403 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd38c06ac90 0x7fd38c06d150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd398000c00 tx=0x7fd39801c040 comp rx=0 tx=0).stop 2024-05-08T21:22:30.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.542+0000 7fd3afd57700 1 -- 172.21.15.3:0/1995172403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3a80d1740 msgr2=0x7fd3a80c84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:30.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.542+0000 7fd3afd57700 1 --2- 172.21.15.3:0/1995172403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3a80d1740 0x7fd3a80c84e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400e9b0 tx=0x7fd3a400ecc0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.542+0000 7fd3afd57700 1 -- 172.21.15.3:0/1995172403 shutdown_connections 2024-05-08T21:22:30.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.542+0000 7fd3afd57700 1 --2- 172.21.15.3:0/1995172403 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3a815b8c0 0x7fd3a814aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.542+0000 7fd3afd57700 1 --2- 172.21.15.3:0/1995172403 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd38c06ac90 0x7fd38c06d150 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.542+0000 7fd3afd57700 1 --2- 172.21.15.3:0/1995172403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3a80d1740 0x7fd3a80c84e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.542+0000 7fd3afd57700 1 --2- 172.21.15.3:0/1995172403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a815b4e0 0x7fd3a815e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.542+0000 7fd3afd57700 1 -- 172.21.15.3:0/1995172403 >> 172.21.15.3:0/1995172403 conn(0x7fd3a808afe0 msgr2=0x7fd3a80d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:30.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.542+0000 7fd3afd57700 1 -- 172.21.15.3:0/1995172403 shutdown_connections 2024-05-08T21:22:30.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.542+0000 7fd3afd57700 1 -- 172.21.15.3:0/1995172403 wait complete. 2024-05-08T21:22:30.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-08T21:22:30.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.698+0000 7f9359cdf700 1 Processor -- start 2024-05-08T21:22:30.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.698+0000 7f9359cdf700 1 -- start start 2024-05-08T21:22:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.699+0000 7f9359cdf700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93540d0a60 0x7f93540b5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.699+0000 7f9359cdf700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f935415fbb0 0x7f93540afb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.699+0000 7f9359cdf700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93540af6e0 con 0x7f9354145040 2024-05-08T21:22:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.699+0000 7f9359cdf700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93540b6460 con 0x7f935415fbb0 2024-05-08T21:22:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.699+0000 7f9359cdf700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93540aeb50 con 0x7f93540d0a60 2024-05-08T21:22:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.699+0000 7f9353fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f935415fbb0 0x7f93540afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.699+0000 7f93537fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9354145040 0x7f93540d0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59558/0 (socket says 172.21.15.3:59558) 2024-05-08T21:22:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.699+0000 7f9353fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f935415fbb0 0x7f93540afb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44216/0 (socket says 172.21.15.3:44216) 2024-05-08T21:22:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.699+0000 7f93537fe700 1 -- 172.21.15.3:0/1543400610 learned_addr learned my addr 172.21.15.3:0/1543400610 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.699+0000 7f9352ffd700 1 --2- 172.21.15.3:0/1543400610 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93540d0a60 0x7f93540b5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.699+0000 7f9353fff700 1 -- 172.21.15.3:0/1543400610 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93540d0a60 msgr2=0x7f93540b5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.699+0000 7f9353fff700 1 --2- 172.21.15.3:0/1543400610 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93540d0a60 0x7f93540b5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.699+0000 7f9353fff700 1 -- 172.21.15.3:0/1543400610 >> v1:172.21.15.3:6789/0 conn(0x7f9354145040 legacy=0x7f93540d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.699+0000 7f9353fff700 1 -- 172.21.15.3:0/1543400610 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93540aedd0 con 0x7f935415fbb0 2024-05-08T21:22:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.699+0000 7f93527fc700 1 -- 172.21.15.3:0/1543400610 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983038490 0 0) 0x7f93540af6e0 con 0x7f9354145040 2024-05-08T21:22:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.699+0000 7f9353fff700 1 --2- 172.21.15.3:0/1543400610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f935415fbb0 0x7f93540afb70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f934c009d90 tx=0x7f934c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50a1d4201e90890 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.700+0000 7f93527fc700 1 -- 172.21.15.3:0/1543400610 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c00e070 con 0x7f935415fbb0 2024-05-08T21:22:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.700+0000 7f93527fc700 1 -- 172.21.15.3:0/1543400610 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c00bbd0 con 0x7f935415fbb0 2024-05-08T21:22:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.700+0000 7f93527fc700 1 -- 172.21.15.3:0/1543400610 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c012430 con 0x7f935415fbb0 2024-05-08T21:22:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.700+0000 7f9359cdf700 1 -- 172.21.15.3:0/1543400610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f935415fbb0 msgr2=0x7f93540afb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.700+0000 7f9359cdf700 1 --2- 172.21.15.3:0/1543400610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f935415fbb0 0x7f93540afb70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f934c009d90 tx=0x7f934c00b330 comp rx=0 tx=0).stop 2024-05-08T21:22:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.700+0000 7f9359cdf700 1 -- 172.21.15.3:0/1543400610 shutdown_connections 2024-05-08T21:22:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.700+0000 7f9359cdf700 1 --2- 172.21.15.3:0/1543400610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f935415fbb0 0x7f93540afb70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.700+0000 7f9359cdf700 1 --2- 172.21.15.3:0/1543400610 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93540d0a60 0x7f93540b5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.700+0000 7f9359cdf700 1 -- 172.21.15.3:0/1543400610 >> 172.21.15.3:0/1543400610 conn(0x7f9354099a50 msgr2=0x7f93540b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:30.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.700+0000 7f9359cdf700 1 -- 172.21.15.3:0/1543400610 shutdown_connections 2024-05-08T21:22:30.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.700+0000 7f9359cdf700 1 -- 172.21.15.3:0/1543400610 wait complete. 2024-05-08T21:22:30.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.701+0000 7f9359cdf700 1 Processor -- start 2024-05-08T21:22:30.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.702+0000 7f9359cdf700 1 -- start start 2024-05-08T21:22:30.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.702+0000 7f9359cdf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93540d0a60 0x7f935413b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:30.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.702+0000 7f9359cdf700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9354145040 0x7f935415e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:30.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f9359cdf700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f935415fbb0 0x7f93540b0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:30.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f9359cdf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935413bcf0 con 0x7f93540d0a60 2024-05-08T21:22:30.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f9359cdf700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93540cd9f0 con 0x7f9354145040 2024-05-08T21:22:30.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f9359cdf700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93540ccde0 con 0x7f935415fbb0 2024-05-08T21:22:30.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f93537fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93540d0a60 0x7f935413b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:30.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f93537fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93540d0a60 0x7f935413b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44228/0 (socket says 172.21.15.3:44228) 2024-05-08T21:22:30.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f93537fe700 1 -- 172.21.15.3:0/1897091960 learned_addr learned my addr 172.21.15.3:0/1897091960 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:30.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f9353fff700 1 --2- 172.21.15.3:0/1897091960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f935415fbb0 0x7f93540b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:30.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f9352ffd700 1 --2- 172.21.15.3:0/1897091960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9354145040 0x7f935415e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:30.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f93537fe700 1 -- 172.21.15.3:0/1897091960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f935415fbb0 msgr2=0x7f93540b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:30.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f93537fe700 1 --2- 172.21.15.3:0/1897091960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f935415fbb0 0x7f93540b0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f93537fe700 1 -- 172.21.15.3:0/1897091960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9354145040 msgr2=0x7f935415e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:30.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f93537fe700 1 --2- 172.21.15.3:0/1897091960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9354145040 0x7f935415e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f93537fe700 1 -- 172.21.15.3:0/1897091960 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c009a70 con 0x7f93540d0a60 2024-05-08T21:22:30.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f93537fe700 1 --2- 172.21.15.3:0/1897091960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93540d0a60 0x7f935413b7b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f933c0011e0 tx=0x7f933c00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:30.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f934bfff700 1 -- 172.21.15.3:0/1897091960 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933c004c80 con 0x7f93540d0a60 2024-05-08T21:22:30.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f934bfff700 1 -- 172.21.15.3:0/1897091960 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933c004e00 con 0x7f93540d0a60 2024-05-08T21:22:30.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f9359cdf700 1 -- 172.21.15.3:0/1897091960 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93540cd0c0 con 0x7f93540d0a60 2024-05-08T21:22:30.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f934bfff700 1 -- 172.21.15.3:0/1897091960 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933c007310 con 0x7f93540d0a60 2024-05-08T21:22:30.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.703+0000 7f9359cdf700 1 -- 172.21.15.3:0/1897091960 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93540cd680 con 0x7f93540d0a60 2024-05-08T21:22:30.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.704+0000 7f9359cdf700 1 -- 172.21.15.3:0/1897091960 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9354074bc0 con 0x7f93540d0a60 2024-05-08T21:22:30.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.705+0000 7f934bfff700 1 -- 172.21.15.3:0/1897091960 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f933c007470 con 0x7f93540d0a60 2024-05-08T21:22:30.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.705+0000 7f934bfff700 1 --2- 172.21.15.3:0/1897091960 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f933406ac90 0x7f933406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:30.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.705+0000 7f934bfff700 1 -- 172.21.15.3:0/1897091960 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f933c088fc0 con 0x7f93540d0a60 2024-05-08T21:22:30.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.706+0000 7f9352ffd700 1 --2- 172.21.15.3:0/1897091960 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f933406ac90 0x7f933406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:30.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.706+0000 7f9352ffd700 1 --2- 172.21.15.3:0/1897091960 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f933406ac90 0x7f933406d150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f935415fae0 tx=0x7f9344008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:30.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.710+0000 7f934bfff700 1 -- 172.21.15.3:0/1897091960 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f933c054100 con 0x7f93540d0a60 2024-05-08T21:22:30.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.925+0000 7f9359cdf700 1 -- 172.21.15.3:0/1897091960 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9354144860 con 0x7f93540d0a60 2024-05-08T21:22:30.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.925+0000 7f934bfff700 1 -- 172.21.15.3:0/1897091960 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f933c057720 con 0x7f93540d0a60 2024-05-08T21:22:30.927 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:22:30.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.927+0000 7f9359cdf700 1 -- 172.21.15.3:0/1897091960 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f933406ac90 msgr2=0x7f933406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:30.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.927+0000 7f9359cdf700 1 --2- 172.21.15.3:0/1897091960 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f933406ac90 0x7f933406d150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f935415fae0 tx=0x7f9344008040 comp rx=0 tx=0).stop 2024-05-08T21:22:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.927+0000 7f9359cdf700 1 -- 172.21.15.3:0/1897091960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93540d0a60 msgr2=0x7f935413b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.927+0000 7f9359cdf700 1 --2- 172.21.15.3:0/1897091960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93540d0a60 0x7f935413b7b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f933c0011e0 tx=0x7f933c00ca30 comp rx=0 tx=0).stop 2024-05-08T21:22:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.927+0000 7f9359cdf700 1 -- 172.21.15.3:0/1897091960 shutdown_connections 2024-05-08T21:22:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.927+0000 7f9359cdf700 1 --2- 172.21.15.3:0/1897091960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f935415fbb0 0x7f93540b0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.927+0000 7f9359cdf700 1 --2- 172.21.15.3:0/1897091960 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f933406ac90 0x7f933406d150 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.927+0000 7f9359cdf700 1 --2- 172.21.15.3:0/1897091960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93540d0a60 0x7f935413b7b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.927+0000 7f9359cdf700 1 --2- 172.21.15.3:0/1897091960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9354145040 0x7f935415e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.927+0000 7f9359cdf700 1 -- 172.21.15.3:0/1897091960 >> 172.21.15.3:0/1897091960 conn(0x7f9354099a50 msgr2=0x7f93540d0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.928+0000 7f9359cdf700 1 -- 172.21.15.3:0/1897091960 shutdown_connections 2024-05-08T21:22:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:30.928+0000 7f9359cdf700 1 -- 172.21.15.3:0/1897091960 wait complete. 2024-05-08T21:22:30.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:22:31.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.080+0000 7f542af1c700 1 Processor -- start 2024-05-08T21:22:31.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.080+0000 7f542af1c700 1 -- start start 2024-05-08T21:22:31.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.081+0000 7f542af1c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f54240b0a80 0x7f54240cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:31.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.081+0000 7f542af1c700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54240d0120 con 0x7f542409a160 2024-05-08T21:22:31.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.081+0000 7f542af1c700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54240d0400 con 0x7f542409aa40 2024-05-08T21:22:31.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.081+0000 7f542af1c700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54240d06e0 con 0x7f54240b0a80 2024-05-08T21:22:31.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.081+0000 7f54294b9700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f542409a160 0x7f542415d120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59570/0 (socket says 172.21.15.3:59570) 2024-05-08T21:22:31.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.081+0000 7f54294b9700 1 -- 172.21.15.3:0/4170540909 learned_addr learned my addr 172.21.15.3:0/4170540909 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:31.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.081+0000 7f5428cb8700 1 --2- 172.21.15.3:0/4170540909 >> v2:172.21.15.186:3300/0 conn(0x7f54240b0a80 0x7f54240cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:31.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.082+0000 7f54237fe700 1 -- 172.21.15.3:0/4170540909 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018157385 0 0) 0x7f54240d0120 con 0x7f542409a160 2024-05-08T21:22:31.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.082+0000 7f54237fe700 1 -- 172.21.15.3:0/4170540909 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54100035b0 con 0x7f542409a160 2024-05-08T21:22:31.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.082+0000 7f5428cb8700 1 -- 172.21.15.3:0/4170540909 >> v1:172.21.15.186:6789/0 conn(0x7f542409aa40 legacy=0x7f5424099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:31.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.082+0000 7f5428cb8700 1 -- 172.21.15.3:0/4170540909 >> v1:172.21.15.3:6789/0 conn(0x7f542409a160 legacy=0x7f542415d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.082+0000 7f5428cb8700 1 -- 172.21.15.3:0/4170540909 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54240d09c0 con 0x7f54240b0a80 2024-05-08T21:22:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.082+0000 7f5428cb8700 1 --2- 172.21.15.3:0/4170540909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54240b0a80 0x7f54240cf6b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5414009840 tx=0x7f5414009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec6b43cf3aba171 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.082+0000 7f54237fe700 1 -- 172.21.15.3:0/4170540909 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414011830 con 0x7f54240b0a80 2024-05-08T21:22:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.082+0000 7f54237fe700 1 -- 172.21.15.3:0/4170540909 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5414011990 con 0x7f54240b0a80 2024-05-08T21:22:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.082+0000 7f54237fe700 1 -- 172.21.15.3:0/4170540909 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414011d20 con 0x7f54240b0a80 2024-05-08T21:22:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.083+0000 7f542af1c700 1 -- 172.21.15.3:0/4170540909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54240b0a80 msgr2=0x7f54240cf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.083+0000 7f542af1c700 1 --2- 172.21.15.3:0/4170540909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54240b0a80 0x7f54240cf6b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5414009840 tx=0x7f5414009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.083+0000 7f542af1c700 1 -- 172.21.15.3:0/4170540909 shutdown_connections 2024-05-08T21:22:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.083+0000 7f542af1c700 1 --2- 172.21.15.3:0/4170540909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54240b0a80 0x7f54240cf6b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.083+0000 7f542af1c700 1 -- 172.21.15.3:0/4170540909 >> 172.21.15.3:0/4170540909 conn(0x7f54240b2fa0 msgr2=0x7f54240b33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.083+0000 7f542af1c700 1 -- 172.21.15.3:0/4170540909 shutdown_connections 2024-05-08T21:22:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.083+0000 7f542af1c700 1 -- 172.21.15.3:0/4170540909 wait complete. 2024-05-08T21:22:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.084+0000 7f542af1c700 1 Processor -- start 2024-05-08T21:22:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.084+0000 7f542af1c700 1 -- start start 2024-05-08T21:22:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.085+0000 7f542af1c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f542409a160 0x7f5424157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.085+0000 7f542af1c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f542409aa40 0x7f54240ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.085+0000 7f542af1c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54240b0a80 0x7f54240af6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.085+0000 7f542af1c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54241f3a90 con 0x7f54240b0a80 2024-05-08T21:22:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.085+0000 7f542af1c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54240b4f10 con 0x7f542409a160 2024-05-08T21:22:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.085+0000 7f542af1c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54240b4390 con 0x7f542409aa40 2024-05-08T21:22:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.085+0000 7f54294b9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54240b0a80 0x7f54240af6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.085+0000 7f54294b9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54240b0a80 0x7f54240af6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44244/0 (socket says 172.21.15.3:44244) 2024-05-08T21:22:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.085+0000 7f54294b9700 1 -- 172.21.15.3:0/2540100778 learned_addr learned my addr 172.21.15.3:0/2540100778 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.086+0000 7f5428cb8700 1 --2- 172.21.15.3:0/2540100778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f542409a160 0x7f5424157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.086+0000 7f5423fff700 1 --2- 172.21.15.3:0/2540100778 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f542409aa40 0x7f54240ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.086+0000 7f54294b9700 1 -- 172.21.15.3:0/2540100778 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f542409aa40 msgr2=0x7f54240ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.086+0000 7f54294b9700 1 --2- 172.21.15.3:0/2540100778 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f542409aa40 0x7f54240ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.086+0000 7f54294b9700 1 -- 172.21.15.3:0/2540100778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f542409a160 msgr2=0x7f5424157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.086+0000 7f54294b9700 1 --2- 172.21.15.3:0/2540100778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f542409a160 0x7f5424157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.086+0000 7f54294b9700 1 -- 172.21.15.3:0/2540100778 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54140094a0 con 0x7f54240b0a80 2024-05-08T21:22:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.086+0000 7f5428cb8700 1 --2- 172.21.15.3:0/2540100778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f542409a160 0x7f5424157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.086+0000 7f54294b9700 1 --2- 172.21.15.3:0/2540100778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54240b0a80 0x7f54240af6d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f541c00cad0 tx=0x7f541c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.086+0000 7f54217fa700 1 -- 172.21.15.3:0/2540100778 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c00aa60 con 0x7f54240b0a80 2024-05-08T21:22:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.086+0000 7f54217fa700 1 -- 172.21.15.3:0/2540100778 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541c00abc0 con 0x7f54240b0a80 2024-05-08T21:22:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.086+0000 7f542af1c700 1 -- 172.21.15.3:0/2540100778 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54240b4670 con 0x7f54240b0a80 2024-05-08T21:22:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.086+0000 7f54217fa700 1 -- 172.21.15.3:0/2540100778 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c012400 con 0x7f54240b0a80 2024-05-08T21:22:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.086+0000 7f542af1c700 1 -- 172.21.15.3:0/2540100778 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54240b4c30 con 0x7f54240b0a80 2024-05-08T21:22:31.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.087+0000 7f542af1c700 1 -- 172.21.15.3:0/2540100778 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54241f3c00 con 0x7f54240b0a80 2024-05-08T21:22:31.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.088+0000 7f54217fa700 1 -- 172.21.15.3:0/2540100778 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f541c01b3b0 con 0x7f54240b0a80 2024-05-08T21:22:31.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.088+0000 7f54217fa700 1 --2- 172.21.15.3:0/2540100778 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f540406ac90 0x7f540406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:31.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.088+0000 7f54217fa700 1 -- 172.21.15.3:0/2540100778 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f541c088fa0 con 0x7f54240b0a80 2024-05-08T21:22:31.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.088+0000 7f5428cb8700 1 --2- 172.21.15.3:0/2540100778 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f540406ac90 0x7f540406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:31.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.089+0000 7f5428cb8700 1 --2- 172.21.15.3:0/2540100778 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f540406ac90 0x7f540406d150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f54140090b0 tx=0x7f5414008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:31.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.093+0000 7f54217fa700 1 -- 172.21.15.3:0/2540100778 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f541c0bc900 con 0x7f54240b0a80 2024-05-08T21:22:31.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.310+0000 7f542af1c700 1 -- 172.21.15.3:0/2540100778 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5424074bc0 con 0x7f54240b0a80 2024-05-08T21:22:31.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.310+0000 7f54217fa700 1 -- 172.21.15.3:0/2540100778 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f541c017020 con 0x7f54240b0a80 2024-05-08T21:22:31.312 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:22:31.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.312+0000 7f542af1c700 1 -- 172.21.15.3:0/2540100778 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f540406ac90 msgr2=0x7f540406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:31.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.312+0000 7f542af1c700 1 --2- 172.21.15.3:0/2540100778 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f540406ac90 0x7f540406d150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f54140090b0 tx=0x7f5414008030 comp rx=0 tx=0).stop 2024-05-08T21:22:31.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.312+0000 7f542af1c700 1 -- 172.21.15.3:0/2540100778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54240b0a80 msgr2=0x7f54240af6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:31.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.312+0000 7f542af1c700 1 --2- 172.21.15.3:0/2540100778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54240b0a80 0x7f54240af6d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f541c00cad0 tx=0x7f541c00cde0 comp rx=0 tx=0).stop 2024-05-08T21:22:31.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.313+0000 7f542af1c700 1 -- 172.21.15.3:0/2540100778 shutdown_connections 2024-05-08T21:22:31.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.313+0000 7f542af1c700 1 --2- 172.21.15.3:0/2540100778 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f540406ac90 0x7f540406d150 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:31.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.313+0000 7f542af1c700 1 --2- 172.21.15.3:0/2540100778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54240b0a80 0x7f54240af6d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:31.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.313+0000 7f542af1c700 1 --2- 172.21.15.3:0/2540100778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f542409a160 0x7f5424157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:31.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.313+0000 7f542af1c700 1 --2- 172.21.15.3:0/2540100778 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f542409aa40 0x7f54240ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:31.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.313+0000 7f542af1c700 1 -- 172.21.15.3:0/2540100778 >> 172.21.15.3:0/2540100778 conn(0x7f54240b2fa0 msgr2=0x7f54240cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:31.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.313+0000 7f542af1c700 1 -- 172.21.15.3:0/2540100778 shutdown_connections 2024-05-08T21:22:31.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.313+0000 7f542af1c700 1 -- 172.21.15.3:0/2540100778 wait complete. 2024-05-08T21:22:31.328 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-08T21:22:31.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.463+0000 7f1788523700 1 Processor -- start 2024-05-08T21:22:31.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.463+0000 7f1788523700 1 -- start start 2024-05-08T21:22:31.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.464+0000 7f1788523700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f178006b490 0x7f178006cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:31.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.464+0000 7f1788523700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178015bf20 con 0x7f1780155dd0 2024-05-08T21:22:31.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.464+0000 7f1788523700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178015c200 con 0x7f17801559f0 2024-05-08T21:22:31.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.464+0000 7f1788523700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f178015c4e0 con 0x7f178006b490 2024-05-08T21:22:31.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.464+0000 7f1786ac0700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f178006b490 0x7f178006cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:31.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.464+0000 7f1785abe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1780155dd0 0x7f178006ad80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47988/0 (socket says 172.21.15.3:47988) 2024-05-08T21:22:31.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.464+0000 7f1785abe700 1 -- 172.21.15.3:0/3489520585 learned_addr learned my addr 172.21.15.3:0/3489520585 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:31.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.465+0000 7f1786ac0700 1 -- 172.21.15.3:0/3489520585 >> v1:172.21.15.186:6789/0 conn(0x7f17801559f0 legacy=0x7f17800ae2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:31.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.465+0000 7f1786ac0700 1 -- 172.21.15.3:0/3489520585 >> v1:172.21.15.3:6789/0 conn(0x7f1780155dd0 legacy=0x7f178006ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:31.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.465+0000 7f1786ac0700 1 -- 172.21.15.3:0/3489520585 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178015c7c0 con 0x7f178006b490 2024-05-08T21:22:31.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.465+0000 7f17852bd700 1 -- 172.21.15.3:0/3489520585 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270640874 0 0) 0x7f178015bf20 con 0x7f1780155dd0 2024-05-08T21:22:31.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.465+0000 7f1786ac0700 1 --2- 172.21.15.3:0/3489520585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f178006b490 0x7f178006cb20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1778009dc0 tx=0x7f177800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65a713b9e1989a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:31.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.465+0000 7f17852bd700 1 -- 172.21.15.3:0/3489520585 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177800e070 con 0x7f178006b490 2024-05-08T21:22:31.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.465+0000 7f17852bd700 1 -- 172.21.15.3:0/3489520585 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177800bb90 con 0x7f178006b490 2024-05-08T21:22:31.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.465+0000 7f17852bd700 1 -- 172.21.15.3:0/3489520585 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1778012430 con 0x7f178006b490 2024-05-08T21:22:31.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.466+0000 7f1788523700 1 -- 172.21.15.3:0/3489520585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f178006b490 msgr2=0x7f178006cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:31.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.466+0000 7f1788523700 1 --2- 172.21.15.3:0/3489520585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f178006b490 0x7f178006cb20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1778009dc0 tx=0x7f177800b330 comp rx=0 tx=0).stop 2024-05-08T21:22:31.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.466+0000 7f1788523700 1 -- 172.21.15.3:0/3489520585 shutdown_connections 2024-05-08T21:22:31.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.466+0000 7f1788523700 1 --2- 172.21.15.3:0/3489520585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f178006b490 0x7f178006cb20 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:31.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.466+0000 7f1788523700 1 -- 172.21.15.3:0/3489520585 >> 172.21.15.3:0/3489520585 conn(0x7f17800b1290 msgr2=0x7f17800b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:31.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.466+0000 7f1788523700 1 -- 172.21.15.3:0/3489520585 shutdown_connections 2024-05-08T21:22:31.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.466+0000 7f1788523700 1 -- 172.21.15.3:0/3489520585 wait complete. 2024-05-08T21:22:31.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.467+0000 7f1788523700 1 Processor -- start 2024-05-08T21:22:31.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.467+0000 7f1788523700 1 -- start start 2024-05-08T21:22:31.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.468+0000 7f1788523700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f178006b490 0x7f17801f1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:31.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.468+0000 7f1788523700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17801559f0 0x7f178009cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:31.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.468+0000 7f1788523700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1780155dd0 0x7f178014ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:31.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.468+0000 7f1788523700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178015cd80 con 0x7f1780155dd0 2024-05-08T21:22:31.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.468+0000 7f1788523700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178009cab0 con 0x7f178006b490 2024-05-08T21:22:31.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.468+0000 7f1788523700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17801f1780 con 0x7f17801559f0 2024-05-08T21:22:31.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.468+0000 7f1786ac0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1780155dd0 0x7f178014ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:31.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.468+0000 7f1786ac0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1780155dd0 0x7f178014ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51242/0 (socket says 172.21.15.3:51242) 2024-05-08T21:22:31.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.468+0000 7f1786ac0700 1 -- 172.21.15.3:0/910350937 learned_addr learned my addr 172.21.15.3:0/910350937 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:31.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.468+0000 7f1785abe700 1 --2- 172.21.15.3:0/910350937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17801559f0 0x7f178009cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:31.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.468+0000 7f17862bf700 1 --2- 172.21.15.3:0/910350937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f178006b490 0x7f17801f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:31.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.468+0000 7f1786ac0700 1 -- 172.21.15.3:0/910350937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17801559f0 msgr2=0x7f178009cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:31.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.468+0000 7f1786ac0700 1 --2- 172.21.15.3:0/910350937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17801559f0 0x7f178009cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:31.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.468+0000 7f1786ac0700 1 -- 172.21.15.3:0/910350937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f178006b490 msgr2=0x7f17801f1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:31.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.468+0000 7f1786ac0700 1 --2- 172.21.15.3:0/910350937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f178006b490 0x7f17801f1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:31.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.468+0000 7f1786ac0700 1 -- 172.21.15.3:0/910350937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1778009a70 con 0x7f1780155dd0 2024-05-08T21:22:31.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.469+0000 7f1785abe700 1 --2- 172.21.15.3:0/910350937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17801559f0 0x7f178009cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:31.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.469+0000 7f17862bf700 1 --2- 172.21.15.3:0/910350937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f178006b490 0x7f17801f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:31.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.469+0000 7f1786ac0700 1 --2- 172.21.15.3:0/910350937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1780155dd0 0x7f178014ebd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1778014040 tx=0x7f1778004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:31.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.469+0000 7f1776ffd700 1 -- 172.21.15.3:0/910350937 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177801b960 con 0x7f1780155dd0 2024-05-08T21:22:31.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.469+0000 7f1776ffd700 1 -- 172.21.15.3:0/910350937 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1778020070 con 0x7f1780155dd0 2024-05-08T21:22:31.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.469+0000 7f1776ffd700 1 -- 172.21.15.3:0/910350937 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177801bd00 con 0x7f1780155dd0 2024-05-08T21:22:31.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.469+0000 7f1788523700 1 -- 172.21.15.3:0/910350937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17801f1a60 con 0x7f1780155dd0 2024-05-08T21:22:31.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.469+0000 7f1788523700 1 -- 172.21.15.3:0/910350937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17801f2020 con 0x7f1780155dd0 2024-05-08T21:22:31.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.470+0000 7f1788523700 1 -- 172.21.15.3:0/910350937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f178015cef0 con 0x7f1780155dd0 2024-05-08T21:22:31.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.470+0000 7f1776ffd700 1 -- 172.21.15.3:0/910350937 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1778026400 con 0x7f1780155dd0 2024-05-08T21:22:31.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.471+0000 7f1776ffd700 1 --2- 172.21.15.3:0/910350937 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f176406ac90 0x7f176406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:31.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.471+0000 7f1776ffd700 1 -- 172.21.15.3:0/910350937 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f177808b220 con 0x7f1780155dd0 2024-05-08T21:22:31.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.471+0000 7f17862bf700 1 --2- 172.21.15.3:0/910350937 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f176406ac90 0x7f176406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:31.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.472+0000 7f17862bf700 1 --2- 172.21.15.3:0/910350937 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f176406ac90 0x7f176406d150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f177c000f60 tx=0x7f177c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:31.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.475+0000 7f1776ffd700 1 -- 172.21.15.3:0/910350937 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f1778056490 con 0x7f1780155dd0 2024-05-08T21:22:31.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.714+0000 7f1788523700 1 -- 172.21.15.3:0/910350937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f17800678f0 con 0x7f1780155dd0 2024-05-08T21:22:31.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.714+0000 7f1776ffd700 1 -- 172.21.15.3:0/910350937 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f177801be60 con 0x7f1780155dd0 2024-05-08T21:22:31.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.717+0000 7f1788523700 1 -- 172.21.15.3:0/910350937 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f176406ac90 msgr2=0x7f176406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:31.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.717+0000 7f1788523700 1 --2- 172.21.15.3:0/910350937 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f176406ac90 0x7f176406d150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f177c000f60 tx=0x7f177c009040 comp rx=0 tx=0).stop 2024-05-08T21:22:31.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.717+0000 7f1788523700 1 -- 172.21.15.3:0/910350937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1780155dd0 msgr2=0x7f178014ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:31.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.717+0000 7f1788523700 1 --2- 172.21.15.3:0/910350937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1780155dd0 0x7f178014ebd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1778014040 tx=0x7f1778004ad0 comp rx=0 tx=0).stop 2024-05-08T21:22:31.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.717+0000 7f1788523700 1 -- 172.21.15.3:0/910350937 shutdown_connections 2024-05-08T21:22:31.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.717+0000 7f1788523700 1 --2- 172.21.15.3:0/910350937 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f176406ac90 0x7f176406d150 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:31.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.717+0000 7f1788523700 1 --2- 172.21.15.3:0/910350937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1780155dd0 0x7f178014ebd0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:31.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.717+0000 7f1788523700 1 --2- 172.21.15.3:0/910350937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f178006b490 0x7f17801f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:31.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.717+0000 7f1788523700 1 --2- 172.21.15.3:0/910350937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17801559f0 0x7f178009cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:31.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.717+0000 7f1788523700 1 -- 172.21.15.3:0/910350937 >> 172.21.15.3:0/910350937 conn(0x7f17800b1290 msgr2=0x7f178006d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:31.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.717+0000 7f1788523700 1 -- 172.21.15.3:0/910350937 shutdown_connections 2024-05-08T21:22:31.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:31.717+0000 7f1788523700 1 -- 172.21.15.3:0/910350937 wait complete. 2024-05-08T21:22:31.720 INFO:teuthology.orchestra.run.smithi003.stderr:listed 0 entries 2024-05-08T21:22:31.732 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-05-08T21:22:31.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:31.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-05-08T21:22:31.773 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi003.front.sepia.ceph.com 2024-05-08T21:22:31.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:31.774 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-05-08T21:22:31.834 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:22:31.834 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:31.856 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:22:31.856 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:31.917 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:22:31.918 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:31.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:32.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.123+0000 7f9e491b2700 1 Processor -- start 2024-05-08T21:22:32.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.123+0000 7f9e491b2700 1 -- start start 2024-05-08T21:22:32.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.124+0000 7f9e491b2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9e440aef20 0x7f9e4415c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.124+0000 7f9e491b2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e4415f230 0x7f9e440cc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.124+0000 7f9e491b2700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e440cd3b0 con 0x7f9e440cca70 2024-05-08T21:22:32.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.124+0000 7f9e491b2700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e44141df0 con 0x7f9e440aef20 2024-05-08T21:22:32.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.124+0000 7f9e491b2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e440ccf00 con 0x7f9e4415f230 2024-05-08T21:22:32.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.124+0000 7f9e42d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9e440aef20 0x7f9e4415c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.124+0000 7f9e42d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9e440aef20 0x7f9e4415c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51258/0 (socket says 172.21.15.3:51258) 2024-05-08T21:22:32.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.124+0000 7f9e42d9d700 1 -- 172.21.15.3:0/1255001012 learned_addr learned my addr 172.21.15.3:0/1255001012 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:32.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.124+0000 7f9e4259c700 1 --2- 172.21.15.3:0/1255001012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e4415f230 0x7f9e440cc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.124+0000 7f9e42d9d700 1 -- 172.21.15.3:0/1255001012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e4415f230 msgr2=0x7f9e440cc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.124+0000 7f9e42d9d700 1 --2- 172.21.15.3:0/1255001012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e4415f230 0x7f9e440cc530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.124+0000 7f9e42d9d700 1 -- 172.21.15.3:0/1255001012 >> v1:172.21.15.186:6789/0 conn(0x7f9e440cca70 legacy=0x7f9e440ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.124+0000 7f9e42d9d700 1 -- 172.21.15.3:0/1255001012 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e440af300 con 0x7f9e440aef20 2024-05-08T21:22:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.124+0000 7f9e4259c700 1 --2- 172.21.15.3:0/1255001012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e4415f230 0x7f9e440cc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.124+0000 7f9e42d9d700 1 --2- 172.21.15.3:0/1255001012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e440aef20 0x7f9e4415c670 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9e340098f0 tx=0x7f9e34009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59f052adfa04d83 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.125+0000 7f9e41d9b700 1 -- 172.21.15.3:0/1255001012 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e34004230 con 0x7f9e440aef20 2024-05-08T21:22:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.125+0000 7f9e41d9b700 1 -- 172.21.15.3:0/1255001012 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e34004390 con 0x7f9e440aef20 2024-05-08T21:22:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.125+0000 7f9e41d9b700 1 -- 172.21.15.3:0/1255001012 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e340036a0 con 0x7f9e440aef20 2024-05-08T21:22:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.125+0000 7f9e491b2700 1 -- 172.21.15.3:0/1255001012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e440aef20 msgr2=0x7f9e4415c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.125+0000 7f9e491b2700 1 --2- 172.21.15.3:0/1255001012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e440aef20 0x7f9e4415c670 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9e340098f0 tx=0x7f9e34009c00 comp rx=0 tx=0).stop 2024-05-08T21:22:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.125+0000 7f9e491b2700 1 -- 172.21.15.3:0/1255001012 shutdown_connections 2024-05-08T21:22:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.125+0000 7f9e491b2700 1 --2- 172.21.15.3:0/1255001012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e440aef20 0x7f9e4415c670 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.125+0000 7f9e491b2700 1 --2- 172.21.15.3:0/1255001012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e4415f230 0x7f9e440cc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.125+0000 7f9e491b2700 1 -- 172.21.15.3:0/1255001012 >> 172.21.15.3:0/1255001012 conn(0x7f9e440cfce0 msgr2=0x7f9e440d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.125+0000 7f9e491b2700 1 -- 172.21.15.3:0/1255001012 shutdown_connections 2024-05-08T21:22:32.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.125+0000 7f9e491b2700 1 -- 172.21.15.3:0/1255001012 wait complete. 2024-05-08T21:22:32.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.127+0000 7f9e491b2700 1 Processor -- start 2024-05-08T21:22:32.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.127+0000 7f9e491b2700 1 -- start start 2024-05-08T21:22:32.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e491b2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e440aef20 0x7f9e44156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e491b2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e440cca70 0x7f9e44140590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e491b2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e4415f230 0x7f9e44158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e491b2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e440b33c0 con 0x7f9e4415f230 2024-05-08T21:22:32.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e491b2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e44140e80 con 0x7f9e440cca70 2024-05-08T21:22:32.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e491b2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e44141160 con 0x7f9e440aef20 2024-05-08T21:22:32.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e4359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e4415f230 0x7f9e44158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e4359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e4415f230 0x7f9e44158520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51262/0 (socket says 172.21.15.3:51262) 2024-05-08T21:22:32.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e4359e700 1 -- 172.21.15.3:0/1012108067 learned_addr learned my addr 172.21.15.3:0/1012108067 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:32.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e42d9d700 1 --2- 172.21.15.3:0/1012108067 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e440aef20 0x7f9e44156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e4259c700 1 --2- 172.21.15.3:0/1012108067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e440cca70 0x7f9e44140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e4359e700 1 -- 172.21.15.3:0/1012108067 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e440aef20 msgr2=0x7f9e44156d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:32.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e4359e700 1 --2- 172.21.15.3:0/1012108067 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e440aef20 0x7f9e44156d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e4359e700 1 -- 172.21.15.3:0/1012108067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e440cca70 msgr2=0x7f9e44140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:32.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e4359e700 1 --2- 172.21.15.3:0/1012108067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e440cca70 0x7f9e44140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e4359e700 1 -- 172.21.15.3:0/1012108067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e34009550 con 0x7f9e4415f230 2024-05-08T21:22:32.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.128+0000 7f9e4359e700 1 --2- 172.21.15.3:0/1012108067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e4415f230 0x7f9e44158520 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c00cc10 tx=0x7f9e3c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:32.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.129+0000 7f9e3b7fe700 1 -- 172.21.15.3:0/1012108067 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c00aa60 con 0x7f9e4415f230 2024-05-08T21:22:32.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.129+0000 7f9e3b7fe700 1 -- 172.21.15.3:0/1012108067 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3c00abc0 con 0x7f9e4415f230 2024-05-08T21:22:32.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.129+0000 7f9e491b2700 1 -- 172.21.15.3:0/1012108067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e440b3660 con 0x7f9e4415f230 2024-05-08T21:22:32.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.129+0000 7f9e3b7fe700 1 -- 172.21.15.3:0/1012108067 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c012420 con 0x7f9e4415f230 2024-05-08T21:22:32.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.129+0000 7f9e491b2700 1 -- 172.21.15.3:0/1012108067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e441572c0 con 0x7f9e4415f230 2024-05-08T21:22:32.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.130+0000 7f9e491b2700 1 -- 172.21.15.3:0/1012108067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e440b3b80 con 0x7f9e4415f230 2024-05-08T21:22:32.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.130+0000 7f9e3b7fe700 1 -- 172.21.15.3:0/1012108067 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9e3c012580 con 0x7f9e4415f230 2024-05-08T21:22:32.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.131+0000 7f9e3b7fe700 1 --2- 172.21.15.3:0/1012108067 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9e2406ac90 0x7f9e2406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.131+0000 7f9e3b7fe700 1 -- 172.21.15.3:0/1012108067 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e3c0895f0 con 0x7f9e4415f230 2024-05-08T21:22:32.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.131+0000 7f9e42d9d700 1 --2- 172.21.15.3:0/1012108067 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9e2406ac90 0x7f9e2406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.132+0000 7f9e42d9d700 1 --2- 172.21.15.3:0/1012108067 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9e2406ac90 0x7f9e2406d150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9e34009130 tx=0x7f9e34008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:32.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.135+0000 7f9e3b7fe700 1 -- 172.21.15.3:0/1012108067 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9e3c0547d0 con 0x7f9e4415f230 2024-05-08T21:22:32.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.356+0000 7f9e491b2700 1 -- 172.21.15.3:0/1012108067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e44068000 con 0x7f9e4415f230 2024-05-08T21:22:32.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.357+0000 7f9e3b7fe700 1 -- 172.21.15.3:0/1012108067 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15989 (secure 0 0 0) 0x7f9e3c057df0 con 0x7f9e4415f230 2024-05-08T21:22:32.359 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:32.359 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:21:53.064342+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:32.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.360+0000 7f9e491b2700 1 -- 172.21.15.3:0/1012108067 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9e2406ac90 msgr2=0x7f9e2406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:32.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.360+0000 7f9e491b2700 1 --2- 172.21.15.3:0/1012108067 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9e2406ac90 0x7f9e2406d150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9e34009130 tx=0x7f9e34008030 comp rx=0 tx=0).stop 2024-05-08T21:22:32.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.360+0000 7f9e491b2700 1 -- 172.21.15.3:0/1012108067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e4415f230 msgr2=0x7f9e44158520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:32.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.360+0000 7f9e491b2700 1 --2- 172.21.15.3:0/1012108067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e4415f230 0x7f9e44158520 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c00cc10 tx=0x7f9e3c00cf20 comp rx=0 tx=0).stop 2024-05-08T21:22:32.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.360+0000 7f9e491b2700 1 -- 172.21.15.3:0/1012108067 shutdown_connections 2024-05-08T21:22:32.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.360+0000 7f9e491b2700 1 --2- 172.21.15.3:0/1012108067 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e440aef20 0x7f9e44156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.360+0000 7f9e491b2700 1 --2- 172.21.15.3:0/1012108067 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9e2406ac90 0x7f9e2406d150 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.360+0000 7f9e491b2700 1 --2- 172.21.15.3:0/1012108067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e4415f230 0x7f9e44158520 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.360+0000 7f9e491b2700 1 --2- 172.21.15.3:0/1012108067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e440cca70 0x7f9e44140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.360+0000 7f9e491b2700 1 -- 172.21.15.3:0/1012108067 >> 172.21.15.3:0/1012108067 conn(0x7f9e440cfce0 msgr2=0x7f9e440b0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:32.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.360+0000 7f9e491b2700 1 -- 172.21.15.3:0/1012108067 shutdown_connections 2024-05-08T21:22:32.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.360+0000 7f9e491b2700 1 -- 172.21.15.3:0/1012108067 wait complete. 2024-05-08T21:22:32.375 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-08T21:21:49.638998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:21:51.024632+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:22:32.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:22:32.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.514+0000 7fab18db7700 1 Processor -- start 2024-05-08T21:22:32.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.514+0000 7fab18db7700 1 -- start start 2024-05-08T21:22:32.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.515+0000 7fab18db7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab14072170 0x7fab140ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.515+0000 7fab18db7700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab140aed90 con 0x7fab1415dc60 2024-05-08T21:22:32.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.515+0000 7fab18db7700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab1406dcd0 con 0x7fab140ae620 2024-05-08T21:22:32.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.515+0000 7fab18db7700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab1406e290 con 0x7fab14072170 2024-05-08T21:22:32.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.515+0000 7fab1259c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fab1415dc60 0x7fab140ad9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:48004/0 (socket says 172.21.15.3:48004) 2024-05-08T21:22:32.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.515+0000 7fab1259c700 1 -- 172.21.15.3:0/1206602975 learned_addr learned my addr 172.21.15.3:0/1206602975 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:32.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.515+0000 7fab11d9b700 1 --2- 172.21.15.3:0/1206602975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab14072170 0x7fab140ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.515+0000 7fab1159a700 1 -- 172.21.15.3:0/1206602975 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077304044 0 0) 0x7fab04001110 con 0x7fab1415dc60 2024-05-08T21:22:32.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.515+0000 7fab1159a700 1 -- 172.21.15.3:0/1206602975 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faafc0035b0 con 0x7fab1415dc60 2024-05-08T21:22:32.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.516+0000 7fab1159a700 1 -- 172.21.15.3:0/1206602975 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972179797 0 0) 0x7fab1406dcd0 con 0x7fab140ae620 2024-05-08T21:22:32.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.516+0000 7fab1159a700 1 -- 172.21.15.3:0/1206602975 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab04001110 con 0x7fab140ae620 2024-05-08T21:22:32.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.516+0000 7fab11d9b700 1 -- 172.21.15.3:0/1206602975 >> v1:172.21.15.186:6789/0 conn(0x7fab140ae620 legacy=0x7fab1406e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:32.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.516+0000 7fab11d9b700 1 -- 172.21.15.3:0/1206602975 >> v1:172.21.15.3:6789/0 conn(0x7fab1415dc60 legacy=0x7fab140ad9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:32.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.516+0000 7fab11d9b700 1 -- 172.21.15.3:0/1206602975 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab140b2aa0 con 0x7fab14072170 2024-05-08T21:22:32.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.516+0000 7fab11d9b700 1 --2- 172.21.15.3:0/1206602975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab14072170 0x7fab140ae0e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fab08009870 tx=0x7fab08009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83823a96a7a24571 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:32.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.516+0000 7fab1159a700 1 -- 172.21.15.3:0/1206602975 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab08011830 con 0x7fab14072170 2024-05-08T21:22:32.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.516+0000 7fab1159a700 1 -- 172.21.15.3:0/1206602975 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab08011990 con 0x7fab14072170 2024-05-08T21:22:32.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.516+0000 7fab1159a700 1 -- 172.21.15.3:0/1206602975 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab08011d00 con 0x7fab14072170 2024-05-08T21:22:32.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.517+0000 7fab18db7700 1 -- 172.21.15.3:0/1206602975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab14072170 msgr2=0x7fab140ae0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:32.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.517+0000 7fab18db7700 1 --2- 172.21.15.3:0/1206602975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab14072170 0x7fab140ae0e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fab08009870 tx=0x7fab08009b80 comp rx=0 tx=0).stop 2024-05-08T21:22:32.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.517+0000 7fab18db7700 1 -- 172.21.15.3:0/1206602975 shutdown_connections 2024-05-08T21:22:32.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.517+0000 7fab18db7700 1 --2- 172.21.15.3:0/1206602975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab14072170 0x7fab140ae0e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.517+0000 7fab18db7700 1 -- 172.21.15.3:0/1206602975 >> 172.21.15.3:0/1206602975 conn(0x7fab140b5a30 msgr2=0x7fab140b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:32.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.517+0000 7fab18db7700 1 -- 172.21.15.3:0/1206602975 shutdown_connections 2024-05-08T21:22:32.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.517+0000 7fab18db7700 1 -- 172.21.15.3:0/1206602975 wait complete. 2024-05-08T21:22:32.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.518+0000 7fab18db7700 1 Processor -- start 2024-05-08T21:22:32.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.518+0000 7fab18db7700 1 -- start start 2024-05-08T21:22:32.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.519+0000 7fab18db7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab14072170 0x7fab141450d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.519+0000 7fab18db7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab140ae620 0x7fab14153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.519+0000 7fab18db7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab1415dc60 0x7fab14152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.519+0000 7fab18db7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab14155700 con 0x7fab1415dc60 2024-05-08T21:22:32.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.519+0000 7fab18db7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab14151eb0 con 0x7fab140ae620 2024-05-08T21:22:32.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.519+0000 7fab18db7700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab141548d0 con 0x7fab14072170 2024-05-08T21:22:32.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.519+0000 7fab12d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab1415dc60 0x7fab14152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.519+0000 7fab12d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab1415dc60 0x7fab14152ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51274/0 (socket says 172.21.15.3:51274) 2024-05-08T21:22:32.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.519+0000 7fab12d9d700 1 -- 172.21.15.3:0/327982367 learned_addr learned my addr 172.21.15.3:0/327982367 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:32.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.520+0000 7fab11d9b700 1 --2- 172.21.15.3:0/327982367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab140ae620 0x7fab14153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.520+0000 7fab1259c700 1 --2- 172.21.15.3:0/327982367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab14072170 0x7fab141450d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.520+0000 7fab12d9d700 1 -- 172.21.15.3:0/327982367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab14072170 msgr2=0x7fab141450d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:32.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.520+0000 7fab12d9d700 1 --2- 172.21.15.3:0/327982367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab14072170 0x7fab141450d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.520+0000 7fab12d9d700 1 -- 172.21.15.3:0/327982367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab140ae620 msgr2=0x7fab14153f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:32.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.520+0000 7fab12d9d700 1 --2- 172.21.15.3:0/327982367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab140ae620 0x7fab14153f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.520+0000 7fab12d9d700 1 -- 172.21.15.3:0/327982367 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab080094a0 con 0x7fab1415dc60 2024-05-08T21:22:32.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.520+0000 7fab11d9b700 1 --2- 172.21.15.3:0/327982367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab140ae620 0x7fab14153f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:32.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.520+0000 7fab1259c700 1 --2- 172.21.15.3:0/327982367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab14072170 0x7fab141450d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:32.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.520+0000 7fab12d9d700 1 --2- 172.21.15.3:0/327982367 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab1415dc60 0x7fab14152ba0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fab0c00cc20 tx=0x7fab0c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:32.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.520+0000 7fab02ffd700 1 -- 172.21.15.3:0/327982367 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0c00ac20 con 0x7fab1415dc60 2024-05-08T21:22:32.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.520+0000 7fab02ffd700 1 -- 172.21.15.3:0/327982367 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab0c00ad80 con 0x7fab1415dc60 2024-05-08T21:22:32.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.520+0000 7fab02ffd700 1 -- 172.21.15.3:0/327982367 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0c003c20 con 0x7fab1415dc60 2024-05-08T21:22:32.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.520+0000 7fab18db7700 1 -- 172.21.15.3:0/327982367 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab141520c0 con 0x7fab1415dc60 2024-05-08T21:22:32.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.520+0000 7fab18db7700 1 -- 172.21.15.3:0/327982367 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab14152680 con 0x7fab1415dc60 2024-05-08T21:22:32.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.521+0000 7fab18db7700 1 -- 172.21.15.3:0/327982367 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab14074bc0 con 0x7fab1415dc60 2024-05-08T21:22:32.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.522+0000 7fab02ffd700 1 -- 172.21.15.3:0/327982367 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fab0c0050d0 con 0x7fab1415dc60 2024-05-08T21:22:32.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.522+0000 7fab02ffd700 1 --2- 172.21.15.3:0/327982367 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faaf406ac90 0x7faaf406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.522+0000 7fab02ffd700 1 -- 172.21.15.3:0/327982367 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab0c089440 con 0x7fab1415dc60 2024-05-08T21:22:32.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.522+0000 7fab1259c700 1 --2- 172.21.15.3:0/327982367 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faaf406ac90 0x7faaf406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.523+0000 7fab1259c700 1 --2- 172.21.15.3:0/327982367 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faaf406ac90 0x7faaf406d150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fab04001370 tx=0x7fab04009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:32.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.527+0000 7fab02ffd700 1 -- 172.21.15.3:0/327982367 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fab0c0546b0 con 0x7fab1415dc60 2024-05-08T21:22:32.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.746+0000 7fab18db7700 1 -- 172.21.15.3:0/327982367 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab14071ec0 con 0x7fab1415dc60 2024-05-08T21:22:32.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.746+0000 7fab02ffd700 1 -- 172.21.15.3:0/327982367 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab0c057cd0 con 0x7fab1415dc60 2024-05-08T21:22:32.748 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:22:32.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.748+0000 7fab18db7700 1 -- 172.21.15.3:0/327982367 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faaf406ac90 msgr2=0x7faaf406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:32.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.748+0000 7fab18db7700 1 --2- 172.21.15.3:0/327982367 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faaf406ac90 0x7faaf406d150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fab04001370 tx=0x7fab04009040 comp rx=0 tx=0).stop 2024-05-08T21:22:32.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.748+0000 7fab18db7700 1 -- 172.21.15.3:0/327982367 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab1415dc60 msgr2=0x7fab14152ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:32.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.748+0000 7fab18db7700 1 --2- 172.21.15.3:0/327982367 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab1415dc60 0x7fab14152ba0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fab0c00cc20 tx=0x7fab0c00a060 comp rx=0 tx=0).stop 2024-05-08T21:22:32.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.749+0000 7fab18db7700 1 -- 172.21.15.3:0/327982367 shutdown_connections 2024-05-08T21:22:32.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.749+0000 7fab18db7700 1 --2- 172.21.15.3:0/327982367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab14072170 0x7fab141450d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.749+0000 7fab18db7700 1 --2- 172.21.15.3:0/327982367 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faaf406ac90 0x7faaf406d150 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.749+0000 7fab18db7700 1 --2- 172.21.15.3:0/327982367 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab1415dc60 0x7fab14152ba0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.749+0000 7fab18db7700 1 --2- 172.21.15.3:0/327982367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab140ae620 0x7fab14153f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.749+0000 7fab18db7700 1 -- 172.21.15.3:0/327982367 >> 172.21.15.3:0/327982367 conn(0x7fab140b5a30 msgr2=0x7fab1406f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:32.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.749+0000 7fab18db7700 1 -- 172.21.15.3:0/327982367 shutdown_connections 2024-05-08T21:22:32.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.749+0000 7fab18db7700 1 -- 172.21.15.3:0/327982367 wait complete. 2024-05-08T21:22:32.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-08T21:22:32.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.905+0000 7f8927896700 1 Processor -- start 2024-05-08T21:22:32.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.905+0000 7f8927896700 1 -- start start 2024-05-08T21:22:32.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.906+0000 7f8927896700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f892014b790 0x7f89200c43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.906+0000 7f8927896700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f89200c4920 0x7f89200ae2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.906+0000 7f8927896700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89200c4da0 con 0x7f89200ae800 2024-05-08T21:22:32.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.906+0000 7f8927896700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89200b00b0 con 0x7f892014b790 2024-05-08T21:22:32.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.906+0000 7f8927896700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89200af010 con 0x7f89200c4920 2024-05-08T21:22:32.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.906+0000 7f8925632700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f892014b790 0x7f89200c43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.907+0000 7f8925632700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f892014b790 0x7f89200c43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51288/0 (socket says 172.21.15.3:51288) 2024-05-08T21:22:32.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.907+0000 7f8925632700 1 -- 172.21.15.3:0/3932593239 learned_addr learned my addr 172.21.15.3:0/3932593239 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:32.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.907+0000 7f8924e31700 1 --2- 172.21.15.3:0/3932593239 >> v2:172.21.15.186:3300/0 conn(0x7f89200c4920 0x7f89200ae2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.907+0000 7f8925632700 1 -- 172.21.15.3:0/3932593239 >> v2:172.21.15.186:3300/0 conn(0x7f89200c4920 msgr2=0x7f89200ae2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:32.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.907+0000 7f8925632700 1 --2- 172.21.15.3:0/3932593239 >> v2:172.21.15.186:3300/0 conn(0x7f89200c4920 0x7f89200ae2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.907+0000 7f8925632700 1 -- 172.21.15.3:0/3932593239 >> v1:172.21.15.186:6789/0 conn(0x7f89200ae800 legacy=0x7f8920165930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:32.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.907+0000 7f8925632700 1 -- 172.21.15.3:0/3932593239 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89200af2f0 con 0x7f892014b790 2024-05-08T21:22:32.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.907+0000 7f8925632700 1 --2- 172.21.15.3:0/3932593239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f892014b790 0x7f89200c43e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f89200b3fa0 tx=0x7f891c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de693824222dbf5 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:32.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.907+0000 7f8917fff700 1 -- 172.21.15.3:0/3932593239 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c011830 con 0x7f892014b790 2024-05-08T21:22:32.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.907+0000 7f8917fff700 1 -- 172.21.15.3:0/3932593239 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c011990 con 0x7f892014b790 2024-05-08T21:22:32.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.907+0000 7f8917fff700 1 -- 172.21.15.3:0/3932593239 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c011d20 con 0x7f892014b790 2024-05-08T21:22:32.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.908+0000 7f8927896700 1 -- 172.21.15.3:0/3932593239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f892014b790 msgr2=0x7f89200c43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:32.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.908+0000 7f8927896700 1 --2- 172.21.15.3:0/3932593239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f892014b790 0x7f89200c43e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f89200b3fa0 tx=0x7f891c009bd0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.908+0000 7f8927896700 1 -- 172.21.15.3:0/3932593239 shutdown_connections 2024-05-08T21:22:32.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.908+0000 7f8927896700 1 --2- 172.21.15.3:0/3932593239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f892014b790 0x7f89200c43e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.908+0000 7f8927896700 1 --2- 172.21.15.3:0/3932593239 >> v2:172.21.15.186:3300/0 conn(0x7f89200c4920 0x7f89200ae2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.908+0000 7f8927896700 1 -- 172.21.15.3:0/3932593239 >> 172.21.15.3:0/3932593239 conn(0x7f892008afe0 msgr2=0x7f892008b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:32.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.908+0000 7f8927896700 1 -- 172.21.15.3:0/3932593239 shutdown_connections 2024-05-08T21:22:32.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.908+0000 7f8927896700 1 -- 172.21.15.3:0/3932593239 wait complete. 2024-05-08T21:22:32.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.909+0000 7f8927896700 1 Processor -- start 2024-05-08T21:22:32.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.909+0000 7f8927896700 1 -- start start 2024-05-08T21:22:32.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.910+0000 7f8927896700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89200ae800 0x7f8920149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.910+0000 7f8927896700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89200c4920 0x7f89200b55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.910+0000 7f8927896700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f892014b790 0x7f89200cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.910+0000 7f8927896700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892014a1c0 con 0x7f892014b790 2024-05-08T21:22:32.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.910+0000 7f8927896700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89200b4520 con 0x7f89200ae800 2024-05-08T21:22:32.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.910+0000 7f8927896700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89200b4800 con 0x7f89200c4920 2024-05-08T21:22:32.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.910+0000 7f8925e33700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f892014b790 0x7f89200cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.910+0000 7f8925e33700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f892014b790 0x7f89200cefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51300/0 (socket says 172.21.15.3:51300) 2024-05-08T21:22:32.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.910+0000 7f8925e33700 1 -- 172.21.15.3:0/2421002314 learned_addr learned my addr 172.21.15.3:0/2421002314 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:32.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.910+0000 7f8925632700 1 --2- 172.21.15.3:0/2421002314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89200ae800 0x7f8920149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.910+0000 7f8924e31700 1 --2- 172.21.15.3:0/2421002314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89200c4920 0x7f89200b55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.911+0000 7f8925e33700 1 -- 172.21.15.3:0/2421002314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89200c4920 msgr2=0x7f89200b55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:32.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.911+0000 7f8925e33700 1 --2- 172.21.15.3:0/2421002314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89200c4920 0x7f89200b55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.911+0000 7f8925e33700 1 -- 172.21.15.3:0/2421002314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89200ae800 msgr2=0x7f8920149c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:32.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.911+0000 7f8925e33700 1 --2- 172.21.15.3:0/2421002314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89200ae800 0x7f8920149c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:32.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.911+0000 7f8925e33700 1 -- 172.21.15.3:0/2421002314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891c009550 con 0x7f892014b790 2024-05-08T21:22:32.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.911+0000 7f8925e33700 1 --2- 172.21.15.3:0/2421002314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f892014b790 0x7f89200cefc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f891800cc50 tx=0x7f891800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:32.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.911+0000 7f8915ffb700 1 -- 172.21.15.3:0/2421002314 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918003c00 con 0x7f892014b790 2024-05-08T21:22:32.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.911+0000 7f8915ffb700 1 -- 172.21.15.3:0/2421002314 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8918003d60 con 0x7f892014b790 2024-05-08T21:22:32.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.911+0000 7f8927896700 1 -- 172.21.15.3:0/2421002314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89200b4ae0 con 0x7f892014b790 2024-05-08T21:22:32.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.911+0000 7f8915ffb700 1 -- 172.21.15.3:0/2421002314 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891800aab0 con 0x7f892014b790 2024-05-08T21:22:32.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.911+0000 7f8927896700 1 -- 172.21.15.3:0/2421002314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89200b50a0 con 0x7f892014b790 2024-05-08T21:22:32.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.912+0000 7f8927896700 1 -- 172.21.15.3:0/2421002314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8920074bc0 con 0x7f892014b790 2024-05-08T21:22:32.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.913+0000 7f8915ffb700 1 -- 172.21.15.3:0/2421002314 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f891800ac10 con 0x7f892014b790 2024-05-08T21:22:32.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.913+0000 7f8915ffb700 1 --2- 172.21.15.3:0/2421002314 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f890406ac90 0x7f890406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:32.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.913+0000 7f8915ffb700 1 -- 172.21.15.3:0/2421002314 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8918089160 con 0x7f892014b790 2024-05-08T21:22:32.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.913+0000 7f8925632700 1 --2- 172.21.15.3:0/2421002314 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f890406ac90 0x7f890406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:32.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.914+0000 7f8925632700 1 --2- 172.21.15.3:0/2421002314 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f890406ac90 0x7f890406d150 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f891c014040 tx=0x7f891c008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:32.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:32.918+0000 7f8915ffb700 1 -- 172.21.15.3:0/2421002314 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f8918054340 con 0x7f892014b790 2024-05-08T21:22:33.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.133+0000 7f8927896700 1 -- 172.21.15.3:0/2421002314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f89200c4da0 con 0x7f892014b790 2024-05-08T21:22:33.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.133+0000 7f8915ffb700 1 -- 172.21.15.3:0/2421002314 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f89200c4da0 con 0x7f892014b790 2024-05-08T21:22:33.135 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:22:33.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.136+0000 7f8927896700 1 -- 172.21.15.3:0/2421002314 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f890406ac90 msgr2=0x7f890406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:33.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.136+0000 7f8927896700 1 --2- 172.21.15.3:0/2421002314 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f890406ac90 0x7f890406d150 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f891c014040 tx=0x7f891c008030 comp rx=0 tx=0).stop 2024-05-08T21:22:33.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.136+0000 7f8927896700 1 -- 172.21.15.3:0/2421002314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f892014b790 msgr2=0x7f89200cefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:33.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.136+0000 7f8927896700 1 --2- 172.21.15.3:0/2421002314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f892014b790 0x7f89200cefc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f891800cc50 tx=0x7f891800cf60 comp rx=0 tx=0).stop 2024-05-08T21:22:33.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.136+0000 7f8927896700 1 -- 172.21.15.3:0/2421002314 shutdown_connections 2024-05-08T21:22:33.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.136+0000 7f8927896700 1 --2- 172.21.15.3:0/2421002314 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f890406ac90 0x7f890406d150 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.136+0000 7f8927896700 1 --2- 172.21.15.3:0/2421002314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f892014b790 0x7f89200cefc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.136+0000 7f8927896700 1 --2- 172.21.15.3:0/2421002314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89200ae800 0x7f8920149c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.136+0000 7f8927896700 1 --2- 172.21.15.3:0/2421002314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89200c4920 0x7f89200b55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.136+0000 7f8927896700 1 -- 172.21.15.3:0/2421002314 >> 172.21.15.3:0/2421002314 conn(0x7f892008afe0 msgr2=0x7f892015b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:33.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.136+0000 7f8927896700 1 -- 172.21.15.3:0/2421002314 shutdown_connections 2024-05-08T21:22:33.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.136+0000 7f8927896700 1 -- 172.21.15.3:0/2421002314 wait complete. 2024-05-08T21:22:33.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:22:33.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.289+0000 7fc97ed2f700 1 Processor -- start 2024-05-08T21:22:33.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.289+0000 7fc97ed2f700 1 -- start start 2024-05-08T21:22:33.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.290+0000 7fc97ed2f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc9780d3d20 0x7fc9780b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:33.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.290+0000 7fc97ed2f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9780cf610 con 0x7fc9780cff30 2024-05-08T21:22:33.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.290+0000 7fc97ed2f700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9780d1fc0 con 0x7fc9780b3fc0 2024-05-08T21:22:33.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.290+0000 7fc97ed2f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9780d22a0 con 0x7fc9780d3d20 2024-05-08T21:22:33.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.290+0000 7fc977fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc9780d3d20 0x7fc9780b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:33.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.290+0000 7fc97cacb700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc9780cff30 0x7fc9780d0310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:48012/0 (socket says 172.21.15.3:48012) 2024-05-08T21:22:33.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.290+0000 7fc97cacb700 1 -- 172.21.15.3:0/1423796311 learned_addr learned my addr 172.21.15.3:0/1423796311 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:33.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.291+0000 7fc977fff700 1 -- 172.21.15.3:0/1423796311 >> v1:172.21.15.186:6789/0 conn(0x7fc9780b3fc0 legacy=0x7fc9780b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:33.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.291+0000 7fc977fff700 1 -- 172.21.15.3:0/1423796311 >> v1:172.21.15.3:6789/0 conn(0x7fc9780cff30 legacy=0x7fc9780d0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:33.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.291+0000 7fc977fff700 1 -- 172.21.15.3:0/1423796311 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9780d2580 con 0x7fc9780d3d20 2024-05-08T21:22:33.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.291+0000 7fc977fff700 1 --2- 172.21.15.3:0/1423796311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9780d3d20 0x7fc9780b3a80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009840 tx=0x7fc96c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe791b398825e22 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:33.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.291+0000 7fc9777fe700 1 -- 172.21.15.3:0/1423796311 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c011830 con 0x7fc9780d3d20 2024-05-08T21:22:33.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.291+0000 7fc9777fe700 1 -- 172.21.15.3:0/1423796311 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96c011990 con 0x7fc9780d3d20 2024-05-08T21:22:33.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.291+0000 7fc9777fe700 1 -- 172.21.15.3:0/1423796311 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c011d20 con 0x7fc9780d3d20 2024-05-08T21:22:33.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.292+0000 7fc97ed2f700 1 -- 172.21.15.3:0/1423796311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9780d3d20 msgr2=0x7fc9780b3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:33.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.292+0000 7fc97ed2f700 1 --2- 172.21.15.3:0/1423796311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9780d3d20 0x7fc9780b3a80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009840 tx=0x7fc96c009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:33.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.292+0000 7fc97ed2f700 1 -- 172.21.15.3:0/1423796311 shutdown_connections 2024-05-08T21:22:33.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.292+0000 7fc97ed2f700 1 --2- 172.21.15.3:0/1423796311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9780d3d20 0x7fc9780b3a80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.292+0000 7fc97ed2f700 1 -- 172.21.15.3:0/1423796311 >> 172.21.15.3:0/1423796311 conn(0x7fc97808afe0 msgr2=0x7fc9780d3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:33.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.292+0000 7fc97ed2f700 1 -- 172.21.15.3:0/1423796311 shutdown_connections 2024-05-08T21:22:33.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.292+0000 7fc97ed2f700 1 -- 172.21.15.3:0/1423796311 wait complete. 2024-05-08T21:22:33.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.293+0000 7fc97ed2f700 1 Processor -- start 2024-05-08T21:22:33.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.293+0000 7fc97ed2f700 1 -- start start 2024-05-08T21:22:33.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.294+0000 7fc97ed2f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9780b3fc0 0x7fc97815ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:33.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.294+0000 7fc97ed2f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9780cff30 0x7fc97815dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:33.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.294+0000 7fc97ed2f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9780d3d20 0x7fc9781f7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:33.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.294+0000 7fc97ed2f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9781f8630 con 0x7fc9780b3fc0 2024-05-08T21:22:33.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.294+0000 7fc97ed2f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97815d7d0 con 0x7fc9780d3d20 2024-05-08T21:22:33.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.294+0000 7fc97ed2f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97815b180 con 0x7fc9780cff30 2024-05-08T21:22:33.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.294+0000 7fc97cacb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9780b3fc0 0x7fc97815ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:33.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.294+0000 7fc97cacb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9780b3fc0 0x7fc97815ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51316/0 (socket says 172.21.15.3:51316) 2024-05-08T21:22:33.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.294+0000 7fc97cacb700 1 -- 172.21.15.3:0/1101244300 learned_addr learned my addr 172.21.15.3:0/1101244300 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:33.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.294+0000 7fc977fff700 1 --2- 172.21.15.3:0/1101244300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9780cff30 0x7fc97815dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:33.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.294+0000 7fc97d2cc700 1 --2- 172.21.15.3:0/1101244300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9780d3d20 0x7fc9781f7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:33.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.295+0000 7fc97cacb700 1 -- 172.21.15.3:0/1101244300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9780cff30 msgr2=0x7fc97815dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:33.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.295+0000 7fc97cacb700 1 --2- 172.21.15.3:0/1101244300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9780cff30 0x7fc97815dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.295+0000 7fc97cacb700 1 -- 172.21.15.3:0/1101244300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9780d3d20 msgr2=0x7fc9781f7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:33.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.295+0000 7fc97cacb700 1 --2- 172.21.15.3:0/1101244300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9780d3d20 0x7fc9781f7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.295+0000 7fc97cacb700 1 -- 172.21.15.3:0/1101244300 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96c0094a0 con 0x7fc9780b3fc0 2024-05-08T21:22:33.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.295+0000 7fc97cacb700 1 --2- 172.21.15.3:0/1101244300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9780b3fc0 0x7fc97815ac40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc96800c550 tx=0x7fc96800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:33.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.295+0000 7fc9757fa700 1 -- 172.21.15.3:0/1101244300 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc968005d70 con 0x7fc9780b3fc0 2024-05-08T21:22:33.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.295+0000 7fc9757fa700 1 -- 172.21.15.3:0/1101244300 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9680070d0 con 0x7fc9780b3fc0 2024-05-08T21:22:33.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.295+0000 7fc97ed2f700 1 -- 172.21.15.3:0/1101244300 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9781f8070 con 0x7fc9780b3fc0 2024-05-08T21:22:33.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.295+0000 7fc9757fa700 1 -- 172.21.15.3:0/1101244300 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc968007440 con 0x7fc9780b3fc0 2024-05-08T21:22:33.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.295+0000 7fc97ed2f700 1 -- 172.21.15.3:0/1101244300 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97815d460 con 0x7fc9780b3fc0 2024-05-08T21:22:33.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.296+0000 7fc97ed2f700 1 -- 172.21.15.3:0/1101244300 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc97815dab0 con 0x7fc9780b3fc0 2024-05-08T21:22:33.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.297+0000 7fc9757fa700 1 -- 172.21.15.3:0/1101244300 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc9680075a0 con 0x7fc9780b3fc0 2024-05-08T21:22:33.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.297+0000 7fc9757fa700 1 --2- 172.21.15.3:0/1101244300 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc96406ac90 0x7fc96406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:33.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.297+0000 7fc9757fa700 1 -- 172.21.15.3:0/1101244300 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc968088550 con 0x7fc9780b3fc0 2024-05-08T21:22:33.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.297+0000 7fc977fff700 1 --2- 172.21.15.3:0/1101244300 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc96406ac90 0x7fc96406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:33.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.298+0000 7fc977fff700 1 --2- 172.21.15.3:0/1101244300 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc96406ac90 0x7fc96406d150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0090b0 tx=0x7fc96c008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:33.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.302+0000 7fc9757fa700 1 -- 172.21.15.3:0/1101244300 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc968053870 con 0x7fc9780b3fc0 2024-05-08T21:22:33.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.520+0000 7fc97ed2f700 1 -- 172.21.15.3:0/1101244300 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc978067a00 con 0x7fc9780b3fc0 2024-05-08T21:22:33.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.520+0000 7fc9757fa700 1 -- 172.21.15.3:0/1101244300 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc968056e90 con 0x7fc9780b3fc0 2024-05-08T21:22:33.522 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:22:33.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.522+0000 7fc97ed2f700 1 -- 172.21.15.3:0/1101244300 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc96406ac90 msgr2=0x7fc96406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:33.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.522+0000 7fc97ed2f700 1 --2- 172.21.15.3:0/1101244300 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc96406ac90 0x7fc96406d150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0090b0 tx=0x7fc96c008030 comp rx=0 tx=0).stop 2024-05-08T21:22:33.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.522+0000 7fc97ed2f700 1 -- 172.21.15.3:0/1101244300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9780b3fc0 msgr2=0x7fc97815ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:33.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.522+0000 7fc97ed2f700 1 --2- 172.21.15.3:0/1101244300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9780b3fc0 0x7fc97815ac40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc96800c550 tx=0x7fc96800c860 comp rx=0 tx=0).stop 2024-05-08T21:22:33.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.522+0000 7fc97ed2f700 1 -- 172.21.15.3:0/1101244300 shutdown_connections 2024-05-08T21:22:33.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.522+0000 7fc97ed2f700 1 --2- 172.21.15.3:0/1101244300 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc96406ac90 0x7fc96406d150 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.523+0000 7fc97ed2f700 1 --2- 172.21.15.3:0/1101244300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9780d3d20 0x7fc9781f7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.523+0000 7fc97ed2f700 1 --2- 172.21.15.3:0/1101244300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9780b3fc0 0x7fc97815ac40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.523+0000 7fc97ed2f700 1 --2- 172.21.15.3:0/1101244300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9780cff30 0x7fc97815dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.523+0000 7fc97ed2f700 1 -- 172.21.15.3:0/1101244300 >> 172.21.15.3:0/1101244300 conn(0x7fc97808afe0 msgr2=0x7fc97815ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:33.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.523+0000 7fc97ed2f700 1 -- 172.21.15.3:0/1101244300 shutdown_connections 2024-05-08T21:22:33.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.523+0000 7fc97ed2f700 1 -- 172.21.15.3:0/1101244300 wait complete. 2024-05-08T21:22:33.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-08T21:22:33.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.675+0000 7f26611bd700 1 Processor -- start 2024-05-08T21:22:33.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.675+0000 7f26611bd700 1 -- start start 2024-05-08T21:22:33.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.676+0000 7f26611bd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f265c15dc40 0x7f265c09a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:33.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.676+0000 7f26611bd700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f265c0d0700 con 0x7f265c09a160 2024-05-08T21:22:33.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.676+0000 7f26611bd700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f265c0d09e0 con 0x7f265c0d0270 2024-05-08T21:22:33.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.676+0000 7f26611bd700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f265c0d0cc0 con 0x7f265c15dc40 2024-05-08T21:22:33.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.676+0000 7f265ad9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f265c15dc40 0x7f265c09a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:33.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.676+0000 7f265ad9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f265c15dc40 0x7f265c09a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51326/0 (socket says 172.21.15.3:51326) 2024-05-08T21:22:33.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.676+0000 7f265b59e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f265c09a160 0x7f265c0cc280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:48024/0 (socket says 172.21.15.3:48024) 2024-05-08T21:22:33.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.676+0000 7f265ad9d700 1 -- 172.21.15.3:0/2913906331 learned_addr learned my addr 172.21.15.3:0/2913906331 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:33.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.676+0000 7f265ad9d700 1 -- 172.21.15.3:0/2913906331 >> v1:172.21.15.186:6789/0 conn(0x7f265c0d0270 legacy=0x7f265c099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:33.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.676+0000 7f265ad9d700 1 -- 172.21.15.3:0/2913906331 >> v1:172.21.15.3:6789/0 conn(0x7f265c09a160 legacy=0x7f265c0cc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:33.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.676+0000 7f265ad9d700 1 -- 172.21.15.3:0/2913906331 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265c0cba80 con 0x7f265c15dc40 2024-05-08T21:22:33.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.676+0000 7f2659d9b700 1 -- 172.21.15.3:0/2913906331 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126379804 0 0) 0x7f265c0d0700 con 0x7f265c09a160 2024-05-08T21:22:33.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.677+0000 7f265ad9d700 1 --2- 172.21.15.3:0/2913906331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f265c15dc40 0x7f265c09a790 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f264c0098f0 tx=0x7f264c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=589f42059cd34797 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:33.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.677+0000 7f2659d9b700 1 -- 172.21.15.3:0/2913906331 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c00b8a0 con 0x7f265c15dc40 2024-05-08T21:22:33.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.677+0000 7f2659d9b700 1 -- 172.21.15.3:0/2913906331 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264c00ba00 con 0x7f265c15dc40 2024-05-08T21:22:33.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.677+0000 7f2659d9b700 1 -- 172.21.15.3:0/2913906331 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c00bd90 con 0x7f265c15dc40 2024-05-08T21:22:33.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.677+0000 7f26611bd700 1 -- 172.21.15.3:0/2913906331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f265c15dc40 msgr2=0x7f265c09a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:33.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.677+0000 7f26611bd700 1 --2- 172.21.15.3:0/2913906331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f265c15dc40 0x7f265c09a790 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f264c0098f0 tx=0x7f264c009c00 comp rx=0 tx=0).stop 2024-05-08T21:22:33.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.677+0000 7f26611bd700 1 -- 172.21.15.3:0/2913906331 shutdown_connections 2024-05-08T21:22:33.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.677+0000 7f26611bd700 1 --2- 172.21.15.3:0/2913906331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f265c15dc40 0x7f265c09a790 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.677+0000 7f26611bd700 1 -- 172.21.15.3:0/2913906331 >> 172.21.15.3:0/2913906331 conn(0x7f265c0b2f20 msgr2=0x7f265c0b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:33.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.677+0000 7f26611bd700 1 -- 172.21.15.3:0/2913906331 shutdown_connections 2024-05-08T21:22:33.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.677+0000 7f26611bd700 1 -- 172.21.15.3:0/2913906331 wait complete. 2024-05-08T21:22:33.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.679+0000 7f26611bd700 1 Processor -- start 2024-05-08T21:22:33.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.679+0000 7f26611bd700 1 -- start start 2024-05-08T21:22:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.680+0000 7f26611bd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f265c09a160 0x7f265c151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.680+0000 7f26611bd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f265c15dc40 0x7f265c1544f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.680+0000 7f26611bd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f265c155550 0x7f265c15f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.680+0000 7f26611bd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c15e740 con 0x7f265c155550 2024-05-08T21:22:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.680+0000 7f26611bd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c15e3d0 con 0x7f265c15dc40 2024-05-08T21:22:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.680+0000 7f26611bd700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265c155a50 con 0x7f265c09a160 2024-05-08T21:22:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.680+0000 7f265b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f265c155550 0x7f265c15f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.680+0000 7f265b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f265c155550 0x7f265c15f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51330/0 (socket says 172.21.15.3:51330) 2024-05-08T21:22:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.680+0000 7f265b59e700 1 -- 172.21.15.3:0/2177845363 learned_addr learned my addr 172.21.15.3:0/2177845363 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.680+0000 7f265ad9d700 1 --2- 172.21.15.3:0/2177845363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f265c09a160 0x7f265c151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.680+0000 7f265a59c700 1 --2- 172.21.15.3:0/2177845363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f265c15dc40 0x7f265c1544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.680+0000 7f265b59e700 1 -- 172.21.15.3:0/2177845363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f265c09a160 msgr2=0x7f265c151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.680+0000 7f265b59e700 1 --2- 172.21.15.3:0/2177845363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f265c09a160 0x7f265c151010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.680+0000 7f265b59e700 1 -- 172.21.15.3:0/2177845363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f265c15dc40 msgr2=0x7f265c1544f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:33.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.680+0000 7f265b59e700 1 --2- 172.21.15.3:0/2177845363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f265c15dc40 0x7f265c1544f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.680+0000 7f265b59e700 1 -- 172.21.15.3:0/2177845363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264c009550 con 0x7f265c155550 2024-05-08T21:22:33.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.681+0000 7f265b59e700 1 --2- 172.21.15.3:0/2177845363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f265c155550 0x7f265c15f730 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f265400cca0 tx=0x7f265400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:33.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.681+0000 7f26437fe700 1 -- 172.21.15.3:0/2177845363 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265400cea0 con 0x7f265c155550 2024-05-08T21:22:33.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.681+0000 7f26437fe700 1 -- 172.21.15.3:0/2177845363 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f265400ad60 con 0x7f265c155550 2024-05-08T21:22:33.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.681+0000 7f26611bd700 1 -- 172.21.15.3:0/2177845363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f265c15e1a0 con 0x7f265c155550 2024-05-08T21:22:33.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.681+0000 7f26437fe700 1 -- 172.21.15.3:0/2177845363 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2654007310 con 0x7f265c155550 2024-05-08T21:22:33.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.681+0000 7f26611bd700 1 -- 172.21.15.3:0/2177845363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265c151550 con 0x7f265c155550 2024-05-08T21:22:33.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.682+0000 7f26611bd700 1 -- 172.21.15.3:0/2177845363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f265c15e880 con 0x7f265c155550 2024-05-08T21:22:33.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.682+0000 7f26437fe700 1 -- 172.21.15.3:0/2177845363 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2654007470 con 0x7f265c155550 2024-05-08T21:22:33.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.683+0000 7f26437fe700 1 --2- 172.21.15.3:0/2177845363 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f264806ac90 0x7f264806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:33.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.683+0000 7f26437fe700 1 -- 172.21.15.3:0/2177845363 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2654089260 con 0x7f265c155550 2024-05-08T21:22:33.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.683+0000 7f265ad9d700 1 --2- 172.21.15.3:0/2177845363 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f264806ac90 0x7f264806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:33.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.684+0000 7f265ad9d700 1 --2- 172.21.15.3:0/2177845363 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f264806ac90 0x7f264806d150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f264c009130 tx=0x7f264c01c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:33.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.688+0000 7f26437fe700 1 -- 172.21.15.3:0/2177845363 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f265c15e880 con 0x7f265c155550 2024-05-08T21:22:33.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.920+0000 7f26611bd700 1 -- 172.21.15.3:0/2177845363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f265c15e880 con 0x7f265c155550 2024-05-08T21:22:33.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.921+0000 7f26437fe700 1 -- 172.21.15.3:0/2177845363 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f265c15e880 con 0x7f265c155550 2024-05-08T21:22:33.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.923+0000 7f26611bd700 1 -- 172.21.15.3:0/2177845363 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f264806ac90 msgr2=0x7f264806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:33.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.923+0000 7f26611bd700 1 --2- 172.21.15.3:0/2177845363 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f264806ac90 0x7f264806d150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f264c009130 tx=0x7f264c01c040 comp rx=0 tx=0).stop 2024-05-08T21:22:33.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.923+0000 7f26611bd700 1 -- 172.21.15.3:0/2177845363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f265c155550 msgr2=0x7f265c15f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:33.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.923+0000 7f26611bd700 1 --2- 172.21.15.3:0/2177845363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f265c155550 0x7f265c15f730 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f265400cca0 tx=0x7f265400a340 comp rx=0 tx=0).stop 2024-05-08T21:22:33.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.923+0000 7f26611bd700 1 -- 172.21.15.3:0/2177845363 shutdown_connections 2024-05-08T21:22:33.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.923+0000 7f26611bd700 1 --2- 172.21.15.3:0/2177845363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f265c09a160 0x7f265c151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.923+0000 7f26611bd700 1 --2- 172.21.15.3:0/2177845363 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f264806ac90 0x7f264806d150 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.923+0000 7f26611bd700 1 --2- 172.21.15.3:0/2177845363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f265c155550 0x7f265c15f730 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.923+0000 7f26611bd700 1 --2- 172.21.15.3:0/2177845363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f265c15dc40 0x7f265c1544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:33.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.923+0000 7f26611bd700 1 -- 172.21.15.3:0/2177845363 >> 172.21.15.3:0/2177845363 conn(0x7f265c0b2f20 msgr2=0x7f265c070320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:33.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.923+0000 7f26611bd700 1 -- 172.21.15.3:0/2177845363 shutdown_connections 2024-05-08T21:22:33.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:33.923+0000 7f26611bd700 1 -- 172.21.15.3:0/2177845363 wait complete. 2024-05-08T21:22:33.926 INFO:teuthology.orchestra.run.smithi003.stderr:listed 0 entries 2024-05-08T21:22:33.939 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-05-08T21:22:33.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:33.939 DEBUG:teuthology.orchestra.run.smithi186:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-05-08T21:22:33.984 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi186.front.sepia.ceph.com 2024-05-08T21:22:33.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:33.984 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-05-08T21:22:34.003 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-08T21:22:34.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:22:34.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.140+0000 7f81a007a700 1 Processor -- start 2024-05-08T21:22:34.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.141+0000 7f81a007a700 1 -- start start 2024-05-08T21:22:34.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.142+0000 7f81a007a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f81980b2340 0x7f81980cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.142+0000 7f81a007a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f81980b4620 0x7f81980b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.142+0000 7f81a007a700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f819809ad40 con 0x7f819815ef60 2024-05-08T21:22:34.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.142+0000 7f81a007a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81980cd400 con 0x7f81980b2340 2024-05-08T21:22:34.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.142+0000 7f81a007a700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81980ccb60 con 0x7f81980b4620 2024-05-08T21:22:34.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.142+0000 7f819d615700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f81980b2340 0x7f81980cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.142+0000 7f819d615700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f81980b2340 0x7f81980cf710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51342/0 (socket says 172.21.15.3:51342) 2024-05-08T21:22:34.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.142+0000 7f819d615700 1 -- 172.21.15.3:0/2906368895 learned_addr learned my addr 172.21.15.3:0/2906368895 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:34.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.142+0000 7f819e617700 1 --2- 172.21.15.3:0/2906368895 >> v2:172.21.15.186:3300/0 conn(0x7f81980b4620 0x7f81980b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.142+0000 7f819d615700 1 -- 172.21.15.3:0/2906368895 >> v2:172.21.15.186:3300/0 conn(0x7f81980b4620 msgr2=0x7f81980b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:34.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.142+0000 7f819d615700 1 --2- 172.21.15.3:0/2906368895 >> v2:172.21.15.186:3300/0 conn(0x7f81980b4620 0x7f81980b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.142+0000 7f819d615700 1 -- 172.21.15.3:0/2906368895 >> v1:172.21.15.186:6789/0 conn(0x7f819815ef60 legacy=0x7f81980cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:34.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.142+0000 7f819d615700 1 -- 172.21.15.3:0/2906368895 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81980cce40 con 0x7f81980b2340 2024-05-08T21:22:34.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.142+0000 7f819e617700 1 --2- 172.21.15.3:0/2906368895 >> v2:172.21.15.186:3300/0 conn(0x7f81980b4620 0x7f81980b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:34.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.142+0000 7f819d615700 1 --2- 172.21.15.3:0/2906368895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81980b2340 0x7f81980cf710 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8198072340 tx=0x7f8188009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358a0c831e4f6115 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:34.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.143+0000 7f819ce14700 1 -- 172.21.15.3:0/2906368895 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8188004230 con 0x7f81980b2340 2024-05-08T21:22:34.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.143+0000 7f819ce14700 1 -- 172.21.15.3:0/2906368895 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8188004390 con 0x7f81980b2340 2024-05-08T21:22:34.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.143+0000 7f819ce14700 1 -- 172.21.15.3:0/2906368895 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81880036a0 con 0x7f81980b2340 2024-05-08T21:22:34.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.143+0000 7f81a007a700 1 -- 172.21.15.3:0/2906368895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81980b2340 msgr2=0x7f81980cf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:34.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.143+0000 7f81a007a700 1 --2- 172.21.15.3:0/2906368895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81980b2340 0x7f81980cf710 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8198072340 tx=0x7f8188009bd0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.143+0000 7f81a007a700 1 -- 172.21.15.3:0/2906368895 shutdown_connections 2024-05-08T21:22:34.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.143+0000 7f81a007a700 1 --2- 172.21.15.3:0/2906368895 >> v2:172.21.15.186:3300/0 conn(0x7f81980b4620 0x7f81980b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.143+0000 7f81a007a700 1 --2- 172.21.15.3:0/2906368895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81980b2340 0x7f81980cf710 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.143+0000 7f81a007a700 1 -- 172.21.15.3:0/2906368895 >> 172.21.15.3:0/2906368895 conn(0x7f81980b2f20 msgr2=0x7f81980b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:34.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.143+0000 7f81a007a700 1 -- 172.21.15.3:0/2906368895 shutdown_connections 2024-05-08T21:22:34.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.143+0000 7f81a007a700 1 -- 172.21.15.3:0/2906368895 wait complete. 2024-05-08T21:22:34.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.145+0000 7f81a007a700 1 Processor -- start 2024-05-08T21:22:34.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.145+0000 7f81a007a700 1 -- start start 2024-05-08T21:22:34.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.145+0000 7f81a007a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81980b2340 0x7f81980b1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.145+0000 7f81a007a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f81980b4620 0x7f81980b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.145+0000 7f81a007a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f819815ef60 0x7f81980b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.145+0000 7f81a007a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8198158250 con 0x7f81980b2340 2024-05-08T21:22:34.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.145+0000 7f81a007a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81980c9b40 con 0x7f819815ef60 2024-05-08T21:22:34.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.145+0000 7f81a007a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81980c9e20 con 0x7f81980b4620 2024-05-08T21:22:34.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.145+0000 7f819de16700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81980b2340 0x7f81980b1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.146+0000 7f819de16700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81980b2340 0x7f81980b1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51350/0 (socket says 172.21.15.3:51350) 2024-05-08T21:22:34.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.146+0000 7f819de16700 1 -- 172.21.15.3:0/74862238 learned_addr learned my addr 172.21.15.3:0/74862238 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:34.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.146+0000 7f819e617700 1 --2- 172.21.15.3:0/74862238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f819815ef60 0x7f81980b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.146+0000 7f819d615700 1 --2- 172.21.15.3:0/74862238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f81980b4620 0x7f81980b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.146+0000 7f819de16700 1 -- 172.21.15.3:0/74862238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f81980b4620 msgr2=0x7f81980b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:34.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.146+0000 7f819de16700 1 --2- 172.21.15.3:0/74862238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f81980b4620 0x7f81980b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.146+0000 7f819de16700 1 -- 172.21.15.3:0/74862238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f819815ef60 msgr2=0x7f81980b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:34.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.146+0000 7f819de16700 1 --2- 172.21.15.3:0/74862238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f819815ef60 0x7f81980b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.146+0000 7f819de16700 1 -- 172.21.15.3:0/74862238 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8188009550 con 0x7f81980b2340 2024-05-08T21:22:34.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.146+0000 7f819e617700 1 --2- 172.21.15.3:0/74862238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f819815ef60 0x7f81980b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:34.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.146+0000 7f819de16700 1 --2- 172.21.15.3:0/74862238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81980b2340 0x7f81980b1920 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f819400b4a0 tx=0x7f819400b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:34.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.146+0000 7f818e7fc700 1 -- 172.21.15.3:0/74862238 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194011830 con 0x7f81980b2340 2024-05-08T21:22:34.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.146+0000 7f818e7fc700 1 -- 172.21.15.3:0/74862238 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8194011990 con 0x7f81980b2340 2024-05-08T21:22:34.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.146+0000 7f81a007a700 1 -- 172.21.15.3:0/74862238 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81980ca100 con 0x7f81980b2340 2024-05-08T21:22:34.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.146+0000 7f818e7fc700 1 -- 172.21.15.3:0/74862238 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194011d40 con 0x7f81980b2340 2024-05-08T21:22:34.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.146+0000 7f81a007a700 1 -- 172.21.15.3:0/74862238 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81980b1e60 con 0x7f81980b2340 2024-05-08T21:22:34.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.147+0000 7f81a007a700 1 -- 172.21.15.3:0/74862238 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f819815d3d0 con 0x7f81980b2340 2024-05-08T21:22:34.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.148+0000 7f818e7fc700 1 -- 172.21.15.3:0/74862238 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f819401a400 con 0x7f81980b2340 2024-05-08T21:22:34.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.148+0000 7f818e7fc700 1 --2- 172.21.15.3:0/74862238 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f817c06ac90 0x7f817c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.148+0000 7f818e7fc700 1 -- 172.21.15.3:0/74862238 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8194088fe0 con 0x7f81980b2340 2024-05-08T21:22:34.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.149+0000 7f819d615700 1 --2- 172.21.15.3:0/74862238 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f817c06ac90 0x7f817c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.149+0000 7f819d615700 1 --2- 172.21.15.3:0/74862238 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f817c06ac90 0x7f817c06d150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8188009130 tx=0x7f8188008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:34.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.153+0000 7f818e7fc700 1 -- 172.21.15.3:0/74862238 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f8194054180 con 0x7f81980b2340 2024-05-08T21:22:34.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.420+0000 7f81a007a700 1 -- 172.21.15.3:0/74862238 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81980ca340 con 0x7f81980b2340 2024-05-08T21:22:34.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.420+0000 7f818e7fc700 1 -- 172.21.15.3:0/74862238 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f81980ca340 con 0x7f81980b2340 2024-05-08T21:22:34.422 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:34.422 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/2839854293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2839854293},{"type":"v1","addr":"172.21.15.3:6834","nonce":2839854293}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1878326567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":1878326567},{"type":"v1","addr":"172.21.15.3:6835","nonce":1878326567}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/3624849830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3624849830},{"type":"v1","addr":"172.21.15.186:6836","nonce":3624849830}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:21:52.062548+0000","modified":"2024-05-08T21:21:55.079806+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6837/1462886680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1462886680},{"type":"v1","addr":"172.21.15.186:6837","nonce":1462886680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-05-08T21:22:34.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.422+0000 7f81a007a700 1 -- 172.21.15.3:0/74862238 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f817c06ac90 msgr2=0x7f817c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:34.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.422+0000 7f81a007a700 1 --2- 172.21.15.3:0/74862238 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f817c06ac90 0x7f817c06d150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8188009130 tx=0x7f8188008030 comp rx=0 tx=0).stop 2024-05-08T21:22:34.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.422+0000 7f81a007a700 1 -- 172.21.15.3:0/74862238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81980b2340 msgr2=0x7f81980b1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:34.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.422+0000 7f81a007a700 1 --2- 172.21.15.3:0/74862238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81980b2340 0x7f81980b1920 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f819400b4a0 tx=0x7f819400b7b0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.423+0000 7f81a007a700 1 -- 172.21.15.3:0/74862238 shutdown_connections 2024-05-08T21:22:34.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.423+0000 7f81a007a700 1 --2- 172.21.15.3:0/74862238 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f817c06ac90 0x7f817c06d150 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.423+0000 7f81a007a700 1 --2- 172.21.15.3:0/74862238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f819815ef60 0x7f81980b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.423+0000 7f81a007a700 1 --2- 172.21.15.3:0/74862238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81980b2340 0x7f81980b1920 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.423+0000 7f81a007a700 1 --2- 172.21.15.3:0/74862238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f81980b4620 0x7f81980b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.423+0000 7f81a007a700 1 -- 172.21.15.3:0/74862238 >> 172.21.15.3:0/74862238 conn(0x7f81980b2f20 msgr2=0x7f819815d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:34.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.423+0000 7f81a007a700 1 -- 172.21.15.3:0/74862238 shutdown_connections 2024-05-08T21:22:34.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.423+0000 7f81a007a700 1 -- 172.21.15.3:0/74862238 wait complete. 2024-05-08T21:22:34.426 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 8 2024-05-08T21:22:34.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:34.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.573+0000 7fcec945e700 1 Processor -- start 2024-05-08T21:22:34.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.573+0000 7fcec945e700 1 -- start start 2024-05-08T21:22:34.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.574+0000 7fcec945e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcec415d8b0 0x7fcec40d23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.574+0000 7fcec945e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcec40d0610 0x7fcec40cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.574+0000 7fcec945e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec40d4020 con 0x7fcec40cede0 2024-05-08T21:22:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.574+0000 7fcec945e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcec415dc90 con 0x7fcec415d8b0 2024-05-08T21:22:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.574+0000 7fcec945e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec40d1310 con 0x7fcec40d0610 2024-05-08T21:22:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.574+0000 7fcec37fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcec40d0610 0x7fcec40cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.574+0000 7fcec2ffd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcec415d8b0 0x7fcec40d23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.574+0000 7fcec27fc700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fcec40cede0 0x7fcec40cff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:54028/0 (socket says 172.21.15.3:54028) 2024-05-08T21:22:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.574+0000 7fcec27fc700 1 -- 172.21.15.3:0/3434658098 learned_addr learned my addr 172.21.15.3:0/3434658098 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.575+0000 7fcec2ffd700 1 -- 172.21.15.3:0/3434658098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcec40d0610 msgr2=0x7fcec40cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.575+0000 7fcec2ffd700 1 --2- 172.21.15.3:0/3434658098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcec40d0610 0x7fcec40cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.575+0000 7fcec2ffd700 1 -- 172.21.15.3:0/3434658098 >> v1:172.21.15.186:6789/0 conn(0x7fcec40cede0 legacy=0x7fcec40cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.575+0000 7fcec2ffd700 1 -- 172.21.15.3:0/3434658098 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec40d15f0 con 0x7fcec415d8b0 2024-05-08T21:22:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.575+0000 7fcec37fe700 1 --2- 172.21.15.3:0/3434658098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcec40d0610 0x7fcec40cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.575+0000 7fcec2ffd700 1 --2- 172.21.15.3:0/3434658098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec415d8b0 0x7fcec40d23b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009840 tx=0x7fceb4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e3871807e66e77a server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.575+0000 7fcec1ffb700 1 -- 172.21.15.3:0/3434658098 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb4011830 con 0x7fcec415d8b0 2024-05-08T21:22:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.575+0000 7fcec1ffb700 1 -- 172.21.15.3:0/3434658098 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb4011990 con 0x7fcec415d8b0 2024-05-08T21:22:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.575+0000 7fcec1ffb700 1 -- 172.21.15.3:0/3434658098 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb4011d20 con 0x7fcec415d8b0 2024-05-08T21:22:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.576+0000 7fcec945e700 1 -- 172.21.15.3:0/3434658098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec415d8b0 msgr2=0x7fcec40d23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.576+0000 7fcec945e700 1 --2- 172.21.15.3:0/3434658098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec415d8b0 0x7fcec40d23b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009840 tx=0x7fceb4009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.576+0000 7fcec945e700 1 -- 172.21.15.3:0/3434658098 shutdown_connections 2024-05-08T21:22:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.576+0000 7fcec945e700 1 --2- 172.21.15.3:0/3434658098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcec40d0610 0x7fcec40cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.576+0000 7fcec945e700 1 --2- 172.21.15.3:0/3434658098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec415d8b0 0x7fcec40d23b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.576+0000 7fcec945e700 1 -- 172.21.15.3:0/3434658098 >> 172.21.15.3:0/3434658098 conn(0x7fcec408afe0 msgr2=0x7fcec40d3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.576+0000 7fcec945e700 1 -- 172.21.15.3:0/3434658098 shutdown_connections 2024-05-08T21:22:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.576+0000 7fcec945e700 1 -- 172.21.15.3:0/3434658098 wait complete. 2024-05-08T21:22:34.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.577+0000 7fcec945e700 1 Processor -- start 2024-05-08T21:22:34.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.577+0000 7fcec945e700 1 -- start start 2024-05-08T21:22:34.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.578+0000 7fcec945e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec40cede0 0x7fcec414fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.578+0000 7fcec945e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcec40d0610 0x7fcec40b1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.578+0000 7fcec945e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcec415d8b0 0x7fcec40b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.578+0000 7fcec945e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec40b0fb0 con 0x7fcec40d0610 2024-05-08T21:22:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.578+0000 7fcec945e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec40b1950 con 0x7fcec40cede0 2024-05-08T21:22:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.578+0000 7fcec945e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec40b0330 con 0x7fcec415d8b0 2024-05-08T21:22:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.578+0000 7fcec27fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcec40d0610 0x7fcec40b1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.578+0000 7fcec27fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcec40d0610 0x7fcec40b1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51366/0 (socket says 172.21.15.3:51366) 2024-05-08T21:22:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.578+0000 7fcec27fc700 1 -- 172.21.15.3:0/1847283509 learned_addr learned my addr 172.21.15.3:0/1847283509 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.578+0000 7fcec2ffd700 1 --2- 172.21.15.3:0/1847283509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec40cede0 0x7fcec414fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.578+0000 7fcec37fe700 1 --2- 172.21.15.3:0/1847283509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcec415d8b0 0x7fcec40b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.579+0000 7fcec27fc700 1 -- 172.21.15.3:0/1847283509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcec415d8b0 msgr2=0x7fcec40b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.579+0000 7fcec27fc700 1 --2- 172.21.15.3:0/1847283509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcec415d8b0 0x7fcec40b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.579+0000 7fcec27fc700 1 -- 172.21.15.3:0/1847283509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec40cede0 msgr2=0x7fcec414fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.579+0000 7fcec27fc700 1 --2- 172.21.15.3:0/1847283509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec40cede0 0x7fcec414fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.579+0000 7fcec27fc700 1 -- 172.21.15.3:0/1847283509 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc009a70 con 0x7fcec40d0610 2024-05-08T21:22:34.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.579+0000 7fcec27fc700 1 --2- 172.21.15.3:0/1847283509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcec40d0610 0x7fcec40b1b00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fceb800c520 tx=0x7fceb800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:34.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.579+0000 7fceab7fe700 1 -- 172.21.15.3:0/1847283509 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb8005dc0 con 0x7fcec40d0610 2024-05-08T21:22:34.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.579+0000 7fceab7fe700 1 -- 172.21.15.3:0/1847283509 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb80070d0 con 0x7fcec40d0610 2024-05-08T21:22:34.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.579+0000 7fcec945e700 1 -- 172.21.15.3:0/1847283509 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb40094a0 con 0x7fcec40d0610 2024-05-08T21:22:34.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.579+0000 7fceab7fe700 1 -- 172.21.15.3:0/1847283509 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb8007460 con 0x7fcec40d0610 2024-05-08T21:22:34.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.579+0000 7fcec945e700 1 -- 172.21.15.3:0/1847283509 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec40b08f0 con 0x7fcec40d0610 2024-05-08T21:22:34.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.580+0000 7fcec945e700 1 -- 172.21.15.3:0/1847283509 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec40b10f0 con 0x7fcec40d0610 2024-05-08T21:22:34.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.581+0000 7fceab7fe700 1 -- 172.21.15.3:0/1847283509 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fceb80075c0 con 0x7fcec40d0610 2024-05-08T21:22:34.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.581+0000 7fceab7fe700 1 --2- 172.21.15.3:0/1847283509 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fceb006ac90 0x7fceb006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.581+0000 7fceab7fe700 1 -- 172.21.15.3:0/1847283509 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fceb8088670 con 0x7fcec40d0610 2024-05-08T21:22:34.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.581+0000 7fcec2ffd700 1 --2- 172.21.15.3:0/1847283509 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fceb006ac90 0x7fceb006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.582+0000 7fcec2ffd700 1 --2- 172.21.15.3:0/1847283509 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fceb006ac90 0x7fceb006d150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fceb4000c00 tx=0x7fceb401c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:34.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.586+0000 7fceab7fe700 1 -- 172.21.15.3:0/1847283509 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fcec40b10f0 con 0x7fcec40d0610 2024-05-08T21:22:34.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.809+0000 7fcec945e700 1 -- 172.21.15.3:0/1847283509 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcec40b10f0 con 0x7fcec40d0610 2024-05-08T21:22:34.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.810+0000 7fceab7fe700 1 -- 172.21.15.3:0/1847283509 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16179 (secure 0 0 0) 0x7fcec40b10f0 con 0x7fcec40d0610 2024-05-08T21:22:34.812 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:34.812 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":21,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:22:34.648193+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.813+0000 7fcec945e700 1 -- 172.21.15.3:0/1847283509 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fceb006ac90 msgr2=0x7fceb006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.813+0000 7fcec945e700 1 --2- 172.21.15.3:0/1847283509 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fceb006ac90 0x7fceb006d150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fceb4000c00 tx=0x7fceb401c040 comp rx=0 tx=0).stop 2024-05-08T21:22:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.813+0000 7fcec945e700 1 -- 172.21.15.3:0/1847283509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcec40d0610 msgr2=0x7fcec40b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.813+0000 7fcec945e700 1 --2- 172.21.15.3:0/1847283509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcec40d0610 0x7fcec40b1b00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fceb800c520 tx=0x7fceb800c830 comp rx=0 tx=0).stop 2024-05-08T21:22:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.813+0000 7fcec945e700 1 -- 172.21.15.3:0/1847283509 shutdown_connections 2024-05-08T21:22:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.813+0000 7fcec945e700 1 --2- 172.21.15.3:0/1847283509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcec415d8b0 0x7fcec40b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.813+0000 7fcec945e700 1 --2- 172.21.15.3:0/1847283509 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fceb006ac90 0x7fceb006d150 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.813+0000 7fcec945e700 1 --2- 172.21.15.3:0/1847283509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec40cede0 0x7fcec414fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.813+0000 7fcec945e700 1 --2- 172.21.15.3:0/1847283509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcec40d0610 0x7fcec40b1b00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.813+0000 7fcec945e700 1 -- 172.21.15.3:0/1847283509 >> 172.21.15.3:0/1847283509 conn(0x7fcec408afe0 msgr2=0x7fcec406b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.813+0000 7fcec945e700 1 -- 172.21.15.3:0/1847283509 shutdown_connections 2024-05-08T21:22:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.813+0000 7fcec945e700 1 -- 172.21.15.3:0/1847283509 wait complete. 2024-05-08T21:22:34.828 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-08T21:21:49.638998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:21:51.024632+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:22:34.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:22:34.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.967+0000 7f5585100700 1 Processor -- start 2024-05-08T21:22:34.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.967+0000 7f5585100700 1 -- start start 2024-05-08T21:22:34.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f5585100700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f55800cff30 0x7f55800d0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f5585100700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55800d3d20 0x7f55800ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f5585100700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55800d0850 con 0x7f55800ce7a0 2024-05-08T21:22:34.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f5585100700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55800cf340 con 0x7f55800cff30 2024-05-08T21:22:34.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f5585100700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55800cf610 con 0x7f55800d3d20 2024-05-08T21:22:34.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f557ed9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f55800cff30 0x7f55800d0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f557ed9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f55800cff30 0x7f55800d0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51372/0 (socket says 172.21.15.3:51372) 2024-05-08T21:22:34.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f557ed9d700 1 -- 172.21.15.3:0/827907279 learned_addr learned my addr 172.21.15.3:0/827907279 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:34.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f557f59e700 1 --1- 172.21.15.3:0/827907279 >> v1:172.21.15.3:6789/0 conn(0x7f55800ce7a0 0x7f55800cec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:48040/0 (socket says 172.21.15.3:48040) 2024-05-08T21:22:34.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f557e59c700 1 --2- 172.21.15.3:0/827907279 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55800d3d20 0x7f55800ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f557ed9d700 1 -- 172.21.15.3:0/827907279 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55800d3d20 msgr2=0x7f55800ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:34.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f557ed9d700 1 --2- 172.21.15.3:0/827907279 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55800d3d20 0x7f55800ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f557ed9d700 1 -- 172.21.15.3:0/827907279 >> v1:172.21.15.3:6789/0 conn(0x7f55800ce7a0 legacy=0x7f55800cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:34.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f557ed9d700 1 -- 172.21.15.3:0/827907279 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55800d1fc0 con 0x7f55800cff30 2024-05-08T21:22:34.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f557dd9b700 1 -- 172.21.15.3:0/827907279 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445605647 0 0) 0x7f55800d0850 con 0x7f55800ce7a0 2024-05-08T21:22:34.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f557ed9d700 1 --2- 172.21.15.3:0/827907279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55800cff30 0x7f55800d0310 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f55700098f0 tx=0x7f5570009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd1a7d00b7a27474 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:34.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.968+0000 7f557dd9b700 1 -- 172.21.15.3:0/827907279 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557000b8a0 con 0x7f55800cff30 2024-05-08T21:22:34.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.969+0000 7f557dd9b700 1 -- 172.21.15.3:0/827907279 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557000ba00 con 0x7f55800cff30 2024-05-08T21:22:34.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.969+0000 7f557dd9b700 1 -- 172.21.15.3:0/827907279 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557000bd90 con 0x7f55800cff30 2024-05-08T21:22:34.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.969+0000 7f5585100700 1 -- 172.21.15.3:0/827907279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55800cff30 msgr2=0x7f55800d0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:34.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.969+0000 7f5585100700 1 --2- 172.21.15.3:0/827907279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55800cff30 0x7f55800d0310 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f55700098f0 tx=0x7f5570009c00 comp rx=0 tx=0).stop 2024-05-08T21:22:34.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.969+0000 7f5585100700 1 -- 172.21.15.3:0/827907279 shutdown_connections 2024-05-08T21:22:34.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.969+0000 7f5585100700 1 --2- 172.21.15.3:0/827907279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55800cff30 0x7f55800d0310 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.969+0000 7f5585100700 1 --2- 172.21.15.3:0/827907279 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55800d3d20 0x7f55800ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.969+0000 7f5585100700 1 -- 172.21.15.3:0/827907279 >> 172.21.15.3:0/827907279 conn(0x7f558008afe0 msgr2=0x7f55800d3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:34.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.969+0000 7f5585100700 1 -- 172.21.15.3:0/827907279 shutdown_connections 2024-05-08T21:22:34.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.969+0000 7f5585100700 1 -- 172.21.15.3:0/827907279 wait complete. 2024-05-08T21:22:34.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.971+0000 7f5585100700 1 Processor -- start 2024-05-08T21:22:34.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.971+0000 7f5585100700 1 -- start start 2024-05-08T21:22:34.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f5585100700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55800ce7a0 0x7f558015b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f5585100700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55800cff30 0x7f55800b8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f5585100700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55800d3d20 0x7f55800c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f5585100700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558015b910 con 0x7f55800ce7a0 2024-05-08T21:22:34.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f5585100700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558015bbf0 con 0x7f55800d3d20 2024-05-08T21:22:34.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f5585100700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558015bed0 con 0x7f55800cff30 2024-05-08T21:22:34.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f557ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55800ce7a0 0x7f558015b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f557ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55800ce7a0 0x7f558015b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51388/0 (socket says 172.21.15.3:51388) 2024-05-08T21:22:34.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f557ed9d700 1 -- 172.21.15.3:0/1638049707 learned_addr learned my addr 172.21.15.3:0/1638049707 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:34.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f557e59c700 1 --2- 172.21.15.3:0/1638049707 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55800cff30 0x7f55800b8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f557f59e700 1 --2- 172.21.15.3:0/1638049707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55800d3d20 0x7f55800c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f557ed9d700 1 -- 172.21.15.3:0/1638049707 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55800cff30 msgr2=0x7f55800b8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:34.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f557ed9d700 1 --2- 172.21.15.3:0/1638049707 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55800cff30 0x7f55800b8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f557ed9d700 1 -- 172.21.15.3:0/1638049707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55800d3d20 msgr2=0x7f55800c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:34.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f557ed9d700 1 --2- 172.21.15.3:0/1638049707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55800d3d20 0x7f55800c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:34.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f557ed9d700 1 -- 172.21.15.3:0/1638049707 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5570009550 con 0x7f55800ce7a0 2024-05-08T21:22:34.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f557ed9d700 1 --2- 172.21.15.3:0/1638049707 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55800ce7a0 0x7f558015b330 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5570009c40 tx=0x7f5570004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:34.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.972+0000 7f55677fe700 1 -- 172.21.15.3:0/1638049707 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5570004240 con 0x7f55800ce7a0 2024-05-08T21:22:34.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.973+0000 7f55677fe700 1 -- 172.21.15.3:0/1638049707 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557001f070 con 0x7f55800ce7a0 2024-05-08T21:22:34.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.973+0000 7f5585100700 1 -- 172.21.15.3:0/1638049707 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558015c1b0 con 0x7f55800ce7a0 2024-05-08T21:22:34.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.973+0000 7f55677fe700 1 -- 172.21.15.3:0/1638049707 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55700045a0 con 0x7f55800ce7a0 2024-05-08T21:22:34.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.973+0000 7f5585100700 1 -- 172.21.15.3:0/1638049707 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55800b9270 con 0x7f55800ce7a0 2024-05-08T21:22:34.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.974+0000 7f5585100700 1 -- 172.21.15.3:0/1638049707 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558015c6d0 con 0x7f55800ce7a0 2024-05-08T21:22:34.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.974+0000 7f55677fe700 1 -- 172.21.15.3:0/1638049707 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5570021030 con 0x7f55800ce7a0 2024-05-08T21:22:34.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.974+0000 7f55677fe700 1 --2- 172.21.15.3:0/1638049707 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f556c06ac90 0x7f556c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:34.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.975+0000 7f55677fe700 1 -- 172.21.15.3:0/1638049707 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f557008a6b0 con 0x7f55800ce7a0 2024-05-08T21:22:34.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.975+0000 7f557e59c700 1 --2- 172.21.15.3:0/1638049707 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f556c06ac90 0x7f556c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:34.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.975+0000 7f557e59c700 1 --2- 172.21.15.3:0/1638049707 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f556c06ac90 0x7f556c06d150 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f55801493b0 tx=0x7f5574008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:34.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:34.979+0000 7f55677fe700 1 -- 172.21.15.3:0/1638049707 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f5570055820 con 0x7f55800ce7a0 2024-05-08T21:22:35.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.201+0000 7f5585100700 1 -- 172.21.15.3:0/1638049707 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5580074bc0 con 0x7f55800ce7a0 2024-05-08T21:22:35.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.202+0000 7f55677fe700 1 -- 172.21.15.3:0/1638049707 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5570058e40 con 0x7f55800ce7a0 2024-05-08T21:22:35.203 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:22:35.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.204+0000 7f5585100700 1 -- 172.21.15.3:0/1638049707 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f556c06ac90 msgr2=0x7f556c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:35.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.204+0000 7f5585100700 1 --2- 172.21.15.3:0/1638049707 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f556c06ac90 0x7f556c06d150 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f55801493b0 tx=0x7f5574008040 comp rx=0 tx=0).stop 2024-05-08T21:22:35.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.204+0000 7f5585100700 1 -- 172.21.15.3:0/1638049707 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55800ce7a0 msgr2=0x7f558015b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:35.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.204+0000 7f5585100700 1 --2- 172.21.15.3:0/1638049707 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55800ce7a0 0x7f558015b330 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5570009c40 tx=0x7f5570004750 comp rx=0 tx=0).stop 2024-05-08T21:22:35.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.204+0000 7f5585100700 1 -- 172.21.15.3:0/1638049707 shutdown_connections 2024-05-08T21:22:35.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.204+0000 7f5585100700 1 --2- 172.21.15.3:0/1638049707 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f556c06ac90 0x7f556c06d150 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.204+0000 7f5585100700 1 --2- 172.21.15.3:0/1638049707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55800d3d20 0x7f55800c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.204+0000 7f5585100700 1 --2- 172.21.15.3:0/1638049707 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55800ce7a0 0x7f558015b330 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.204+0000 7f5585100700 1 --2- 172.21.15.3:0/1638049707 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55800cff30 0x7f55800b8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.204+0000 7f5585100700 1 -- 172.21.15.3:0/1638049707 >> 172.21.15.3:0/1638049707 conn(0x7f558008afe0 msgr2=0x7f55800b4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:35.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.204+0000 7f5585100700 1 -- 172.21.15.3:0/1638049707 shutdown_connections 2024-05-08T21:22:35.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.204+0000 7f5585100700 1 -- 172.21.15.3:0/1638049707 wait complete. 2024-05-08T21:22:35.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-08T21:22:35.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.357+0000 7fed074bc700 1 Processor -- start 2024-05-08T21:22:35.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.357+0000 7fed074bc700 1 -- start start 2024-05-08T21:22:35.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.358+0000 7fed074bc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fed0015e760 0x7fed0006a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:35.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.358+0000 7fed074bc700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed000cf970 con 0x7fed000cbf50 2024-05-08T21:22:35.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.358+0000 7fed074bc700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed000cfc50 con 0x7fed000d0f10 2024-05-08T21:22:35.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.358+0000 7fed074bc700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed000cff30 con 0x7fed0015e760 2024-05-08T21:22:35.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.359+0000 7fed05a59700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fed0015e760 0x7fed0006a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:35.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.359+0000 7fed05258700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fed000cbf50 0x7fed00161d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:48044/0 (socket says 172.21.15.3:48044) 2024-05-08T21:22:35.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.359+0000 7fed05258700 1 -- 172.21.15.3:0/1099693665 learned_addr learned my addr 172.21.15.3:0/1099693665 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:35.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.359+0000 7fed05a59700 1 -- 172.21.15.3:0/1099693665 >> v1:172.21.15.186:6789/0 conn(0x7fed000d0f10 legacy=0x7fed0015e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:35.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.359+0000 7fed05a59700 1 -- 172.21.15.3:0/1099693665 >> v1:172.21.15.3:6789/0 conn(0x7fed000cbf50 legacy=0x7fed00161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:35.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.359+0000 7fed05a59700 1 -- 172.21.15.3:0/1099693665 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed001617e0 con 0x7fed0015e760 2024-05-08T21:22:35.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.359+0000 7fed05a59700 1 --2- 172.21.15.3:0/1099693665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed0015e760 0x7fed0006a960 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fecf8009d90 tx=0x7fecf800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9fa85ad74d63f34 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:35.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.359+0000 7fecf7fff700 1 -- 172.21.15.3:0/1099693665 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf800e070 con 0x7fed0015e760 2024-05-08T21:22:35.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.359+0000 7fecf7fff700 1 -- 172.21.15.3:0/1099693665 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf800bbd0 con 0x7fed0015e760 2024-05-08T21:22:35.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.359+0000 7fecf7fff700 1 -- 172.21.15.3:0/1099693665 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf8012430 con 0x7fed0015e760 2024-05-08T21:22:35.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.360+0000 7fed074bc700 1 -- 172.21.15.3:0/1099693665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed0015e760 msgr2=0x7fed0006a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:35.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.360+0000 7fed074bc700 1 --2- 172.21.15.3:0/1099693665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed0015e760 0x7fed0006a960 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fecf8009d90 tx=0x7fecf800b330 comp rx=0 tx=0).stop 2024-05-08T21:22:35.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.360+0000 7fed074bc700 1 -- 172.21.15.3:0/1099693665 shutdown_connections 2024-05-08T21:22:35.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.360+0000 7fed074bc700 1 --2- 172.21.15.3:0/1099693665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed0015e760 0x7fed0006a960 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.360+0000 7fed074bc700 1 -- 172.21.15.3:0/1099693665 >> 172.21.15.3:0/1099693665 conn(0x7fed00099a50 msgr2=0x7fed000d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:35.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.360+0000 7fed074bc700 1 -- 172.21.15.3:0/1099693665 shutdown_connections 2024-05-08T21:22:35.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.360+0000 7fed074bc700 1 -- 172.21.15.3:0/1099693665 wait complete. 2024-05-08T21:22:35.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.361+0000 7fed074bc700 1 Processor -- start 2024-05-08T21:22:35.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.361+0000 7fed074bc700 1 -- start start 2024-05-08T21:22:35.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.362+0000 7fed074bc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed000cbf50 0x7fed001e9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:35.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.362+0000 7fed074bc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed000d0f10 0x7fed001e9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.362+0000 7fed074bc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed000b2ef0 0x7fed0014b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.362+0000 7fed074bc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed000b4610 con 0x7fed000cbf50 2024-05-08T21:22:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.362+0000 7fed074bc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed000b3490 con 0x7fed000d0f10 2024-05-08T21:22:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.362+0000 7fed074bc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed000b3770 con 0x7fed000b2ef0 2024-05-08T21:22:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.362+0000 7fed05258700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed000cbf50 0x7fed001e9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.362+0000 7fed05258700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed000cbf50 0x7fed001e9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51398/0 (socket says 172.21.15.3:51398) 2024-05-08T21:22:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.362+0000 7fed05258700 1 -- 172.21.15.3:0/649735386 learned_addr learned my addr 172.21.15.3:0/649735386 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.362+0000 7fed04a57700 1 --2- 172.21.15.3:0/649735386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed000d0f10 0x7fed001e9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.362+0000 7fed05a59700 1 --2- 172.21.15.3:0/649735386 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed000b2ef0 0x7fed0014b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.363+0000 7fed05258700 1 -- 172.21.15.3:0/649735386 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed000b2ef0 msgr2=0x7fed0014b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.363+0000 7fed05258700 1 --2- 172.21.15.3:0/649735386 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed000b2ef0 0x7fed0014b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.363+0000 7fed05258700 1 -- 172.21.15.3:0/649735386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed000d0f10 msgr2=0x7fed001e9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.363+0000 7fed05258700 1 --2- 172.21.15.3:0/649735386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed000d0f10 0x7fed001e9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.363+0000 7fed05258700 1 -- 172.21.15.3:0/649735386 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf8009a70 con 0x7fed000cbf50 2024-05-08T21:22:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.363+0000 7fed05258700 1 --2- 172.21.15.3:0/649735386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed000cbf50 0x7fed001e9620 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fecfc00c520 tx=0x7fecfc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.363+0000 7fecf5ffb700 1 -- 172.21.15.3:0/649735386 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc005d70 con 0x7fed000cbf50 2024-05-08T21:22:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.363+0000 7fecf5ffb700 1 -- 172.21.15.3:0/649735386 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecfc0070d0 con 0x7fed000cbf50 2024-05-08T21:22:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.363+0000 7fed074bc700 1 -- 172.21.15.3:0/649735386 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed000b3a50 con 0x7fed000cbf50 2024-05-08T21:22:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.363+0000 7fecf5ffb700 1 -- 172.21.15.3:0/649735386 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc007440 con 0x7fed000cbf50 2024-05-08T21:22:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.363+0000 7fed074bc700 1 -- 172.21.15.3:0/649735386 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0014b640 con 0x7fed000cbf50 2024-05-08T21:22:35.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.364+0000 7fed074bc700 1 -- 172.21.15.3:0/649735386 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed00074bc0 con 0x7fed000cbf50 2024-05-08T21:22:35.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.365+0000 7fecf5ffb700 1 -- 172.21.15.3:0/649735386 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fecfc0075a0 con 0x7fed000cbf50 2024-05-08T21:22:35.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.365+0000 7fecf5ffb700 1 --2- 172.21.15.3:0/649735386 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fece406ac90 0x7fece406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:35.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.365+0000 7fecf5ffb700 1 -- 172.21.15.3:0/649735386 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fecfc088710 con 0x7fed000cbf50 2024-05-08T21:22:35.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.366+0000 7fed04a57700 1 --2- 172.21.15.3:0/649735386 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fece406ac90 0x7fece406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:35.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.366+0000 7fed04a57700 1 --2- 172.21.15.3:0/649735386 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fece406ac90 0x7fece406d150 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fed000b2900 tx=0x7fecf0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:35.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.370+0000 7fecf5ffb700 1 -- 172.21.15.3:0/649735386 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fecfc053890 con 0x7fed000cbf50 2024-05-08T21:22:35.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.586+0000 7fed074bc700 1 -- 172.21.15.3:0/649735386 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fed000b62b0 con 0x7fed000cbf50 2024-05-08T21:22:35.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.587+0000 7fecf5ffb700 1 -- 172.21.15.3:0/649735386 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fecfc056eb0 con 0x7fed000cbf50 2024-05-08T21:22:35.589 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:22:35.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.589+0000 7fed074bc700 1 -- 172.21.15.3:0/649735386 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fece406ac90 msgr2=0x7fece406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:35.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.589+0000 7fed074bc700 1 --2- 172.21.15.3:0/649735386 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fece406ac90 0x7fece406d150 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fed000b2900 tx=0x7fecf0009040 comp rx=0 tx=0).stop 2024-05-08T21:22:35.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.589+0000 7fed074bc700 1 -- 172.21.15.3:0/649735386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed000cbf50 msgr2=0x7fed001e9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:35.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.589+0000 7fed074bc700 1 --2- 172.21.15.3:0/649735386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed000cbf50 0x7fed001e9620 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fecfc00c520 tx=0x7fecfc00c830 comp rx=0 tx=0).stop 2024-05-08T21:22:35.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.589+0000 7fed074bc700 1 -- 172.21.15.3:0/649735386 shutdown_connections 2024-05-08T21:22:35.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.589+0000 7fed074bc700 1 --2- 172.21.15.3:0/649735386 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed000b2ef0 0x7fed0014b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.589+0000 7fed074bc700 1 --2- 172.21.15.3:0/649735386 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fece406ac90 0x7fece406d150 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.589+0000 7fed074bc700 1 --2- 172.21.15.3:0/649735386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed000cbf50 0x7fed001e9620 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.589+0000 7fed074bc700 1 --2- 172.21.15.3:0/649735386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed000d0f10 0x7fed001e9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.589+0000 7fed074bc700 1 -- 172.21.15.3:0/649735386 >> 172.21.15.3:0/649735386 conn(0x7fed00099a50 msgr2=0x7fed000ad0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:35.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.589+0000 7fed074bc700 1 -- 172.21.15.3:0/649735386 shutdown_connections 2024-05-08T21:22:35.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.589+0000 7fed074bc700 1 -- 172.21.15.3:0/649735386 wait complete. 2024-05-08T21:22:35.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:22:35.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.748+0000 7fb34aaa0700 1 Processor -- start 2024-05-08T21:22:35.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.748+0000 7fb34aaa0700 1 -- start start 2024-05-08T21:22:35.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb34aaa0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34415b4e0 0x7fb3440b1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:35.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb34aaa0700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb3440d1910 0x7fb3440cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:35.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb34aaa0700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3440cf560 con 0x7fb34415b8c0 2024-05-08T21:22:35.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb34aaa0700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3440cefe0 con 0x7fb3440d1910 2024-05-08T21:22:35.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb34aaa0700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3440ccff0 con 0x7fb34415b4e0 2024-05-08T21:22:35.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb343fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb34415b8c0 0x7fb3440d1200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:48052/0 (socket says 172.21.15.3:48052) 2024-05-08T21:22:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb343fff700 1 -- 172.21.15.3:0/4208784412 learned_addr learned my addr 172.21.15.3:0/4208784412 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb34883c700 1 --2- 172.21.15.3:0/4208784412 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34415b4e0 0x7fb3440b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb34903d700 1 --2- 172.21.15.3:0/4208784412 >> v2:172.21.15.186:3300/0 conn(0x7fb3440d1910 0x7fb3440cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb3437fe700 1 -- 172.21.15.3:0/4208784412 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337465434 0 0) 0x7fb3440cf560 con 0x7fb34415b8c0 2024-05-08T21:22:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb3437fe700 1 -- 172.21.15.3:0/4208784412 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb32c0035b0 con 0x7fb34415b8c0 2024-05-08T21:22:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb3437fe700 1 -- 172.21.15.3:0/4208784412 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3506732459 0 0) 0x7fb32c0035b0 con 0x7fb34415b8c0 2024-05-08T21:22:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb34883c700 1 -- 172.21.15.3:0/4208784412 >> v2:172.21.15.186:3300/0 conn(0x7fb3440d1910 msgr2=0x7fb3440cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb34883c700 1 --2- 172.21.15.3:0/4208784412 >> v2:172.21.15.186:3300/0 conn(0x7fb3440d1910 0x7fb3440cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb34883c700 1 -- 172.21.15.3:0/4208784412 >> v1:172.21.15.3:6789/0 conn(0x7fb34415b8c0 legacy=0x7fb3440d1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb34883c700 1 -- 172.21.15.3:0/4208784412 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3440cd2d0 con 0x7fb34415b4e0 2024-05-08T21:22:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.749+0000 7fb34903d700 1 --2- 172.21.15.3:0/4208784412 >> v2:172.21.15.186:3300/0 conn(0x7fb3440d1910 0x7fb3440cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.750+0000 7fb34883c700 1 --2- 172.21.15.3:0/4208784412 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34415b4e0 0x7fb3440b1200 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb334009840 tx=0x7fb334009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1959b5303ba68d2d server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.750+0000 7fb3437fe700 1 -- 172.21.15.3:0/4208784412 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338001670 con 0x7fb34415b4e0 2024-05-08T21:22:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.750+0000 7fb3437fe700 1 -- 172.21.15.3:0/4208784412 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb334004030 con 0x7fb34415b4e0 2024-05-08T21:22:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.750+0000 7fb3437fe700 1 -- 172.21.15.3:0/4208784412 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3340043a0 con 0x7fb34415b4e0 2024-05-08T21:22:35.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.750+0000 7fb34aaa0700 1 -- 172.21.15.3:0/4208784412 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34415b4e0 msgr2=0x7fb3440b1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:35.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.750+0000 7fb34aaa0700 1 --2- 172.21.15.3:0/4208784412 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34415b4e0 0x7fb3440b1200 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb334009840 tx=0x7fb334009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:35.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.751+0000 7fb34aaa0700 1 -- 172.21.15.3:0/4208784412 shutdown_connections 2024-05-08T21:22:35.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.751+0000 7fb34aaa0700 1 --2- 172.21.15.3:0/4208784412 >> v2:172.21.15.186:3300/0 conn(0x7fb3440d1910 0x7fb3440cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.751+0000 7fb34aaa0700 1 --2- 172.21.15.3:0/4208784412 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34415b4e0 0x7fb3440b1200 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.751+0000 7fb34aaa0700 1 -- 172.21.15.3:0/4208784412 >> 172.21.15.3:0/4208784412 conn(0x7fb34408afe0 msgr2=0x7fb34408b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:35.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.751+0000 7fb34aaa0700 1 -- 172.21.15.3:0/4208784412 shutdown_connections 2024-05-08T21:22:35.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.751+0000 7fb34aaa0700 1 -- 172.21.15.3:0/4208784412 wait complete. 2024-05-08T21:22:35.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.752+0000 7fb34aaa0700 1 Processor -- start 2024-05-08T21:22:35.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.752+0000 7fb34aaa0700 1 -- start start 2024-05-08T21:22:35.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.753+0000 7fb34aaa0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3440d1910 0x7fb3440b5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:35.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.753+0000 7fb34aaa0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb34415b4e0 0x7fb3440af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:35.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.753+0000 7fb34aaa0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34415b8c0 0x7fb34414efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:35.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.753+0000 7fb34aaa0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3440b0960 con 0x7fb3440d1910 2024-05-08T21:22:35.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.753+0000 7fb34aaa0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3440af8c0 con 0x7fb34415b4e0 2024-05-08T21:22:35.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.753+0000 7fb34aaa0700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3440afba0 con 0x7fb34415b8c0 2024-05-08T21:22:35.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.753+0000 7fb34883c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3440d1910 0x7fb3440b5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:35.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.753+0000 7fb34883c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3440d1910 0x7fb3440b5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51414/0 (socket says 172.21.15.3:51414) 2024-05-08T21:22:35.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.753+0000 7fb34883c700 1 -- 172.21.15.3:0/1264885727 learned_addr learned my addr 172.21.15.3:0/1264885727 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:35.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.753+0000 7fb34903d700 1 --2- 172.21.15.3:0/1264885727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34415b8c0 0x7fb34414efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:35.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.753+0000 7fb343fff700 1 --2- 172.21.15.3:0/1264885727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb34415b4e0 0x7fb3440af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:35.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.753+0000 7fb34883c700 1 -- 172.21.15.3:0/1264885727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34415b8c0 msgr2=0x7fb34414efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:35.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.753+0000 7fb34883c700 1 --2- 172.21.15.3:0/1264885727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34415b8c0 0x7fb34414efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.753+0000 7fb34883c700 1 -- 172.21.15.3:0/1264885727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb34415b4e0 msgr2=0x7fb3440af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:35.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.753+0000 7fb34883c700 1 --2- 172.21.15.3:0/1264885727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb34415b4e0 0x7fb3440af070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.753+0000 7fb34883c700 1 -- 172.21.15.3:0/1264885727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c009a70 con 0x7fb3440d1910 2024-05-08T21:22:35.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.754+0000 7fb34883c700 1 --2- 172.21.15.3:0/1264885727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3440d1910 0x7fb3440b5f50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb334009b90 tx=0x7fb334004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:35.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.754+0000 7fb3417fa700 1 -- 172.21.15.3:0/1264885727 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334004590 con 0x7fb3440d1910 2024-05-08T21:22:35.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.754+0000 7fb3417fa700 1 -- 172.21.15.3:0/1264885727 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33401f070 con 0x7fb3440d1910 2024-05-08T21:22:35.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.754+0000 7fb34aaa0700 1 -- 172.21.15.3:0/1264885727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3340094a0 con 0x7fb3440d1910 2024-05-08T21:22:35.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.754+0000 7fb3417fa700 1 -- 172.21.15.3:0/1264885727 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334011870 con 0x7fb3440d1910 2024-05-08T21:22:35.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.754+0000 7fb34aaa0700 1 -- 172.21.15.3:0/1264885727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3440b0160 con 0x7fb3440d1910 2024-05-08T21:22:35.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.755+0000 7fb34aaa0700 1 -- 172.21.15.3:0/1264885727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb344074bc0 con 0x7fb3440d1910 2024-05-08T21:22:35.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.755+0000 7fb3417fa700 1 -- 172.21.15.3:0/1264885727 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb3340119d0 con 0x7fb3440d1910 2024-05-08T21:22:35.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.756+0000 7fb3417fa700 1 --2- 172.21.15.3:0/1264885727 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb33006ac90 0x7fb33006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:35.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.756+0000 7fb3417fa700 1 -- 172.21.15.3:0/1264885727 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb334021070 con 0x7fb3440d1910 2024-05-08T21:22:35.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.756+0000 7fb343fff700 1 --2- 172.21.15.3:0/1264885727 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb33006ac90 0x7fb33006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:35.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.756+0000 7fb343fff700 1 --2- 172.21.15.3:0/1264885727 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb33006ac90 0x7fb33006d150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb344156e00 tx=0x7fb338007a20 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:35.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.760+0000 7fb3417fa700 1 -- 172.21.15.3:0/1264885727 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb334055ec0 con 0x7fb3440d1910 2024-05-08T21:22:35.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.978+0000 7fb34aaa0700 1 -- 172.21.15.3:0/1264885727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb3440b0fe0 con 0x7fb3440d1910 2024-05-08T21:22:35.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.979+0000 7fb3417fa700 1 -- 172.21.15.3:0/1264885727 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb33402b0a0 con 0x7fb3440d1910 2024-05-08T21:22:35.981 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:22:35.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.981+0000 7fb34aaa0700 1 -- 172.21.15.3:0/1264885727 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb33006ac90 msgr2=0x7fb33006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:35.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.981+0000 7fb34aaa0700 1 --2- 172.21.15.3:0/1264885727 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb33006ac90 0x7fb33006d150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb344156e00 tx=0x7fb338007a20 comp rx=0 tx=0).stop 2024-05-08T21:22:35.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.981+0000 7fb34aaa0700 1 -- 172.21.15.3:0/1264885727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3440d1910 msgr2=0x7fb3440b5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:35.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.981+0000 7fb34aaa0700 1 --2- 172.21.15.3:0/1264885727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3440d1910 0x7fb3440b5f50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb334009b90 tx=0x7fb334004750 comp rx=0 tx=0).stop 2024-05-08T21:22:35.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.981+0000 7fb34aaa0700 1 -- 172.21.15.3:0/1264885727 shutdown_connections 2024-05-08T21:22:35.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.981+0000 7fb34aaa0700 1 --2- 172.21.15.3:0/1264885727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34415b8c0 0x7fb34414efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.981+0000 7fb34aaa0700 1 --2- 172.21.15.3:0/1264885727 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb33006ac90 0x7fb33006d150 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.981+0000 7fb34aaa0700 1 --2- 172.21.15.3:0/1264885727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3440d1910 0x7fb3440b5f50 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.981+0000 7fb34aaa0700 1 --2- 172.21.15.3:0/1264885727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb34415b4e0 0x7fb3440af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:35.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.981+0000 7fb34aaa0700 1 -- 172.21.15.3:0/1264885727 >> 172.21.15.3:0/1264885727 conn(0x7fb34408afe0 msgr2=0x7fb3440b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:35.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.981+0000 7fb34aaa0700 1 -- 172.21.15.3:0/1264885727 shutdown_connections 2024-05-08T21:22:35.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:35.981+0000 7fb34aaa0700 1 -- 172.21.15.3:0/1264885727 wait complete. 2024-05-08T21:22:35.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:22:36.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.131+0000 7fa0fb5fb700 1 Processor -- start 2024-05-08T21:22:36.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.132+0000 7fa0fb5fb700 1 -- start start 2024-05-08T21:22:36.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.132+0000 7fa0fb5fb700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa0f40cf840 0x7fa0f40af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.132+0000 7fa0fb5fb700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa0f406a960 0x7fa0f415d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.132+0000 7fa0fb5fb700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0f40d0eb0 0x7fa0f415da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.132+0000 7fa0fb5fb700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f40cd980 con 0x7fa0f40cf840 2024-05-08T21:22:36.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0fb5fb700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f40ccdb0 con 0x7fa0f406a960 2024-05-08T21:22:36.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0fb5fb700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f40cd050 con 0x7fa0f40d0eb0 2024-05-08T21:22:36.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0f9397700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa0f40cf840 0x7fa0f40af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0f9397700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa0f40cf840 0x7fa0f40af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51428/0 (socket says 172.21.15.3:51428) 2024-05-08T21:22:36.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0f9397700 1 -- 172.21.15.3:0/914251536 learned_addr learned my addr 172.21.15.3:0/914251536 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:36.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0f8b96700 1 --2- 172.21.15.3:0/914251536 >> v2:172.21.15.186:3300/0 conn(0x7fa0f406a960 0x7fa0f415d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0f9b98700 1 --2- 172.21.15.3:0/914251536 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0f40d0eb0 0x7fa0f415da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0f9397700 1 -- 172.21.15.3:0/914251536 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0f40d0eb0 msgr2=0x7fa0f415da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:36.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0f9397700 1 --2- 172.21.15.3:0/914251536 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0f40d0eb0 0x7fa0f415da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0f9397700 1 -- 172.21.15.3:0/914251536 >> v2:172.21.15.186:3300/0 conn(0x7fa0f406a960 msgr2=0x7fa0f415d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:36.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0f9397700 1 --2- 172.21.15.3:0/914251536 >> v2:172.21.15.186:3300/0 conn(0x7fa0f406a960 0x7fa0f415d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0f9397700 1 -- 172.21.15.3:0/914251536 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f40cd330 con 0x7fa0f40cf840 2024-05-08T21:22:36.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0f8b96700 1 --2- 172.21.15.3:0/914251536 >> v2:172.21.15.186:3300/0 conn(0x7fa0f406a960 0x7fa0f415d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:36.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0f9397700 1 --2- 172.21.15.3:0/914251536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0f40cf840 0x7fa0f40af440 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40717d0 tx=0x7fa0f0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7de8a3a8713b4cc server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:36.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0ebfff700 1 -- 172.21.15.3:0/914251536 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f000b870 con 0x7fa0f40cf840 2024-05-08T21:22:36.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0ebfff700 1 -- 172.21.15.3:0/914251536 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f000b9d0 con 0x7fa0f40cf840 2024-05-08T21:22:36.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.133+0000 7fa0ebfff700 1 -- 172.21.15.3:0/914251536 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f000bd60 con 0x7fa0f40cf840 2024-05-08T21:22:36.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.134+0000 7fa0fb5fb700 1 -- 172.21.15.3:0/914251536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0f40cf840 msgr2=0x7fa0f40af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:36.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.134+0000 7fa0fb5fb700 1 --2- 172.21.15.3:0/914251536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0f40cf840 0x7fa0f40af440 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40717d0 tx=0x7fa0f0009bd0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.134+0000 7fa0fb5fb700 1 -- 172.21.15.3:0/914251536 shutdown_connections 2024-05-08T21:22:36.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.134+0000 7fa0fb5fb700 1 --2- 172.21.15.3:0/914251536 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0f40d0eb0 0x7fa0f415da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.134+0000 7fa0fb5fb700 1 --2- 172.21.15.3:0/914251536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0f40cf840 0x7fa0f40af440 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.134+0000 7fa0fb5fb700 1 --2- 172.21.15.3:0/914251536 >> v2:172.21.15.186:3300/0 conn(0x7fa0f406a960 0x7fa0f415d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.134+0000 7fa0fb5fb700 1 -- 172.21.15.3:0/914251536 >> 172.21.15.3:0/914251536 conn(0x7fa0f40d0320 msgr2=0x7fa0f40d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:36.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.134+0000 7fa0fb5fb700 1 -- 172.21.15.3:0/914251536 shutdown_connections 2024-05-08T21:22:36.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.134+0000 7fa0fb5fb700 1 -- 172.21.15.3:0/914251536 wait complete. 2024-05-08T21:22:36.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.135+0000 7fa0fb5fb700 1 Processor -- start 2024-05-08T21:22:36.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.136+0000 7fa0fb5fb700 1 -- start start 2024-05-08T21:22:36.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.136+0000 7fa0fb5fb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0f406a960 0x7fa0f415dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.136+0000 7fa0fb5fb700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0f40cf840 0x7fa0f40c56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.136+0000 7fa0fb5fb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0f40d0eb0 0x7fa0f414b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.136+0000 7fa0fb5fb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f4160590 con 0x7fa0f40d0eb0 2024-05-08T21:22:36.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.136+0000 7fa0fb5fb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f4160870 con 0x7fa0f406a960 2024-05-08T21:22:36.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.136+0000 7fa0fb5fb700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f4160b50 con 0x7fa0f40cf840 2024-05-08T21:22:36.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.136+0000 7fa0f9b98700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0f40d0eb0 0x7fa0f414b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.136+0000 7fa0f9b98700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0f40d0eb0 0x7fa0f414b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51434/0 (socket says 172.21.15.3:51434) 2024-05-08T21:22:36.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.136+0000 7fa0f9b98700 1 -- 172.21.15.3:0/2295698470 learned_addr learned my addr 172.21.15.3:0/2295698470 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:36.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.137+0000 7fa0f9397700 1 --2- 172.21.15.3:0/2295698470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0f406a960 0x7fa0f415dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.137+0000 7fa0f8b96700 1 --2- 172.21.15.3:0/2295698470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0f40cf840 0x7fa0f40c56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.137+0000 7fa0f9b98700 1 -- 172.21.15.3:0/2295698470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0f40cf840 msgr2=0x7fa0f40c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:36.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.137+0000 7fa0f9b98700 1 --2- 172.21.15.3:0/2295698470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0f40cf840 0x7fa0f40c56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.137+0000 7fa0f9b98700 1 -- 172.21.15.3:0/2295698470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0f406a960 msgr2=0x7fa0f415dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:36.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.137+0000 7fa0f9b98700 1 --2- 172.21.15.3:0/2295698470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0f406a960 0x7fa0f415dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.137+0000 7fa0f9b98700 1 -- 172.21.15.3:0/2295698470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f0009550 con 0x7fa0f40d0eb0 2024-05-08T21:22:36.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.137+0000 7fa0f8b96700 1 --2- 172.21.15.3:0/2295698470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0f40cf840 0x7fa0f40c56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:36.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.137+0000 7fa0f9b98700 1 --2- 172.21.15.3:0/2295698470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0f40d0eb0 0x7fa0f414b260 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec007e30 tx=0x7fa0ec00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:36.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.137+0000 7fa0e9ffb700 1 -- 172.21.15.3:0/2295698470 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec00eea0 con 0x7fa0f40d0eb0 2024-05-08T21:22:36.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.137+0000 7fa0e9ffb700 1 -- 172.21.15.3:0/2295698470 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ec014420 con 0x7fa0f40d0eb0 2024-05-08T21:22:36.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.137+0000 7fa0fb5fb700 1 -- 172.21.15.3:0/2295698470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f4160e30 con 0x7fa0f40d0eb0 2024-05-08T21:22:36.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.137+0000 7fa0e9ffb700 1 -- 172.21.15.3:0/2295698470 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec0147d0 con 0x7fa0f40d0eb0 2024-05-08T21:22:36.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.137+0000 7fa0fb5fb700 1 -- 172.21.15.3:0/2295698470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f415dff0 con 0x7fa0f40d0eb0 2024-05-08T21:22:36.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.138+0000 7fa0fb5fb700 1 -- 172.21.15.3:0/2295698470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f4068000 con 0x7fa0f40d0eb0 2024-05-08T21:22:36.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.139+0000 7fa0e9ffb700 1 -- 172.21.15.3:0/2295698470 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa0ec014930 con 0x7fa0f40d0eb0 2024-05-08T21:22:36.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.139+0000 7fa0e9ffb700 1 --2- 172.21.15.3:0/2295698470 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa0d806ac90 0x7fa0d806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.139+0000 7fa0e9ffb700 1 -- 172.21.15.3:0/2295698470 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa0ec089ad0 con 0x7fa0f40d0eb0 2024-05-08T21:22:36.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.139+0000 7fa0f9397700 1 --2- 172.21.15.3:0/2295698470 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa0d806ac90 0x7fa0d806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.140+0000 7fa0f9397700 1 --2- 172.21.15.3:0/2295698470 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa0d806ac90 0x7fa0d806d150 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0014040 tx=0x7fa0f0008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:36.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.144+0000 7fa0e9ffb700 1 -- 172.21.15.3:0/2295698470 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fa0ec054cc0 con 0x7fa0f40d0eb0 2024-05-08T21:22:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.414+0000 7fa0fb5fb700 1 -- 172.21.15.3:0/2295698470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0f40b6530 con 0x7fa0f40d0eb0 2024-05-08T21:22:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.415+0000 7fa0e9ffb700 1 -- 172.21.15.3:0/2295698470 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7fa0f40b6530 con 0x7fa0f40d0eb0 2024-05-08T21:22:36.417 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:36.417 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/2839854293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2839854293},{"type":"v1","addr":"172.21.15.3:6834","nonce":2839854293}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1878326567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":1878326567},{"type":"v1","addr":"172.21.15.3:6835","nonce":1878326567}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/3624849830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3624849830},{"type":"v1","addr":"172.21.15.186:6836","nonce":3624849830}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:21:52.062548+0000","modified":"2024-05-08T21:21:55.079806+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6837/1462886680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1462886680},{"type":"v1","addr":"172.21.15.186:6837","nonce":1462886680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-05-08T21:22:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.417+0000 7fa0fb5fb700 1 -- 172.21.15.3:0/2295698470 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa0d806ac90 msgr2=0x7fa0d806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.417+0000 7fa0fb5fb700 1 --2- 172.21.15.3:0/2295698470 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa0d806ac90 0x7fa0d806d150 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0014040 tx=0x7fa0f0008030 comp rx=0 tx=0).stop 2024-05-08T21:22:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.417+0000 7fa0fb5fb700 1 -- 172.21.15.3:0/2295698470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0f40d0eb0 msgr2=0x7fa0f414b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.417+0000 7fa0fb5fb700 1 --2- 172.21.15.3:0/2295698470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0f40d0eb0 0x7fa0f414b260 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec007e30 tx=0x7fa0ec00e3c0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.417+0000 7fa0fb5fb700 1 -- 172.21.15.3:0/2295698470 shutdown_connections 2024-05-08T21:22:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.417+0000 7fa0fb5fb700 1 --2- 172.21.15.3:0/2295698470 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa0d806ac90 0x7fa0d806d150 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.417+0000 7fa0fb5fb700 1 --2- 172.21.15.3:0/2295698470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0f40d0eb0 0x7fa0f414b260 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.417+0000 7fa0fb5fb700 1 --2- 172.21.15.3:0/2295698470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0f406a960 0x7fa0f415dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.417+0000 7fa0fb5fb700 1 --2- 172.21.15.3:0/2295698470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0f40cf840 0x7fa0f40c56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.417+0000 7fa0fb5fb700 1 -- 172.21.15.3:0/2295698470 >> 172.21.15.3:0/2295698470 conn(0x7fa0f40d0320 msgr2=0x7fa0f4158630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.417+0000 7fa0fb5fb700 1 -- 172.21.15.3:0/2295698470 shutdown_connections 2024-05-08T21:22:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.417+0000 7fa0fb5fb700 1 -- 172.21.15.3:0/2295698470 wait complete. 2024-05-08T21:22:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 8 2024-05-08T21:22:36.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:36.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.567+0000 7f41f41cf700 1 Processor -- start 2024-05-08T21:22:36.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.567+0000 7f41f41cf700 1 -- start start 2024-05-08T21:22:36.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.568+0000 7f41f41cf700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41ec161b60 0x7f41ec06f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.568+0000 7f41f41cf700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41ec0c3a00 0x7f41ec1659a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.568+0000 7f41f41cf700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ec166760 con 0x7f41ec161f40 2024-05-08T21:22:36.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.568+0000 7f41f41cf700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41ec06e360 con 0x7f41ec161b60 2024-05-08T21:22:36.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.568+0000 7f41f41cf700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec06e640 con 0x7f41ec0c3a00 2024-05-08T21:22:36.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.568+0000 7f41f276c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41ec0c3a00 0x7f41ec1659a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.568+0000 7f41f1f6b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41ec161b60 0x7f41ec06f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.568+0000 7f41f176a700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f41ec161f40 0x7f41ec0c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:54036/0 (socket says 172.21.15.3:54036) 2024-05-08T21:22:36.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.568+0000 7f41f176a700 1 -- 172.21.15.3:0/2740872272 learned_addr learned my addr 172.21.15.3:0/2740872272 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:36.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.569+0000 7f41f276c700 1 -- 172.21.15.3:0/2740872272 >> v2:172.21.15.186:3300/0 conn(0x7f41ec161b60 msgr2=0x7f41ec06f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:36.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.569+0000 7f41f276c700 1 --2- 172.21.15.3:0/2740872272 >> v2:172.21.15.186:3300/0 conn(0x7f41ec161b60 0x7f41ec06f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.569+0000 7f41f276c700 1 -- 172.21.15.3:0/2740872272 >> v1:172.21.15.186:6789/0 conn(0x7f41ec161f40 legacy=0x7f41ec0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:36.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.569+0000 7f41f276c700 1 -- 172.21.15.3:0/2740872272 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ec06e920 con 0x7f41ec0c3a00 2024-05-08T21:22:36.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.569+0000 7f41f1f6b700 1 --2- 172.21.15.3:0/2740872272 >> v2:172.21.15.186:3300/0 conn(0x7f41ec161b60 0x7f41ec06f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:36.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.569+0000 7f41f276c700 1 --2- 172.21.15.3:0/2740872272 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41ec0c3a00 0x7f41ec1659a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f41e4009d90 tx=0x7f41e400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51cc940554e2f498 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:36.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.569+0000 7f41f0f69700 1 -- 172.21.15.3:0/2740872272 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e400e070 con 0x7f41ec0c3a00 2024-05-08T21:22:36.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.569+0000 7f41f0f69700 1 -- 172.21.15.3:0/2740872272 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e400bb20 con 0x7f41ec0c3a00 2024-05-08T21:22:36.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.569+0000 7f41f0f69700 1 -- 172.21.15.3:0/2740872272 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e4012430 con 0x7f41ec0c3a00 2024-05-08T21:22:36.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.570+0000 7f41f41cf700 1 -- 172.21.15.3:0/2740872272 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41ec0c3a00 msgr2=0x7f41ec1659a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:36.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.570+0000 7f41f41cf700 1 --2- 172.21.15.3:0/2740872272 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41ec0c3a00 0x7f41ec1659a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f41e4009d90 tx=0x7f41e400b330 comp rx=0 tx=0).stop 2024-05-08T21:22:36.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.570+0000 7f41f41cf700 1 -- 172.21.15.3:0/2740872272 shutdown_connections 2024-05-08T21:22:36.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.570+0000 7f41f41cf700 1 --2- 172.21.15.3:0/2740872272 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41ec0c3a00 0x7f41ec1659a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.570+0000 7f41f41cf700 1 --2- 172.21.15.3:0/2740872272 >> v2:172.21.15.186:3300/0 conn(0x7f41ec161b60 0x7f41ec06f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.570+0000 7f41f41cf700 1 -- 172.21.15.3:0/2740872272 >> 172.21.15.3:0/2740872272 conn(0x7f41ec0d0320 msgr2=0x7f41ec0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:36.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.570+0000 7f41f41cf700 1 -- 172.21.15.3:0/2740872272 shutdown_connections 2024-05-08T21:22:36.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.570+0000 7f41f41cf700 1 -- 172.21.15.3:0/2740872272 wait complete. 2024-05-08T21:22:36.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.571+0000 7f41f41cf700 1 Processor -- start 2024-05-08T21:22:36.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.571+0000 7f41f41cf700 1 -- start start 2024-05-08T21:22:36.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.572+0000 7f41f41cf700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41ec0c3a00 0x7f41ec0b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.572+0000 7f41f41cf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41ec161b60 0x7f41ec1509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.572+0000 7f41f41cf700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41ec161f40 0x7f41ec0ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.572+0000 7f41f41cf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec150f20 con 0x7f41ec161b60 2024-05-08T21:22:36.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.572+0000 7f41f41cf700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec164760 con 0x7f41ec161f40 2024-05-08T21:22:36.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.572+0000 7f41f41cf700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec164a40 con 0x7f41ec0c3a00 2024-05-08T21:22:36.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.572+0000 7f41f176a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41ec161b60 0x7f41ec1509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.572+0000 7f41f176a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41ec161b60 0x7f41ec1509e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51440/0 (socket says 172.21.15.3:51440) 2024-05-08T21:22:36.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.572+0000 7f41f176a700 1 -- 172.21.15.3:0/4282757411 learned_addr learned my addr 172.21.15.3:0/4282757411 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:36.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.572+0000 7f41f276c700 1 --2- 172.21.15.3:0/4282757411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41ec161f40 0x7f41ec0ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.572+0000 7f41f1f6b700 1 --2- 172.21.15.3:0/4282757411 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41ec0c3a00 0x7f41ec0b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.573+0000 7f41f176a700 1 -- 172.21.15.3:0/4282757411 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41ec0c3a00 msgr2=0x7f41ec0b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:36.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.573+0000 7f41f176a700 1 --2- 172.21.15.3:0/4282757411 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41ec0c3a00 0x7f41ec0b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.573+0000 7f41f176a700 1 -- 172.21.15.3:0/4282757411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41ec161f40 msgr2=0x7f41ec0ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:36.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.573+0000 7f41f176a700 1 --2- 172.21.15.3:0/4282757411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41ec161f40 0x7f41ec0ad8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.573+0000 7f41f176a700 1 -- 172.21.15.3:0/4282757411 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e80094a0 con 0x7f41ec161b60 2024-05-08T21:22:36.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.573+0000 7f41f176a700 1 --2- 172.21.15.3:0/4282757411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41ec161b60 0x7f41ec1509e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f41dc00c480 tx=0x7f41dc00c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:36.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.573+0000 7f41e27fc700 1 -- 172.21.15.3:0/4282757411 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc005d70 con 0x7f41ec161b60 2024-05-08T21:22:36.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.573+0000 7f41e27fc700 1 -- 172.21.15.3:0/4282757411 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41dc00ce90 con 0x7f41ec161b60 2024-05-08T21:22:36.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.573+0000 7f41f41cf700 1 -- 172.21.15.3:0/4282757411 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e4009a70 con 0x7f41ec161b60 2024-05-08T21:22:36.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.573+0000 7f41e27fc700 1 -- 172.21.15.3:0/4282757411 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc0072d0 con 0x7f41ec161b60 2024-05-08T21:22:36.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.573+0000 7f41f41cf700 1 -- 172.21.15.3:0/4282757411 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ec165000 con 0x7f41ec161b60 2024-05-08T21:22:36.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.574+0000 7f41f41cf700 1 -- 172.21.15.3:0/4282757411 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41ec068000 con 0x7f41ec161b60 2024-05-08T21:22:36.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.575+0000 7f41e27fc700 1 -- 172.21.15.3:0/4282757411 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f41dc007430 con 0x7f41ec161b60 2024-05-08T21:22:36.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.575+0000 7f41e27fc700 1 --2- 172.21.15.3:0/4282757411 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f41d006ac90 0x7f41d006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.575+0000 7f41e27fc700 1 -- 172.21.15.3:0/4282757411 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f41dc088540 con 0x7f41ec161b60 2024-05-08T21:22:36.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.575+0000 7f41f1f6b700 1 --2- 172.21.15.3:0/4282757411 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f41d006ac90 0x7f41d006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.576+0000 7f41f1f6b700 1 --2- 172.21.15.3:0/4282757411 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f41d006ac90 0x7f41d006d150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009d70 tx=0x7f41e8009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:36.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.580+0000 7f41e27fc700 1 -- 172.21.15.3:0/4282757411 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f41dc0537e0 con 0x7f41ec161b60 2024-05-08T21:22:36.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.800+0000 7f41f41cf700 1 -- 172.21.15.3:0/4282757411 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41ec0cf4a0 con 0x7f41ec161b60 2024-05-08T21:22:36.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.802+0000 7f41e27fc700 1 -- 172.21.15.3:0/4282757411 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16179 (secure 0 0 0) 0x7f41dc056e00 con 0x7f41ec161b60 2024-05-08T21:22:36.803 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:36.803 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":22,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:22:35.652541+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.804+0000 7f41f41cf700 1 -- 172.21.15.3:0/4282757411 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f41d006ac90 msgr2=0x7f41d006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.804+0000 7f41f41cf700 1 --2- 172.21.15.3:0/4282757411 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f41d006ac90 0x7f41d006d150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009d70 tx=0x7f41e8009210 comp rx=0 tx=0).stop 2024-05-08T21:22:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.804+0000 7f41f41cf700 1 -- 172.21.15.3:0/4282757411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41ec161b60 msgr2=0x7f41ec1509e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.804+0000 7f41f41cf700 1 --2- 172.21.15.3:0/4282757411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41ec161b60 0x7f41ec1509e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f41dc00c480 tx=0x7f41dc00c790 comp rx=0 tx=0).stop 2024-05-08T21:22:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.804+0000 7f41f41cf700 1 -- 172.21.15.3:0/4282757411 shutdown_connections 2024-05-08T21:22:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.804+0000 7f41f41cf700 1 --2- 172.21.15.3:0/4282757411 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41ec0c3a00 0x7f41ec0b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.804+0000 7f41f41cf700 1 --2- 172.21.15.3:0/4282757411 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f41d006ac90 0x7f41d006d150 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.804+0000 7f41f41cf700 1 --2- 172.21.15.3:0/4282757411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41ec161f40 0x7f41ec0ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.804+0000 7f41f41cf700 1 --2- 172.21.15.3:0/4282757411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41ec161b60 0x7f41ec1509e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.804+0000 7f41f41cf700 1 -- 172.21.15.3:0/4282757411 >> 172.21.15.3:0/4282757411 conn(0x7f41ec0d0320 msgr2=0x7f41ec0ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:36.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.804+0000 7f41f41cf700 1 -- 172.21.15.3:0/4282757411 shutdown_connections 2024-05-08T21:22:36.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.804+0000 7f41f41cf700 1 -- 172.21.15.3:0/4282757411 wait complete. 2024-05-08T21:22:36.819 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-08T21:22:36.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:22:36.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.954+0000 7f2f04ea0700 1 Processor -- start 2024-05-08T21:22:36.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.954+0000 7f2f04ea0700 1 -- start start 2024-05-08T21:22:36.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.955+0000 7f2f04ea0700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2f000cf040 0x7f2f001631e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.955+0000 7f2f04ea0700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0015bd70 con 0x7f2f000cf4f0 2024-05-08T21:22:36.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.955+0000 7f2f04ea0700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0015c050 con 0x7f2f000cf9d0 2024-05-08T21:22:36.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.955+0000 7f2f04ea0700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0015c330 con 0x7f2f000cf040 2024-05-08T21:22:36.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.955+0000 7f2efed9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2f000cf4f0 0x7f2f00165730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:48066/0 (socket says 172.21.15.3:48066) 2024-05-08T21:22:36.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.955+0000 7f2efed9d700 1 -- 172.21.15.3:0/3880158973 learned_addr learned my addr 172.21.15.3:0/3880158973 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:36.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.955+0000 7f2efdd9b700 1 --2- 172.21.15.3:0/3880158973 >> v2:172.21.15.186:3300/0 conn(0x7f2f000cf040 0x7f2f001631e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.956+0000 7f2efd59a700 1 -- 172.21.15.3:0/3880158973 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712751212 0 0) 0x7f2f0015bd70 con 0x7f2f000cf4f0 2024-05-08T21:22:36.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.956+0000 7f2efd59a700 1 -- 172.21.15.3:0/3880158973 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee80035b0 con 0x7f2f000cf4f0 2024-05-08T21:22:36.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.956+0000 7f2efdd9b700 1 -- 172.21.15.3:0/3880158973 >> v1:172.21.15.186:6789/0 conn(0x7f2f000cf9d0 legacy=0x7f2f000cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:36.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.956+0000 7f2efdd9b700 1 -- 172.21.15.3:0/3880158973 >> v1:172.21.15.3:6789/0 conn(0x7f2f000cf4f0 legacy=0x7f2f00165730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:36.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.956+0000 7f2efdd9b700 1 -- 172.21.15.3:0/3880158973 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0015c610 con 0x7f2f000cf040 2024-05-08T21:22:36.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.956+0000 7f2efdd9b700 1 --2- 172.21.15.3:0/3880158973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f000cf040 0x7f2f001631e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009870 tx=0x7f2ef4009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce51e7ab0d3cdc8d server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:36.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.956+0000 7f2efd59a700 1 -- 172.21.15.3:0/3880158973 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4011830 con 0x7f2f000cf040 2024-05-08T21:22:36.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.956+0000 7f2efd59a700 1 -- 172.21.15.3:0/3880158973 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef4011990 con 0x7f2f000cf040 2024-05-08T21:22:36.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.956+0000 7f2efd59a700 1 -- 172.21.15.3:0/3880158973 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4011d20 con 0x7f2f000cf040 2024-05-08T21:22:36.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.957+0000 7f2f04ea0700 1 -- 172.21.15.3:0/3880158973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f000cf040 msgr2=0x7f2f001631e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:36.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.957+0000 7f2f04ea0700 1 --2- 172.21.15.3:0/3880158973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f000cf040 0x7f2f001631e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009870 tx=0x7f2ef4009b80 comp rx=0 tx=0).stop 2024-05-08T21:22:36.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.957+0000 7f2f04ea0700 1 -- 172.21.15.3:0/3880158973 shutdown_connections 2024-05-08T21:22:36.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.957+0000 7f2f04ea0700 1 --2- 172.21.15.3:0/3880158973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f000cf040 0x7f2f001631e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.957+0000 7f2f04ea0700 1 -- 172.21.15.3:0/3880158973 >> 172.21.15.3:0/3880158973 conn(0x7f2f000b2f60 msgr2=0x7f2f000b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:36.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.957+0000 7f2f04ea0700 1 -- 172.21.15.3:0/3880158973 shutdown_connections 2024-05-08T21:22:36.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.957+0000 7f2f04ea0700 1 -- 172.21.15.3:0/3880158973 wait complete. 2024-05-08T21:22:36.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.959+0000 7f2f04ea0700 1 Processor -- start 2024-05-08T21:22:36.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.959+0000 7f2f04ea0700 1 -- start start 2024-05-08T21:22:36.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.959+0000 7f2f04ea0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f000cf040 0x7f2f001f1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.959+0000 7f2f04ea0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f000cf4f0 0x7f2f000c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.959+0000 7f2f04ea0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f000cf9d0 0x7f2f001506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.959+0000 7f2f04ea0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f00154d70 con 0x7f2f000cf4f0 2024-05-08T21:22:36.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.959+0000 7f2f04ea0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f00155050 con 0x7f2f000cf9d0 2024-05-08T21:22:36.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.959+0000 7f2f04ea0700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f00155330 con 0x7f2f000cf040 2024-05-08T21:22:36.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.960+0000 7f2efdd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f000cf4f0 0x7f2f000c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.960+0000 7f2efdd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f000cf4f0 0x7f2f000c64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51446/0 (socket says 172.21.15.3:51446) 2024-05-08T21:22:36.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.960+0000 7f2efdd9b700 1 -- 172.21.15.3:0/1443918572 learned_addr learned my addr 172.21.15.3:0/1443918572 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:36.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.960+0000 7f2efed9d700 1 --2- 172.21.15.3:0/1443918572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f000cf9d0 0x7f2f001506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.960+0000 7f2efe59c700 1 --2- 172.21.15.3:0/1443918572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f000cf040 0x7f2f001f1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.960+0000 7f2efdd9b700 1 -- 172.21.15.3:0/1443918572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f000cf040 msgr2=0x7f2f001f1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:36.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.960+0000 7f2efdd9b700 1 --2- 172.21.15.3:0/1443918572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f000cf040 0x7f2f001f1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.960+0000 7f2efdd9b700 1 -- 172.21.15.3:0/1443918572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f000cf9d0 msgr2=0x7f2f001506c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:36.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.960+0000 7f2efdd9b700 1 --2- 172.21.15.3:0/1443918572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f000cf9d0 0x7f2f001506c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:36.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.960+0000 7f2efdd9b700 1 -- 172.21.15.3:0/1443918572 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef40094a0 con 0x7f2f000cf4f0 2024-05-08T21:22:36.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.960+0000 7f2efdd9b700 1 --2- 172.21.15.3:0/1443918572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f000cf4f0 0x7f2f000c64f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4014040 tx=0x7f2ef4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:36.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.960+0000 7f2eeeffd700 1 -- 172.21.15.3:0/1443918572 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef40036a0 con 0x7f2f000cf4f0 2024-05-08T21:22:36.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.960+0000 7f2eeeffd700 1 -- 172.21.15.3:0/1443918572 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef401e070 con 0x7f2f000cf4f0 2024-05-08T21:22:36.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.960+0000 7f2f04ea0700 1 -- 172.21.15.3:0/1443918572 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f00155610 con 0x7f2f000cf4f0 2024-05-08T21:22:36.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.960+0000 7f2eeeffd700 1 -- 172.21.15.3:0/1443918572 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4003a20 con 0x7f2f000cf4f0 2024-05-08T21:22:36.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.960+0000 7f2f04ea0700 1 -- 172.21.15.3:0/1443918572 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f000c6a30 con 0x7f2f000cf4f0 2024-05-08T21:22:36.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.961+0000 7f2f04ea0700 1 -- 172.21.15.3:0/1443918572 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f00155b30 con 0x7f2f000cf4f0 2024-05-08T21:22:36.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.962+0000 7f2eeeffd700 1 -- 172.21.15.3:0/1443918572 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2ef4003b80 con 0x7f2f000cf4f0 2024-05-08T21:22:36.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.962+0000 7f2eeeffd700 1 --2- 172.21.15.3:0/1443918572 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2ee006ac90 0x7f2ee006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:36.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.962+0000 7f2eeeffd700 1 -- 172.21.15.3:0/1443918572 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2ef408a220 con 0x7f2f000cf4f0 2024-05-08T21:22:36.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.963+0000 7f2efe59c700 1 --2- 172.21.15.3:0/1443918572 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2ee006ac90 0x7f2ee006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:36.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.963+0000 7f2efe59c700 1 --2- 172.21.15.3:0/1443918572 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2ee006ac90 0x7f2ee006d150 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0007fd0 tx=0x7f2ef0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:36.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:36.967+0000 7f2eeeffd700 1 -- 172.21.15.3:0/1443918572 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2f00155b30 con 0x7f2f000cf4f0 2024-05-08T21:22:37.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.235+0000 7f2f04ea0700 1 -- 172.21.15.3:0/1443918572 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f00155b30 con 0x7f2f000cf4f0 2024-05-08T21:22:37.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.235+0000 7f2eeeffd700 1 -- 172.21.15.3:0/1443918572 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f2f00155b30 con 0x7f2f000cf4f0 2024-05-08T21:22:37.237 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:37.237 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/2839854293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2839854293},{"type":"v1","addr":"172.21.15.3:6834","nonce":2839854293}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1878326567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":1878326567},{"type":"v1","addr":"172.21.15.3:6835","nonce":1878326567}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/3624849830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3624849830},{"type":"v1","addr":"172.21.15.186:6836","nonce":3624849830}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:21:52.062548+0000","modified":"2024-05-08T21:21:55.079806+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6837/1462886680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1462886680},{"type":"v1","addr":"172.21.15.186:6837","nonce":1462886680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-05-08T21:22:37.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.237+0000 7f2f04ea0700 1 -- 172.21.15.3:0/1443918572 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2ee006ac90 msgr2=0x7f2ee006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:37.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.237+0000 7f2f04ea0700 1 --2- 172.21.15.3:0/1443918572 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2ee006ac90 0x7f2ee006d150 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0007fd0 tx=0x7f2ef0009040 comp rx=0 tx=0).stop 2024-05-08T21:22:37.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.237+0000 7f2f04ea0700 1 -- 172.21.15.3:0/1443918572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f000cf4f0 msgr2=0x7f2f000c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:37.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.237+0000 7f2f04ea0700 1 --2- 172.21.15.3:0/1443918572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f000cf4f0 0x7f2f000c64f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4014040 tx=0x7f2ef4004750 comp rx=0 tx=0).stop 2024-05-08T21:22:37.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.237+0000 7f2f04ea0700 1 -- 172.21.15.3:0/1443918572 shutdown_connections 2024-05-08T21:22:37.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.237+0000 7f2f04ea0700 1 --2- 172.21.15.3:0/1443918572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f000cf040 0x7f2f001f1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.237+0000 7f2f04ea0700 1 --2- 172.21.15.3:0/1443918572 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2ee006ac90 0x7f2ee006d150 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.237+0000 7f2f04ea0700 1 --2- 172.21.15.3:0/1443918572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f000cf9d0 0x7f2f001506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.237+0000 7f2f04ea0700 1 --2- 172.21.15.3:0/1443918572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f000cf4f0 0x7f2f000c64f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.237+0000 7f2f04ea0700 1 -- 172.21.15.3:0/1443918572 >> 172.21.15.3:0/1443918572 conn(0x7f2f000b2f60 msgr2=0x7f2f0015d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:37.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.237+0000 7f2f04ea0700 1 -- 172.21.15.3:0/1443918572 shutdown_connections 2024-05-08T21:22:37.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.238+0000 7f2f04ea0700 1 -- 172.21.15.3:0/1443918572 wait complete. 2024-05-08T21:22:37.240 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 8 2024-05-08T21:22:37.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:22:37.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.387+0000 7f1f6543a700 1 Processor -- start 2024-05-08T21:22:37.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.387+0000 7f1f6543a700 1 -- start start 2024-05-08T21:22:37.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.388+0000 7f1f6543a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f60099cc0 0x7f1f600d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:37.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.388+0000 7f1f6543a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1f6009a0a0 0x7f1f600d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:37.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.388+0000 7f1f6543a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1f6015d710 0x7f1f6015dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:37.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.388+0000 7f1f6543a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f6006c820 con 0x7f1f6015d710 2024-05-08T21:22:37.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.388+0000 7f1f6543a700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f6006bc10 con 0x7f1f6009a0a0 2024-05-08T21:22:37.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.388+0000 7f1f6543a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6006bef0 con 0x7f1f60099cc0 2024-05-08T21:22:37.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.388+0000 7f1f5f7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1f6015d710 0x7f1f6015dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:37.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.388+0000 7f1f5f7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1f6015d710 0x7f1f6015dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51460/0 (socket says 172.21.15.3:51460) 2024-05-08T21:22:37.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.388+0000 7f1f5f7fe700 1 -- 172.21.15.3:0/1420027637 learned_addr learned my addr 172.21.15.3:0/1420027637 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:37.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.388+0000 7f1f5e7fc700 1 --2- 172.21.15.3:0/1420027637 >> v2:172.21.15.186:3300/0 conn(0x7f1f6009a0a0 0x7f1f600d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:37.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.388+0000 7f1f5effd700 1 --2- 172.21.15.3:0/1420027637 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f60099cc0 0x7f1f600d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:37.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.388+0000 7f1f5f7fe700 1 -- 172.21.15.3:0/1420027637 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f60099cc0 msgr2=0x7f1f600d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:37.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.388+0000 7f1f5f7fe700 1 --2- 172.21.15.3:0/1420027637 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f60099cc0 0x7f1f600d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.388+0000 7f1f5f7fe700 1 -- 172.21.15.3:0/1420027637 >> v2:172.21.15.186:3300/0 conn(0x7f1f6009a0a0 msgr2=0x7f1f600d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:37.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.388+0000 7f1f5f7fe700 1 --2- 172.21.15.3:0/1420027637 >> v2:172.21.15.186:3300/0 conn(0x7f1f6009a0a0 0x7f1f600d0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.388+0000 7f1f5f7fe700 1 -- 172.21.15.3:0/1420027637 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6006c1d0 con 0x7f1f6015d710 2024-05-08T21:22:37.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.389+0000 7f1f5f7fe700 1 --2- 172.21.15.3:0/1420027637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6015d710 0x7f1f6015dba0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1f60072670 tx=0x7f1f5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e06b69b7d64b8ad server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:37.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.389+0000 7f1f5dffb700 1 -- 172.21.15.3:0/1420027637 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5800e070 con 0x7f1f6015d710 2024-05-08T21:22:37.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.389+0000 7f1f5dffb700 1 -- 172.21.15.3:0/1420027637 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5800b910 con 0x7f1f6015d710 2024-05-08T21:22:37.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.389+0000 7f1f5dffb700 1 -- 172.21.15.3:0/1420027637 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5800bcc0 con 0x7f1f6015d710 2024-05-08T21:22:37.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.389+0000 7f1f6543a700 1 -- 172.21.15.3:0/1420027637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6015d710 msgr2=0x7f1f6015dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:37.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.389+0000 7f1f6543a700 1 --2- 172.21.15.3:0/1420027637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6015d710 0x7f1f6015dba0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1f60072670 tx=0x7f1f5800b330 comp rx=0 tx=0).stop 2024-05-08T21:22:37.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.389+0000 7f1f6543a700 1 -- 172.21.15.3:0/1420027637 shutdown_connections 2024-05-08T21:22:37.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.389+0000 7f1f6543a700 1 --2- 172.21.15.3:0/1420027637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6015d710 0x7f1f6015dba0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.389+0000 7f1f6543a700 1 --2- 172.21.15.3:0/1420027637 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f60099cc0 0x7f1f600d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.389+0000 7f1f6543a700 1 --2- 172.21.15.3:0/1420027637 >> v2:172.21.15.186:3300/0 conn(0x7f1f6009a0a0 0x7f1f600d0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.389+0000 7f1f6543a700 1 -- 172.21.15.3:0/1420027637 >> 172.21.15.3:0/1420027637 conn(0x7f1f600b2f80 msgr2=0x7f1f600b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:37.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.389+0000 7f1f6543a700 1 -- 172.21.15.3:0/1420027637 shutdown_connections 2024-05-08T21:22:37.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.389+0000 7f1f6543a700 1 -- 172.21.15.3:0/1420027637 wait complete. 2024-05-08T21:22:37.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.391+0000 7f1f6543a700 1 Processor -- start 2024-05-08T21:22:37.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.391+0000 7f1f6543a700 1 -- start start 2024-05-08T21:22:37.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f6543a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f60099cc0 0x7f1f601f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:37.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f6543a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6009a0a0 0x7f1f601e6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:37.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f6543a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f6015d710 0x7f1f601e6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f6543a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6015b300 con 0x7f1f6009a0a0 2024-05-08T21:22:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f6543a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600b4d70 con 0x7f1f6015d710 2024-05-08T21:22:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f6543a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600cd820 con 0x7f1f60099cc0 2024-05-08T21:22:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f5e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6009a0a0 0x7f1f601e6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f5e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6009a0a0 0x7f1f601e6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51466/0 (socket says 172.21.15.3:51466) 2024-05-08T21:22:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f5e7fc700 1 -- 172.21.15.3:0/914036404 learned_addr learned my addr 172.21.15.3:0/914036404 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f5effd700 1 --2- 172.21.15.3:0/914036404 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f60099cc0 0x7f1f601f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f5f7fe700 1 --2- 172.21.15.3:0/914036404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f6015d710 0x7f1f601e6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f5e7fc700 1 -- 172.21.15.3:0/914036404 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f60099cc0 msgr2=0x7f1f601f6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f5e7fc700 1 --2- 172.21.15.3:0/914036404 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f60099cc0 0x7f1f601f6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f5e7fc700 1 -- 172.21.15.3:0/914036404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f6015d710 msgr2=0x7f1f601e6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f5e7fc700 1 --2- 172.21.15.3:0/914036404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f6015d710 0x7f1f601e6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f5e7fc700 1 -- 172.21.15.3:0/914036404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f58009b30 con 0x7f1f6009a0a0 2024-05-08T21:22:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.392+0000 7f1f5e7fc700 1 --2- 172.21.15.3:0/914036404 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6009a0a0 0x7f1f601e6a00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400e9b0 tx=0x7f1f5400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.393+0000 7f1f477fe700 1 -- 172.21.15.3:0/914036404 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f54009710 con 0x7f1f6009a0a0 2024-05-08T21:22:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.393+0000 7f1f477fe700 1 -- 172.21.15.3:0/914036404 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5400ee50 con 0x7f1f6009a0a0 2024-05-08T21:22:37.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.393+0000 7f1f6543a700 1 -- 172.21.15.3:0/914036404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f600cdb00 con 0x7f1f6009a0a0 2024-05-08T21:22:37.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.393+0000 7f1f477fe700 1 -- 172.21.15.3:0/914036404 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5400ca50 con 0x7f1f6009a0a0 2024-05-08T21:22:37.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.393+0000 7f1f6543a700 1 -- 172.21.15.3:0/914036404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f600ce0c0 con 0x7f1f6009a0a0 2024-05-08T21:22:37.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.394+0000 7f1f6543a700 1 -- 172.21.15.3:0/914036404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f600ce430 con 0x7f1f6009a0a0 2024-05-08T21:22:37.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.394+0000 7f1f477fe700 1 -- 172.21.15.3:0/914036404 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1f5400cbb0 con 0x7f1f6009a0a0 2024-05-08T21:22:37.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.395+0000 7f1f477fe700 1 --2- 172.21.15.3:0/914036404 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1f4c06ac90 0x7f1f4c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:37.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.395+0000 7f1f477fe700 1 -- 172.21.15.3:0/914036404 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1f540893f0 con 0x7f1f6009a0a0 2024-05-08T21:22:37.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.395+0000 7f1f5effd700 1 --2- 172.21.15.3:0/914036404 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1f4c06ac90 0x7f1f4c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:37.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.395+0000 7f1f5effd700 1 --2- 172.21.15.3:0/914036404 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1f4c06ac90 0x7f1f4c06d150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1f601e7780 tx=0x7f1f50009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:37.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.399+0000 7f1f477fe700 1 -- 172.21.15.3:0/914036404 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f1f540544b0 con 0x7f1f6009a0a0 2024-05-08T21:22:37.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.667+0000 7f1f6543a700 1 -- 172.21.15.3:0/914036404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f600ce5e0 con 0x7f1f6009a0a0 2024-05-08T21:22:37.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.670+0000 7f1f477fe700 1 -- 172.21.15.3:0/914036404 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f1f600ce5e0 con 0x7f1f6009a0a0 2024-05-08T21:22:37.671 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:37.671 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/2839854293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2839854293},{"type":"v1","addr":"172.21.15.3:6834","nonce":2839854293}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1878326567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":1878326567},{"type":"v1","addr":"172.21.15.3:6835","nonce":1878326567}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/3624849830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3624849830},{"type":"v1","addr":"172.21.15.186:6836","nonce":3624849830}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:21:52.062548+0000","modified":"2024-05-08T21:21:55.079806+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6837/1462886680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1462886680},{"type":"v1","addr":"172.21.15.186:6837","nonce":1462886680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-05-08T21:22:37.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.672+0000 7f1f6543a700 1 -- 172.21.15.3:0/914036404 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1f4c06ac90 msgr2=0x7f1f4c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:37.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.672+0000 7f1f6543a700 1 --2- 172.21.15.3:0/914036404 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1f4c06ac90 0x7f1f4c06d150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1f601e7780 tx=0x7f1f50009210 comp rx=0 tx=0).stop 2024-05-08T21:22:37.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.672+0000 7f1f6543a700 1 -- 172.21.15.3:0/914036404 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6009a0a0 msgr2=0x7f1f601e6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:37.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.672+0000 7f1f6543a700 1 --2- 172.21.15.3:0/914036404 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6009a0a0 0x7f1f601e6a00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400e9b0 tx=0x7f1f5400ecc0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.672+0000 7f1f6543a700 1 -- 172.21.15.3:0/914036404 shutdown_connections 2024-05-08T21:22:37.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.672+0000 7f1f6543a700 1 --2- 172.21.15.3:0/914036404 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f60099cc0 0x7f1f601f6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.672+0000 7f1f6543a700 1 --2- 172.21.15.3:0/914036404 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1f4c06ac90 0x7f1f4c06d150 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.672+0000 7f1f6543a700 1 --2- 172.21.15.3:0/914036404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f6015d710 0x7f1f601e6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.672+0000 7f1f6543a700 1 --2- 172.21.15.3:0/914036404 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6009a0a0 0x7f1f601e6a00 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.672+0000 7f1f6543a700 1 -- 172.21.15.3:0/914036404 >> 172.21.15.3:0/914036404 conn(0x7f1f600b2f80 msgr2=0x7f1f6015cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:37.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.672+0000 7f1f6543a700 1 -- 172.21.15.3:0/914036404 shutdown_connections 2024-05-08T21:22:37.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.672+0000 7f1f6543a700 1 -- 172.21.15.3:0/914036404 wait complete. 2024-05-08T21:22:37.675 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 8 2024-05-08T21:22:37.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:37.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.822+0000 7f720004a700 1 Processor -- start 2024-05-08T21:22:37.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.822+0000 7f720004a700 1 -- start start 2024-05-08T21:22:37.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.823+0000 7f720004a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f71f80cf840 0x7f71f80af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:37.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.823+0000 7f720004a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71f806a960 0x7f71f815d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:37.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.823+0000 7f720004a700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f80cd980 con 0x7f71f80d0eb0 2024-05-08T21:22:37.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.823+0000 7f720004a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f80ccdb0 con 0x7f71f80cf840 2024-05-08T21:22:37.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.823+0000 7f720004a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f80cd050 con 0x7f71f806a960 2024-05-08T21:22:37.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.823+0000 7f71fdde6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f71f80cf840 0x7f71f80af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.823+0000 7f71fdde6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f71f80cf840 0x7f71f80af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51468/0 (socket says 172.21.15.3:51468) 2024-05-08T21:22:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.823+0000 7f71fdde6700 1 -- 172.21.15.3:0/1385735575 learned_addr learned my addr 172.21.15.3:0/1385735575 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.823+0000 7f71fd5e5700 1 --2- 172.21.15.3:0/1385735575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71f806a960 0x7f71f815d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.823+0000 7f71fdde6700 1 -- 172.21.15.3:0/1385735575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71f806a960 msgr2=0x7f71f815d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.823+0000 7f71fdde6700 1 --2- 172.21.15.3:0/1385735575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71f806a960 0x7f71f815d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.823+0000 7f71fdde6700 1 -- 172.21.15.3:0/1385735575 >> v1:172.21.15.186:6789/0 conn(0x7f71f80d0eb0 legacy=0x7f71f806dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.823+0000 7f71fdde6700 1 -- 172.21.15.3:0/1385735575 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f80cd330 con 0x7f71f80cf840 2024-05-08T21:22:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.823+0000 7f71fdde6700 1 --2- 172.21.15.3:0/1385735575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71f80cf840 0x7f71f80af440 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f71f4009920 tx=0x7f71f4009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd4043ae57394f61 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.824+0000 7f71fcde4700 1 -- 172.21.15.3:0/1385735575 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f4004230 con 0x7f71f80cf840 2024-05-08T21:22:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.824+0000 7f71fcde4700 1 -- 172.21.15.3:0/1385735575 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f4004390 con 0x7f71f80cf840 2024-05-08T21:22:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.824+0000 7f71fcde4700 1 -- 172.21.15.3:0/1385735575 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f4009e70 con 0x7f71f80cf840 2024-05-08T21:22:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.824+0000 7f720004a700 1 -- 172.21.15.3:0/1385735575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71f80cf840 msgr2=0x7f71f80af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.824+0000 7f720004a700 1 --2- 172.21.15.3:0/1385735575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71f80cf840 0x7f71f80af440 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f71f4009920 tx=0x7f71f4009c30 comp rx=0 tx=0).stop 2024-05-08T21:22:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.824+0000 7f720004a700 1 -- 172.21.15.3:0/1385735575 shutdown_connections 2024-05-08T21:22:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.824+0000 7f720004a700 1 --2- 172.21.15.3:0/1385735575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71f80cf840 0x7f71f80af440 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.824+0000 7f720004a700 1 --2- 172.21.15.3:0/1385735575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71f806a960 0x7f71f815d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.824+0000 7f720004a700 1 -- 172.21.15.3:0/1385735575 >> 172.21.15.3:0/1385735575 conn(0x7f71f80d0320 msgr2=0x7f71f80d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.824+0000 7f720004a700 1 -- 172.21.15.3:0/1385735575 shutdown_connections 2024-05-08T21:22:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.824+0000 7f720004a700 1 -- 172.21.15.3:0/1385735575 wait complete. 2024-05-08T21:22:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.826+0000 7f720004a700 1 Processor -- start 2024-05-08T21:22:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.826+0000 7f720004a700 1 -- start start 2024-05-08T21:22:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f720004a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71f806a960 0x7f71f814a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f720004a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71f80cf840 0x7f71f80c4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f720004a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71f80d0eb0 0x7f71f80c63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f720004a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f80c60c0 con 0x7f71f806a960 2024-05-08T21:22:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f720004a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f814ac30 con 0x7f71f80cf840 2024-05-08T21:22:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f720004a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f80c5310 con 0x7f71f80d0eb0 2024-05-08T21:22:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f71fdde6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71f806a960 0x7f71f814a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f71fdde6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71f806a960 0x7f71f814a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51476/0 (socket says 172.21.15.3:51476) 2024-05-08T21:22:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f71fdde6700 1 -- 172.21.15.3:0/23577980 learned_addr learned my addr 172.21.15.3:0/23577980 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f71fd5e5700 1 --2- 172.21.15.3:0/23577980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71f80cf840 0x7f71f80c4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f71fe5e7700 1 --2- 172.21.15.3:0/23577980 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71f80d0eb0 0x7f71f80c63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f71fdde6700 1 -- 172.21.15.3:0/23577980 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71f80d0eb0 msgr2=0x7f71f80c63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f71fdde6700 1 --2- 172.21.15.3:0/23577980 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71f80d0eb0 0x7f71f80c63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f71fdde6700 1 -- 172.21.15.3:0/23577980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71f80cf840 msgr2=0x7f71f80c4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f71fdde6700 1 --2- 172.21.15.3:0/23577980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71f80cf840 0x7f71f80c4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f71fdde6700 1 -- 172.21.15.3:0/23577980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f4009550 con 0x7f71f806a960 2024-05-08T21:22:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f71fdde6700 1 --2- 172.21.15.3:0/23577980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71f806a960 0x7f71f814a6f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f71f80704d0 tx=0x7f71f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.827+0000 7f71ee7fc700 1 -- 172.21.15.3:0/23577980 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f400bb60 con 0x7f71f806a960 2024-05-08T21:22:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.828+0000 7f71ee7fc700 1 -- 172.21.15.3:0/23577980 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f401f070 con 0x7f71f806a960 2024-05-08T21:22:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.828+0000 7f720004a700 1 -- 172.21.15.3:0/23577980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f80c5540 con 0x7f71f806a960 2024-05-08T21:22:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.828+0000 7f71ee7fc700 1 -- 172.21.15.3:0/23577980 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f40036a0 con 0x7f71f806a960 2024-05-08T21:22:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.828+0000 7f720004a700 1 -- 172.21.15.3:0/23577980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f80c5b00 con 0x7f71f806a960 2024-05-08T21:22:37.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.829+0000 7f720004a700 1 -- 172.21.15.3:0/23577980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f8068000 con 0x7f71f806a960 2024-05-08T21:22:37.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.829+0000 7f71ee7fc700 1 -- 172.21.15.3:0/23577980 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f71f4021030 con 0x7f71f806a960 2024-05-08T21:22:37.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.830+0000 7f71ee7fc700 1 --2- 172.21.15.3:0/23577980 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f71dc06ac90 0x7f71dc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:37.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.830+0000 7f71ee7fc700 1 -- 172.21.15.3:0/23577980 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f71f408a980 con 0x7f71f806a960 2024-05-08T21:22:37.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.830+0000 7f71fd5e5700 1 --2- 172.21.15.3:0/23577980 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f71dc06ac90 0x7f71dc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:37.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.830+0000 7f71fd5e5700 1 --2- 172.21.15.3:0/23577980 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f71dc06ac90 0x7f71dc06d150 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f71f80b2380 tx=0x7f71e8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:37.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:37.835+0000 7f71ee7fc700 1 -- 172.21.15.3:0/23577980 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f71f4055b70 con 0x7f71f806a960 2024-05-08T21:22:38.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.051+0000 7f720004a700 1 -- 172.21.15.3:0/23577980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71f8068000 con 0x7f71f806a960 2024-05-08T21:22:38.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.052+0000 7f71ee7fc700 1 -- 172.21.15.3:0/23577980 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16179 (secure 0 0 0) 0x7f71f4059190 con 0x7f71f806a960 2024-05-08T21:22:38.054 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:38.054 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":22,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:22:35.652541+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:21:49.638998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-08T21:21:51.024632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:38.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.055+0000 7f720004a700 1 -- 172.21.15.3:0/23577980 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f71dc06ac90 msgr2=0x7f71dc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.055+0000 7f720004a700 1 --2- 172.21.15.3:0/23577980 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f71dc06ac90 0x7f71dc06d150 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f71f80b2380 tx=0x7f71e8008040 comp rx=0 tx=0).stop 2024-05-08T21:22:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.055+0000 7f720004a700 1 -- 172.21.15.3:0/23577980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71f806a960 msgr2=0x7f71f814a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.055+0000 7f720004a700 1 --2- 172.21.15.3:0/23577980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71f806a960 0x7f71f814a6f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f71f80704d0 tx=0x7f71f4004750 comp rx=0 tx=0).stop 2024-05-08T21:22:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.055+0000 7f720004a700 1 -- 172.21.15.3:0/23577980 shutdown_connections 2024-05-08T21:22:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.055+0000 7f720004a700 1 --2- 172.21.15.3:0/23577980 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71f80d0eb0 0x7f71f80c63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.055+0000 7f720004a700 1 --2- 172.21.15.3:0/23577980 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f71dc06ac90 0x7f71dc06d150 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.055+0000 7f720004a700 1 --2- 172.21.15.3:0/23577980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71f806a960 0x7f71f814a6f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.055+0000 7f720004a700 1 --2- 172.21.15.3:0/23577980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71f80cf840 0x7f71f80c4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.055+0000 7f720004a700 1 -- 172.21.15.3:0/23577980 >> 172.21.15.3:0/23577980 conn(0x7f71f80d0320 msgr2=0x7f71f806f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.055+0000 7f720004a700 1 -- 172.21.15.3:0/23577980 shutdown_connections 2024-05-08T21:22:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.055+0000 7f720004a700 1 -- 172.21.15.3:0/23577980 wait complete. 2024-05-08T21:22:38.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-08T21:22:38.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.205+0000 7f4eeea00700 1 Processor -- start 2024-05-08T21:22:38.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.205+0000 7f4eeea00700 1 -- start start 2024-05-08T21:22:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.206+0000 7f4eeea00700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4ee815b8a0 0x7f4ee815fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.206+0000 7f4eeea00700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4ee80cfcf0 0x7f4ee806b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.206+0000 7f4eeea00700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee815f9f0 con 0x7f4ee806f530 2024-05-08T21:22:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.206+0000 7f4eeea00700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ee815bc80 con 0x7f4ee815b8a0 2024-05-08T21:22:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.206+0000 7f4eeea00700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ee806b880 con 0x7f4ee80cfcf0 2024-05-08T21:22:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.206+0000 7f4ee7fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4ee815b8a0 0x7f4ee815fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.206+0000 7f4ee77fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f4ee806f530 0x7f4ee80cf5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:48068/0 (socket says 172.21.15.3:48068) 2024-05-08T21:22:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.206+0000 7f4ee7fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4ee815b8a0 0x7f4ee815fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51490/0 (socket says 172.21.15.3:51490) 2024-05-08T21:22:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.206+0000 7f4ee77fe700 1 -- 172.21.15.3:0/3563204979 learned_addr learned my addr 172.21.15.3:0/3563204979 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.207+0000 7f4eecf9d700 1 --2- 172.21.15.3:0/3563204979 >> v2:172.21.15.186:3300/0 conn(0x7f4ee80cfcf0 0x7f4ee806b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.207+0000 7f4ee7fff700 1 -- 172.21.15.3:0/3563204979 >> v2:172.21.15.186:3300/0 conn(0x7f4ee80cfcf0 msgr2=0x7f4ee806b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.207+0000 7f4ee7fff700 1 --2- 172.21.15.3:0/3563204979 >> v2:172.21.15.186:3300/0 conn(0x7f4ee80cfcf0 0x7f4ee806b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.207+0000 7f4ee7fff700 1 -- 172.21.15.3:0/3563204979 >> v1:172.21.15.3:6789/0 conn(0x7f4ee806f530 legacy=0x7f4ee80cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.207+0000 7f4ee7fff700 1 -- 172.21.15.3:0/3563204979 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee806f9c0 con 0x7f4ee815b8a0 2024-05-08T21:22:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.207+0000 7f4ee6ffd700 1 -- 172.21.15.3:0/3563204979 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836179097 0 0) 0x7f4ee815f9f0 con 0x7f4ee806f530 2024-05-08T21:22:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.207+0000 7f4eecf9d700 1 --2- 172.21.15.3:0/3563204979 >> v2:172.21.15.186:3300/0 conn(0x7f4ee80cfcf0 0x7f4ee806b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.207+0000 7f4ee7fff700 1 --2- 172.21.15.3:0/3563204979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ee815b8a0 0x7f4ee815fe80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4ee815f960 tx=0x7f4ed8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aedd5a0fd128768 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.207+0000 7f4ee6ffd700 1 -- 172.21.15.3:0/3563204979 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8011830 con 0x7f4ee815b8a0 2024-05-08T21:22:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.207+0000 7f4ee6ffd700 1 -- 172.21.15.3:0/3563204979 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed8011990 con 0x7f4ee815b8a0 2024-05-08T21:22:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.207+0000 7f4ee6ffd700 1 -- 172.21.15.3:0/3563204979 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8011d20 con 0x7f4ee815b8a0 2024-05-08T21:22:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.208+0000 7f4eeea00700 1 -- 172.21.15.3:0/3563204979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ee815b8a0 msgr2=0x7f4ee815fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.208+0000 7f4eeea00700 1 --2- 172.21.15.3:0/3563204979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ee815b8a0 0x7f4ee815fe80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4ee815f960 tx=0x7f4ed8009b20 comp rx=0 tx=0).stop 2024-05-08T21:22:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.208+0000 7f4eeea00700 1 -- 172.21.15.3:0/3563204979 shutdown_connections 2024-05-08T21:22:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.208+0000 7f4eeea00700 1 --2- 172.21.15.3:0/3563204979 >> v2:172.21.15.186:3300/0 conn(0x7f4ee80cfcf0 0x7f4ee806b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.208+0000 7f4eeea00700 1 --2- 172.21.15.3:0/3563204979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ee815b8a0 0x7f4ee815fe80 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.208+0000 7f4eeea00700 1 -- 172.21.15.3:0/3563204979 >> 172.21.15.3:0/3563204979 conn(0x7f4ee80b1f80 msgr2=0x7f4ee80b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.208+0000 7f4eeea00700 1 -- 172.21.15.3:0/3563204979 shutdown_connections 2024-05-08T21:22:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.208+0000 7f4eeea00700 1 -- 172.21.15.3:0/3563204979 wait complete. 2024-05-08T21:22:38.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.210+0000 7f4eeea00700 1 Processor -- start 2024-05-08T21:22:38.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.210+0000 7f4eeea00700 1 -- start start 2024-05-08T21:22:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4eeea00700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ee80cfcf0 0x7f4ee814c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4eeea00700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ee815b8a0 0x7f4ee8148dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4eeea00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ee814bd60 0x7f4ee8159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4eeea00700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee814c780 con 0x7f4ee815b8a0 2024-05-08T21:22:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4eeea00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee815d0e0 con 0x7f4ee814bd60 2024-05-08T21:22:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4eeea00700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee80b5870 con 0x7f4ee80cfcf0 2024-05-08T21:22:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4ee77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ee815b8a0 0x7f4ee8148dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4ee77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ee815b8a0 0x7f4ee8148dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51496/0 (socket says 172.21.15.3:51496) 2024-05-08T21:22:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4ee77fe700 1 -- 172.21.15.3:0/289216296 learned_addr learned my addr 172.21.15.3:0/289216296 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4eecf9d700 1 --2- 172.21.15.3:0/289216296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ee814bd60 0x7f4ee8159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4ee7fff700 1 --2- 172.21.15.3:0/289216296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ee80cfcf0 0x7f4ee814c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4ee77fe700 1 -- 172.21.15.3:0/289216296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ee80cfcf0 msgr2=0x7f4ee814c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4ee77fe700 1 --2- 172.21.15.3:0/289216296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ee80cfcf0 0x7f4ee814c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4ee77fe700 1 -- 172.21.15.3:0/289216296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ee814bd60 msgr2=0x7f4ee8159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4ee77fe700 1 --2- 172.21.15.3:0/289216296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ee814bd60 0x7f4ee8159850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4ee77fe700 1 -- 172.21.15.3:0/289216296 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed80094a0 con 0x7f4ee815b8a0 2024-05-08T21:22:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.211+0000 7f4ee7fff700 1 --2- 172.21.15.3:0/289216296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ee80cfcf0 0x7f4ee814c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.212+0000 7f4ee77fe700 1 --2- 172.21.15.3:0/289216296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ee815b8a0 0x7f4ee8148dd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4ed000c500 tx=0x7f4ed000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.212+0000 7f4ee4ff9700 1 -- 172.21.15.3:0/289216296 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0004c80 con 0x7f4ee815b8a0 2024-05-08T21:22:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.212+0000 7f4ee4ff9700 1 -- 172.21.15.3:0/289216296 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed0004e00 con 0x7f4ee815b8a0 2024-05-08T21:22:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.212+0000 7f4ee4ff9700 1 -- 172.21.15.3:0/289216296 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0007310 con 0x7f4ee815b8a0 2024-05-08T21:22:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.212+0000 7f4eeea00700 1 -- 172.21.15.3:0/289216296 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee8149310 con 0x7f4ee815b8a0 2024-05-08T21:22:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.212+0000 7f4eeea00700 1 -- 172.21.15.3:0/289216296 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee80b5dc0 con 0x7f4ee815b8a0 2024-05-08T21:22:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.213+0000 7f4eeea00700 1 -- 172.21.15.3:0/289216296 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee8068000 con 0x7f4ee815b8a0 2024-05-08T21:22:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.213+0000 7f4ee4ff9700 1 -- 172.21.15.3:0/289216296 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4ed0007470 con 0x7f4ee815b8a0 2024-05-08T21:22:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.214+0000 7f4ee4ff9700 1 --2- 172.21.15.3:0/289216296 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4ec806ac90 0x7f4ec806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.214+0000 7f4ee4ff9700 1 -- 172.21.15.3:0/289216296 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4ed0088fa0 con 0x7f4ee815b8a0 2024-05-08T21:22:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.214+0000 7f4ee7fff700 1 --2- 172.21.15.3:0/289216296 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4ec806ac90 0x7f4ec806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:38.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.215+0000 7f4ee7fff700 1 --2- 172.21.15.3:0/289216296 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4ec806ac90 0x7f4ec806d150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80090b0 tx=0x7f4ed801c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:38.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.219+0000 7f4ee4ff9700 1 -- 172.21.15.3:0/289216296 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f4ed00bc900 con 0x7f4ee815b8a0 2024-05-08T21:22:38.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.489+0000 7f4eeea00700 1 -- 172.21.15.3:0/289216296 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4ee80b19d0 con 0x7f4ee815b8a0 2024-05-08T21:22:38.492 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-08T21:22:38.490+0000 7f74f7652700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-08T21:22:38.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.500+0000 7f4ee4ff9700 1 -- 172.21.15.3:0/289216296 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4ed0054190 con 0x7f4ee815b8a0 2024-05-08T21:22:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.502+0000 7f4eeea00700 1 -- 172.21.15.3:0/289216296 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4ec806ac90 msgr2=0x7f4ec806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.502+0000 7f4eeea00700 1 --2- 172.21.15.3:0/289216296 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4ec806ac90 0x7f4ec806d150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80090b0 tx=0x7f4ed801c040 comp rx=0 tx=0).stop 2024-05-08T21:22:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.502+0000 7f4eeea00700 1 -- 172.21.15.3:0/289216296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ee815b8a0 msgr2=0x7f4ee8148dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.502+0000 7f4eeea00700 1 --2- 172.21.15.3:0/289216296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ee815b8a0 0x7f4ee8148dd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4ed000c500 tx=0x7f4ed000c810 comp rx=0 tx=0).stop 2024-05-08T21:22:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.502+0000 7f4eeea00700 1 -- 172.21.15.3:0/289216296 shutdown_connections 2024-05-08T21:22:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.502+0000 7f4eeea00700 1 --2- 172.21.15.3:0/289216296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ee80cfcf0 0x7f4ee814c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.502+0000 7f4eeea00700 1 --2- 172.21.15.3:0/289216296 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4ec806ac90 0x7f4ec806d150 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.502+0000 7f4eeea00700 1 --2- 172.21.15.3:0/289216296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ee814bd60 0x7f4ee8159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.502+0000 7f4eeea00700 1 --2- 172.21.15.3:0/289216296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ee815b8a0 0x7f4ee8148dd0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.502+0000 7f4eeea00700 1 -- 172.21.15.3:0/289216296 >> 172.21.15.3:0/289216296 conn(0x7f4ee80b1f80 msgr2=0x7f4ee80b0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.502+0000 7f4eeea00700 1 -- 172.21.15.3:0/289216296 shutdown_connections 2024-05-08T21:22:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.502+0000 7f4eeea00700 1 -- 172.21.15.3:0/289216296 wait complete. 2024-05-08T21:22:38.505 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-08T21:22:38.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-08T21:22:38.544 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2024-05-08T21:22:38.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.657+0000 7fad9c1f8700 1 Processor -- start 2024-05-08T21:22:38.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.657+0000 7fad9c1f8700 1 -- start start 2024-05-08T21:22:38.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.658+0000 7fad9c1f8700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad940aea70 0x7fad940b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:38.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.658+0000 7fad9c1f8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fad940d0a60 0x7fad940b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:38.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.658+0000 7fad9c1f8700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad9415add0 con 0x7fad94154df0 2024-05-08T21:22:38.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.658+0000 7fad9c1f8700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad9415abe0 con 0x7fad940d0a60 2024-05-08T21:22:38.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.658+0000 7fad9c1f8700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad9415b070 con 0x7fad940aea70 2024-05-08T21:22:38.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.659+0000 7fad9a795700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fad940d0a60 0x7fad940b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:38.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.659+0000 7fad9a795700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fad940d0a60 0x7fad940b43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51502/0 (socket says 172.21.15.3:51502) 2024-05-08T21:22:38.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.659+0000 7fad9a795700 1 -- 172.21.15.3:0/1932854140 learned_addr learned my addr 172.21.15.3:0/1932854140 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:38.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.659+0000 7fad99793700 1 --2- 172.21.15.3:0/1932854140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad940aea70 0x7fad940b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:38.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.659+0000 7fad9a795700 1 -- 172.21.15.3:0/1932854140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad940aea70 msgr2=0x7fad940b3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.659+0000 7fad9a795700 1 --2- 172.21.15.3:0/1932854140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad940aea70 0x7fad940b3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.659+0000 7fad9a795700 1 -- 172.21.15.3:0/1932854140 >> v1:172.21.15.186:6789/0 conn(0x7fad94154df0 legacy=0x7fad940d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.659+0000 7fad9a795700 1 -- 172.21.15.3:0/1932854140 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9415b350 con 0x7fad940d0a60 2024-05-08T21:22:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.659+0000 7fad9a795700 1 --2- 172.21.15.3:0/1932854140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad940d0a60 0x7fad940b43b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009e50 tx=0x7fad8c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb380ab62bb1fe05 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.659+0000 7fad98f92700 1 -- 172.21.15.3:0/1932854140 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c00e070 con 0x7fad940d0a60 2024-05-08T21:22:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.659+0000 7fad98f92700 1 -- 172.21.15.3:0/1932854140 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8c00b910 con 0x7fad940d0a60 2024-05-08T21:22:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.659+0000 7fad98f92700 1 -- 172.21.15.3:0/1932854140 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c00bcc0 con 0x7fad940d0a60 2024-05-08T21:22:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.660+0000 7fad9c1f8700 1 -- 172.21.15.3:0/1932854140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad940d0a60 msgr2=0x7fad940b43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.660+0000 7fad9c1f8700 1 --2- 172.21.15.3:0/1932854140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad940d0a60 0x7fad940b43b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009e50 tx=0x7fad8c00b330 comp rx=0 tx=0).stop 2024-05-08T21:22:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.660+0000 7fad9c1f8700 1 -- 172.21.15.3:0/1932854140 shutdown_connections 2024-05-08T21:22:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.660+0000 7fad9c1f8700 1 --2- 172.21.15.3:0/1932854140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad940d0a60 0x7fad940b43b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.660+0000 7fad9c1f8700 1 --2- 172.21.15.3:0/1932854140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad940aea70 0x7fad940b3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.660+0000 7fad9c1f8700 1 -- 172.21.15.3:0/1932854140 >> 172.21.15.3:0/1932854140 conn(0x7fad94099a50 msgr2=0x7fad940b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.660+0000 7fad9c1f8700 1 -- 172.21.15.3:0/1932854140 shutdown_connections 2024-05-08T21:22:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.660+0000 7fad9c1f8700 1 -- 172.21.15.3:0/1932854140 wait complete. 2024-05-08T21:22:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.661+0000 7fad9c1f8700 1 Processor -- start 2024-05-08T21:22:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.661+0000 7fad9c1f8700 1 -- start start 2024-05-08T21:22:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.662+0000 7fad9c1f8700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad940aea70 0x7fad940cd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.662+0000 7fad9c1f8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad940d0a60 0x7fad9414c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.662+0000 7fad9c1f8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad94154df0 0x7fad941595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.662+0000 7fad9c1f8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9414d500 con 0x7fad94154df0 2024-05-08T21:22:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.662+0000 7fad9c1f8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad940cd650 con 0x7fad940d0a60 2024-05-08T21:22:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.662+0000 7fad9c1f8700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad940cfb80 con 0x7fad940aea70 2024-05-08T21:22:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.662+0000 7fad9a795700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad94154df0 0x7fad941595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.662+0000 7fad9a795700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad94154df0 0x7fad941595e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51510/0 (socket says 172.21.15.3:51510) 2024-05-08T21:22:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.662+0000 7fad9a795700 1 -- 172.21.15.3:0/1502150252 learned_addr learned my addr 172.21.15.3:0/1502150252 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.662+0000 7fad99f94700 1 --2- 172.21.15.3:0/1502150252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad940aea70 0x7fad940cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.662+0000 7fad99793700 1 --2- 172.21.15.3:0/1502150252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad940d0a60 0x7fad9414c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.663+0000 7fad9a795700 1 -- 172.21.15.3:0/1502150252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad940aea70 msgr2=0x7fad940cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.663+0000 7fad9a795700 1 --2- 172.21.15.3:0/1502150252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad940aea70 0x7fad940cd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.663+0000 7fad9a795700 1 -- 172.21.15.3:0/1502150252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad940d0a60 msgr2=0x7fad9414c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.663+0000 7fad9a795700 1 --2- 172.21.15.3:0/1502150252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad940d0a60 0x7fad9414c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.663+0000 7fad9a795700 1 -- 172.21.15.3:0/1502150252 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8c009b30 con 0x7fad94154df0 2024-05-08T21:22:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.663+0000 7fad9a795700 1 --2- 172.21.15.3:0/1502150252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad94154df0 0x7fad941595e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fad8c012000 tx=0x7fad8c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.663+0000 7fad867fc700 1 -- 172.21.15.3:0/1502150252 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c004960 con 0x7fad94154df0 2024-05-08T21:22:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.663+0000 7fad867fc700 1 -- 172.21.15.3:0/1502150252 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8c0099b0 con 0x7fad94154df0 2024-05-08T21:22:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.663+0000 7fad9c1f8700 1 -- 172.21.15.3:0/1502150252 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad940cfe60 con 0x7fad94154df0 2024-05-08T21:22:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.663+0000 7fad867fc700 1 -- 172.21.15.3:0/1502150252 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c01c680 con 0x7fad94154df0 2024-05-08T21:22:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.663+0000 7fad9c1f8700 1 -- 172.21.15.3:0/1502150252 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad94152310 con 0x7fad94154df0 2024-05-08T21:22:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.664+0000 7fad9c1f8700 1 -- 172.21.15.3:0/1502150252 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad94074bc0 con 0x7fad94154df0 2024-05-08T21:22:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.665+0000 7fad867fc700 1 -- 172.21.15.3:0/1502150252 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fad8c01c7e0 con 0x7fad94154df0 2024-05-08T21:22:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.665+0000 7fad867fc700 1 --2- 172.21.15.3:0/1502150252 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad7806ac90 0x7fad7806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.665+0000 7fad867fc700 1 -- 172.21.15.3:0/1502150252 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fad8c08b0a0 con 0x7fad94154df0 2024-05-08T21:22:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.665+0000 7fad99f94700 1 --2- 172.21.15.3:0/1502150252 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad7806ac90 0x7fad7806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:38.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.666+0000 7fad99f94700 1 --2- 172.21.15.3:0/1502150252 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad7806ac90 0x7fad7806d150 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fad940b04a0 tx=0x7fad90009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:38.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.670+0000 7fad867fc700 1 -- 172.21.15.3:0/1502150252 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fad8c056240 con 0x7fad94154df0 2024-05-08T21:22:38.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:38.936+0000 7fad9c1f8700 1 -- 172.21.15.3:0/1502150252 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fad9414ca20 con 0x7fad94154df0 2024-05-08T21:22:39.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.505+0000 7fad867fc700 1 -- 172.21.15.3:0/1502150252 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7fad8c059860 con 0x7fad94154df0 2024-05-08T21:22:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.508+0000 7fad9c1f8700 1 -- 172.21.15.3:0/1502150252 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad7806ac90 msgr2=0x7fad7806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.508+0000 7fad9c1f8700 1 --2- 172.21.15.3:0/1502150252 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad7806ac90 0x7fad7806d150 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fad940b04a0 tx=0x7fad90009040 comp rx=0 tx=0).stop 2024-05-08T21:22:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.508+0000 7fad9c1f8700 1 -- 172.21.15.3:0/1502150252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad94154df0 msgr2=0x7fad941595e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.508+0000 7fad9c1f8700 1 --2- 172.21.15.3:0/1502150252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad94154df0 0x7fad941595e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fad8c012000 tx=0x7fad8c004ad0 comp rx=0 tx=0).stop 2024-05-08T21:22:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.508+0000 7fad9c1f8700 1 -- 172.21.15.3:0/1502150252 shutdown_connections 2024-05-08T21:22:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.508+0000 7fad9c1f8700 1 --2- 172.21.15.3:0/1502150252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad940aea70 0x7fad940cd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.508+0000 7fad9c1f8700 1 --2- 172.21.15.3:0/1502150252 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fad7806ac90 0x7fad7806d150 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.508+0000 7fad9c1f8700 1 --2- 172.21.15.3:0/1502150252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad94154df0 0x7fad941595e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.508+0000 7fad9c1f8700 1 --2- 172.21.15.3:0/1502150252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad940d0a60 0x7fad9414c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.508+0000 7fad9c1f8700 1 -- 172.21.15.3:0/1502150252 >> 172.21.15.3:0/1502150252 conn(0x7fad94099a50 msgr2=0x7fad940adeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.509+0000 7fad9c1f8700 1 -- 172.21.15.3:0/1502150252 shutdown_connections 2024-05-08T21:22:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.509+0000 7fad9c1f8700 1 -- 172.21.15.3:0/1502150252 wait complete. 2024-05-08T21:22:39.526 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-08T21:22:39.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.669+0000 7fd1c7f57700 1 Processor -- start 2024-05-08T21:22:39.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.669+0000 7fd1c7f57700 1 -- start start 2024-05-08T21:22:39.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.670+0000 7fd1c7f57700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd1c015d8c0 0x7fd1c00ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:39.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.670+0000 7fd1c7f57700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1c00b50d0 0x7fd1c006a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:39.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.670+0000 7fd1c7f57700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c0099ce0 con 0x7fd1c00b5550 2024-05-08T21:22:39.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.670+0000 7fd1c7f57700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1c006fde0 con 0x7fd1c015d8c0 2024-05-08T21:22:39.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.670+0000 7fd1c7f57700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c006f260 con 0x7fd1c00b50d0 2024-05-08T21:22:39.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.670+0000 7fd1c5cf3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd1c015d8c0 0x7fd1c00ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:39.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.670+0000 7fd1c5cf3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd1c015d8c0 0x7fd1c00ad260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51518/0 (socket says 172.21.15.3:51518) 2024-05-08T21:22:39.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.670+0000 7fd1c5cf3700 1 -- 172.21.15.3:0/3863656826 learned_addr learned my addr 172.21.15.3:0/3863656826 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:39.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.670+0000 7fd1c54f2700 1 --2- 172.21.15.3:0/3863656826 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1c00b50d0 0x7fd1c006a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:39.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.670+0000 7fd1c5cf3700 1 -- 172.21.15.3:0/3863656826 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1c00b50d0 msgr2=0x7fd1c006a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:39.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.670+0000 7fd1c5cf3700 1 --2- 172.21.15.3:0/3863656826 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1c00b50d0 0x7fd1c006a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:39.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.670+0000 7fd1c5cf3700 1 -- 172.21.15.3:0/3863656826 >> v1:172.21.15.186:6789/0 conn(0x7fd1c00b5550 legacy=0x7fd1c01618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:39.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.670+0000 7fd1c5cf3700 1 -- 172.21.15.3:0/3863656826 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c006f540 con 0x7fd1c015d8c0 2024-05-08T21:22:39.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.671+0000 7fd1c5cf3700 1 --2- 172.21.15.3:0/3863656826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1c015d8c0 0x7fd1c00ad260 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0098f0 tx=0x7fd1bc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3735c18e77969a9f server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:39.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.671+0000 7fd1c4cf1700 1 -- 172.21.15.3:0/3863656826 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc00b8a0 con 0x7fd1c015d8c0 2024-05-08T21:22:39.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.671+0000 7fd1c4cf1700 1 -- 172.21.15.3:0/3863656826 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1bc00ba00 con 0x7fd1c015d8c0 2024-05-08T21:22:39.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.671+0000 7fd1c4cf1700 1 -- 172.21.15.3:0/3863656826 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc00bd90 con 0x7fd1c015d8c0 2024-05-08T21:22:39.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.671+0000 7fd1c7f57700 1 -- 172.21.15.3:0/3863656826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1c015d8c0 msgr2=0x7fd1c00ad260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:39.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.671+0000 7fd1c7f57700 1 --2- 172.21.15.3:0/3863656826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1c015d8c0 0x7fd1c00ad260 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0098f0 tx=0x7fd1bc009c00 comp rx=0 tx=0).stop 2024-05-08T21:22:39.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.671+0000 7fd1c7f57700 1 -- 172.21.15.3:0/3863656826 shutdown_connections 2024-05-08T21:22:39.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.671+0000 7fd1c7f57700 1 --2- 172.21.15.3:0/3863656826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1c015d8c0 0x7fd1c00ad260 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:39.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.671+0000 7fd1c7f57700 1 --2- 172.21.15.3:0/3863656826 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1c00b50d0 0x7fd1c006a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:39.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.671+0000 7fd1c7f57700 1 -- 172.21.15.3:0/3863656826 >> 172.21.15.3:0/3863656826 conn(0x7fd1c00b5a30 msgr2=0x7fd1c00b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:39.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.671+0000 7fd1c7f57700 1 -- 172.21.15.3:0/3863656826 shutdown_connections 2024-05-08T21:22:39.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.671+0000 7fd1c7f57700 1 -- 172.21.15.3:0/3863656826 wait complete. 2024-05-08T21:22:39.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.673+0000 7fd1c7f57700 1 Processor -- start 2024-05-08T21:22:39.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.673+0000 7fd1c7f57700 1 -- start start 2024-05-08T21:22:39.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.673+0000 7fd1c7f57700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1c00b50d0 0x7fd1c0152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:39.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1c7f57700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1c00b5550 0x7fd1c0156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:39.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1c7f57700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1c015d8c0 0x7fd1c01506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:39.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1c7f57700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c014d110 con 0x7fd1c00b5550 2024-05-08T21:22:39.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1c7f57700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c014c500 con 0x7fd1c00b50d0 2024-05-08T21:22:39.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1c7f57700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c014c7e0 con 0x7fd1c015d8c0 2024-05-08T21:22:39.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1c54f2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1c00b5550 0x7fd1c0156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:39.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1c54f2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1c00b5550 0x7fd1c0156040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51524/0 (socket says 172.21.15.3:51524) 2024-05-08T21:22:39.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1c54f2700 1 -- 172.21.15.3:0/1647806133 learned_addr learned my addr 172.21.15.3:0/1647806133 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:39.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1c5cf3700 1 --2- 172.21.15.3:0/1647806133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1c00b50d0 0x7fd1c0152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:39.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1c64f4700 1 --2- 172.21.15.3:0/1647806133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1c015d8c0 0x7fd1c01506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:39.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1c54f2700 1 -- 172.21.15.3:0/1647806133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1c015d8c0 msgr2=0x7fd1c01506a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:39.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1c54f2700 1 --2- 172.21.15.3:0/1647806133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1c015d8c0 0x7fd1c01506a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:39.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1c54f2700 1 -- 172.21.15.3:0/1647806133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1c00b50d0 msgr2=0x7fd1c0152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:39.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1c54f2700 1 --2- 172.21.15.3:0/1647806133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1c00b50d0 0x7fd1c0152c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:39.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1c54f2700 1 -- 172.21.15.3:0/1647806133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc009550 con 0x7fd1c00b5550 2024-05-08T21:22:39.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1c54f2700 1 --2- 172.21.15.3:0/1647806133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1c00b5550 0x7fd1c0156040 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000e810 tx=0x7fd1b000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:39.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1b67fc700 1 -- 172.21.15.3:0/1647806133 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b000c070 con 0x7fd1c00b5550 2024-05-08T21:22:39.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1b67fc700 1 -- 172.21.15.3:0/1647806133 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b0013610 con 0x7fd1c00b5550 2024-05-08T21:22:39.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.674+0000 7fd1b67fc700 1 -- 172.21.15.3:0/1647806133 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b00139c0 con 0x7fd1c00b5550 2024-05-08T21:22:39.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.675+0000 7fd1c7f57700 1 -- 172.21.15.3:0/1647806133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c014cac0 con 0x7fd1c00b5550 2024-05-08T21:22:39.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.675+0000 7fd1c7f57700 1 -- 172.21.15.3:0/1647806133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c0156580 con 0x7fd1c00b5550 2024-05-08T21:22:39.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.676+0000 7fd1c7f57700 1 -- 172.21.15.3:0/1647806133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c014d2a0 con 0x7fd1c00b5550 2024-05-08T21:22:39.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.676+0000 7fd1b67fc700 1 -- 172.21.15.3:0/1647806133 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd1b0013b20 con 0x7fd1c00b5550 2024-05-08T21:22:39.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.677+0000 7fd1b67fc700 1 --2- 172.21.15.3:0/1647806133 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1a406ac90 0x7fd1a406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:39.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.677+0000 7fd1b67fc700 1 -- 172.21.15.3:0/1647806133 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fd1b0089180 con 0x7fd1c00b5550 2024-05-08T21:22:39.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.677+0000 7fd1c5cf3700 1 --2- 172.21.15.3:0/1647806133 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1a406ac90 0x7fd1a406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:39.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.678+0000 7fd1c5cf3700 1 --2- 172.21.15.3:0/1647806133 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1a406ac90 0x7fd1a406d150 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc014040 tx=0x7fd1bc008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:39.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.683+0000 7fd1b67fc700 1 -- 172.21.15.3:0/1647806133 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd1b00542a0 con 0x7fd1c00b5550 2024-05-08T21:22:39.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:39.900+0000 7fd1c7f57700 1 -- 172.21.15.3:0/1647806133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd1c0074bc0 con 0x7fd1c00b5550 2024-05-08T21:22:40.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.685+0000 7fd1b67fc700 1 -- 172.21.15.3:0/1647806133 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7fd1b00578c0 con 0x7fd1c00b5550 2024-05-08T21:22:40.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.689+0000 7fd1c7f57700 1 -- 172.21.15.3:0/1647806133 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1a406ac90 msgr2=0x7fd1a406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:40.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.689+0000 7fd1c7f57700 1 --2- 172.21.15.3:0/1647806133 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1a406ac90 0x7fd1a406d150 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc014040 tx=0x7fd1bc008030 comp rx=0 tx=0).stop 2024-05-08T21:22:40.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.689+0000 7fd1c7f57700 1 -- 172.21.15.3:0/1647806133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1c00b5550 msgr2=0x7fd1c0156040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:40.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.689+0000 7fd1c7f57700 1 --2- 172.21.15.3:0/1647806133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1c00b5550 0x7fd1c0156040 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000e810 tx=0x7fd1b000eb20 comp rx=0 tx=0).stop 2024-05-08T21:22:40.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.689+0000 7fd1c7f57700 1 -- 172.21.15.3:0/1647806133 shutdown_connections 2024-05-08T21:22:40.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.689+0000 7fd1c7f57700 1 --2- 172.21.15.3:0/1647806133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1c015d8c0 0x7fd1c01506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:40.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.689+0000 7fd1c7f57700 1 --2- 172.21.15.3:0/1647806133 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1a406ac90 0x7fd1a406d150 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:40.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.689+0000 7fd1c7f57700 1 --2- 172.21.15.3:0/1647806133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1c00b50d0 0x7fd1c0152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:40.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.689+0000 7fd1c7f57700 1 --2- 172.21.15.3:0/1647806133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1c00b5550 0x7fd1c0156040 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:40.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.689+0000 7fd1c7f57700 1 -- 172.21.15.3:0/1647806133 >> 172.21.15.3:0/1647806133 conn(0x7fd1c00b5a30 msgr2=0x7fd1c0160230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:40.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.689+0000 7fd1c7f57700 1 -- 172.21.15.3:0/1647806133 shutdown_connections 2024-05-08T21:22:40.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.689+0000 7fd1c7f57700 1 -- 172.21.15.3:0/1647806133 wait complete. 2024-05-08T21:22:40.691 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2024-05-08T21:22:40.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-08T21:22:40.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.850+0000 7f230da74700 1 Processor -- start 2024-05-08T21:22:40.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.850+0000 7f230da74700 1 -- start start 2024-05-08T21:22:40.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.851+0000 7f230da74700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f230815b710 0x7f2308099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:40.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.851+0000 7f230da74700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f23080cff30 0x7f2308099f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:40.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.851+0000 7f230da74700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23080d04b0 con 0x7f23080cf040 2024-05-08T21:22:40.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.851+0000 7f230da74700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23080b0910 con 0x7f230815b710 2024-05-08T21:22:40.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.851+0000 7f230da74700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23080b0bf0 con 0x7f23080cff30 2024-05-08T21:22:40.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.851+0000 7f23077fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f230815b710 0x7f2308099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.851+0000 7f23077fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f230815b710 0x7f2308099a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51536/0 (socket says 172.21.15.3:51536) 2024-05-08T21:22:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.851+0000 7f23077fe700 1 -- 172.21.15.3:0/2163801524 learned_addr learned my addr 172.21.15.3:0/2163801524 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.851+0000 7f2306ffd700 1 --2- 172.21.15.3:0/2163801524 >> v2:172.21.15.186:3300/0 conn(0x7f23080cff30 0x7f2308099f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.851+0000 7f23077fe700 1 -- 172.21.15.3:0/2163801524 >> v2:172.21.15.186:3300/0 conn(0x7f23080cff30 msgr2=0x7f2308099f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.851+0000 7f23077fe700 1 --2- 172.21.15.3:0/2163801524 >> v2:172.21.15.186:3300/0 conn(0x7f23080cff30 0x7f2308099f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.851+0000 7f23077fe700 1 -- 172.21.15.3:0/2163801524 >> v1:172.21.15.186:6789/0 conn(0x7f23080cf040 legacy=0x7f23080cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.851+0000 7f23077fe700 1 -- 172.21.15.3:0/2163801524 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23080b0ed0 con 0x7f230815b710 2024-05-08T21:22:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.851+0000 7f23077fe700 1 --2- 172.21.15.3:0/2163801524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f230815b710 0x7f2308099a50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f22f80098f0 tx=0x7f22f8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7000c0d742e7bfed server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.852+0000 7f23067fc700 1 -- 172.21.15.3:0/2163801524 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f8004230 con 0x7f230815b710 2024-05-08T21:22:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.852+0000 7f23067fc700 1 -- 172.21.15.3:0/2163801524 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f8004390 con 0x7f230815b710 2024-05-08T21:22:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.852+0000 7f23067fc700 1 -- 172.21.15.3:0/2163801524 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f80036a0 con 0x7f230815b710 2024-05-08T21:22:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.852+0000 7f230da74700 1 -- 172.21.15.3:0/2163801524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f230815b710 msgr2=0x7f2308099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.852+0000 7f230da74700 1 --2- 172.21.15.3:0/2163801524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f230815b710 0x7f2308099a50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f22f80098f0 tx=0x7f22f8009c00 comp rx=0 tx=0).stop 2024-05-08T21:22:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.852+0000 7f230da74700 1 -- 172.21.15.3:0/2163801524 shutdown_connections 2024-05-08T21:22:40.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.852+0000 7f230da74700 1 --2- 172.21.15.3:0/2163801524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f230815b710 0x7f2308099a50 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:40.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.852+0000 7f230da74700 1 --2- 172.21.15.3:0/2163801524 >> v2:172.21.15.186:3300/0 conn(0x7f23080cff30 0x7f2308099f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:40.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.852+0000 7f230da74700 1 -- 172.21.15.3:0/2163801524 >> 172.21.15.3:0/2163801524 conn(0x7f23080b2f60 msgr2=0x7f23080b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:40.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.852+0000 7f230da74700 1 -- 172.21.15.3:0/2163801524 shutdown_connections 2024-05-08T21:22:40.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.852+0000 7f230da74700 1 -- 172.21.15.3:0/2163801524 wait complete. 2024-05-08T21:22:40.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.854+0000 7f230da74700 1 Processor -- start 2024-05-08T21:22:40.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.854+0000 7f230da74700 1 -- start start 2024-05-08T21:22:40.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f230da74700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23080cf040 0x7f2308158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:40.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f230da74700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23080cff30 0x7f230809d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:40.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f230da74700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f230815b710 0x7f23080b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:40.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f230da74700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23080b6a60 con 0x7f23080cff30 2024-05-08T21:22:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f230da74700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23080b6780 con 0x7f23080cf040 2024-05-08T21:22:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f230da74700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23080b41c0 con 0x7f230815b710 2024-05-08T21:22:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f2306ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23080cff30 0x7f230809d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f2306ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23080cff30 0x7f230809d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51538/0 (socket says 172.21.15.3:51538) 2024-05-08T21:22:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f2306ffd700 1 -- 172.21.15.3:0/3047035918 learned_addr learned my addr 172.21.15.3:0/3047035918 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f23077fe700 1 --2- 172.21.15.3:0/3047035918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23080cf040 0x7f2308158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f2307fff700 1 --2- 172.21.15.3:0/3047035918 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f230815b710 0x7f23080b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f2306ffd700 1 -- 172.21.15.3:0/3047035918 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f230815b710 msgr2=0x7f23080b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f2306ffd700 1 --2- 172.21.15.3:0/3047035918 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f230815b710 0x7f23080b4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f2306ffd700 1 -- 172.21.15.3:0/3047035918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23080cf040 msgr2=0x7f2308158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f2306ffd700 1 --2- 172.21.15.3:0/3047035918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23080cf040 0x7f2308158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f2306ffd700 1 -- 172.21.15.3:0/3047035918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f8009550 con 0x7f23080cff30 2024-05-08T21:22:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f2306ffd700 1 --2- 172.21.15.3:0/3047035918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23080cff30 0x7f230809d800 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00e7e0 tx=0x7f22fc00eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.855+0000 7f22effff700 1 -- 172.21.15.3:0/3047035918 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc013610 con 0x7f23080cff30 2024-05-08T21:22:40.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.856+0000 7f22effff700 1 -- 172.21.15.3:0/3047035918 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22fc013770 con 0x7f23080cff30 2024-05-08T21:22:40.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.856+0000 7f230da74700 1 -- 172.21.15.3:0/3047035918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23080b44a0 con 0x7f23080cff30 2024-05-08T21:22:40.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.856+0000 7f22effff700 1 -- 172.21.15.3:0/3047035918 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc013ae0 con 0x7f23080cff30 2024-05-08T21:22:40.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.856+0000 7f230da74700 1 -- 172.21.15.3:0/3047035918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23081592c0 con 0x7f23080cff30 2024-05-08T21:22:40.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.857+0000 7f230da74700 1 -- 172.21.15.3:0/3047035918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23080cd310 con 0x7f23080cff30 2024-05-08T21:22:40.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.857+0000 7f22effff700 1 -- 172.21.15.3:0/3047035918 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f22fc013c40 con 0x7f23080cff30 2024-05-08T21:22:40.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.858+0000 7f22effff700 1 --2- 172.21.15.3:0/3047035918 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f22f406ac90 0x7f22f406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:40.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.858+0000 7f22effff700 1 -- 172.21.15.3:0/3047035918 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5473+0+0 (secure 0 0 0) 0x7f22fc088d70 con 0x7f23080cff30 2024-05-08T21:22:40.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.858+0000 7f23077fe700 1 --2- 172.21.15.3:0/3047035918 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f22f406ac90 0x7f22f406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:40.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.859+0000 7f23077fe700 1 --2- 172.21.15.3:0/3047035918 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f22f406ac90 0x7f22f406d150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009130 tx=0x7f22f8008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:40.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:40.862+0000 7f22effff700 1 -- 172.21.15.3:0/3047035918 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f22fc0540c0 con 0x7f23080cff30 2024-05-08T21:22:41.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.078+0000 7f230da74700 1 -- 172.21.15.3:0/3047035918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2308074bc0 con 0x7f23080cff30 2024-05-08T21:22:41.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.687+0000 7f22effff700 1 -- 172.21.15.3:0/3047035918 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7f22fc0210d0 con 0x7f23080cff30 2024-05-08T21:22:41.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.690+0000 7f230da74700 1 -- 172.21.15.3:0/3047035918 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f22f406ac90 msgr2=0x7f22f406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:41.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.690+0000 7f230da74700 1 --2- 172.21.15.3:0/3047035918 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f22f406ac90 0x7f22f406d150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009130 tx=0x7f22f8008030 comp rx=0 tx=0).stop 2024-05-08T21:22:41.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.690+0000 7f230da74700 1 -- 172.21.15.3:0/3047035918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23080cff30 msgr2=0x7f230809d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:41.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.690+0000 7f230da74700 1 --2- 172.21.15.3:0/3047035918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23080cff30 0x7f230809d800 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00e7e0 tx=0x7f22fc00eaf0 comp rx=0 tx=0).stop 2024-05-08T21:22:41.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.690+0000 7f230da74700 1 -- 172.21.15.3:0/3047035918 shutdown_connections 2024-05-08T21:22:41.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.690+0000 7f230da74700 1 --2- 172.21.15.3:0/3047035918 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f230815b710 0x7f23080b4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:41.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.690+0000 7f230da74700 1 --2- 172.21.15.3:0/3047035918 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f22f406ac90 0x7f22f406d150 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:41.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.690+0000 7f230da74700 1 --2- 172.21.15.3:0/3047035918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23080cf040 0x7f2308158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:41.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.690+0000 7f230da74700 1 --2- 172.21.15.3:0/3047035918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23080cff30 0x7f230809d800 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:41.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.690+0000 7f230da74700 1 -- 172.21.15.3:0/3047035918 >> 172.21.15.3:0/3047035918 conn(0x7f23080b2f60 msgr2=0x7f23080ae0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:41.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.691+0000 7f230da74700 1 -- 172.21.15.3:0/3047035918 shutdown_connections 2024-05-08T21:22:41.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.691+0000 7f230da74700 1 -- 172.21.15.3:0/3047035918 wait complete. 2024-05-08T21:22:41.693 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2024-05-08T21:22:41.709 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-08T21:22:41.709 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-08T21:22:41.709 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-08T21:22:41.710 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:22:41.710 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-08T21:22:41.710 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-08T21:22:41.710 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-08T21:22:41.710 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:22:41.710 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-08T21:22:41.710 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-08T21:22:41.711 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-08T21:22:41.711 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:22:41.711 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-08T21:22:41.711 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-08T21:22:41.711 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-08T21:22:41.711 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:22:41.721 INFO:tasks.ceph.mds.a:Stopped 2024-05-08T21:22:41.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-08T21:22:41.721 INFO:tasks.ceph.mds.c:Stopped 2024-05-08T21:22:41.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-08T21:22:41.723 INFO:tasks.ceph.mds.b:Stopped 2024-05-08T21:22:41.723 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-08T21:22:41.724 INFO:tasks.ceph.mds.d:Stopped 2024-05-08T21:22:41.724 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-08T21:22:41.764 INFO:tasks.ceph.mds.a:Started 2024-05-08T21:22:41.764 DEBUG:teuthology.parallel:result is None 2024-05-08T21:22:41.766 INFO:tasks.ceph.mds.c:Started 2024-05-08T21:22:41.766 DEBUG:teuthology.parallel:result is None 2024-05-08T21:22:41.766 INFO:tasks.ceph.mds.b:Started 2024-05-08T21:22:41.767 DEBUG:teuthology.parallel:result is None 2024-05-08T21:22:41.767 INFO:tasks.ceph.mds.d:Started 2024-05-08T21:22:41.767 DEBUG:teuthology.parallel:result is None 2024-05-08T21:22:41.768 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-08T21:22:41.851 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2024-05-08T21:22:41.851 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2024-05-08T21:22:41.860 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2024-05-08T21:22:41.861 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2024-05-08T21:22:41.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.925+0000 7f047f337700 1 Processor -- start 2024-05-08T21:22:41.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.926+0000 7f047f337700 1 -- start start 2024-05-08T21:22:41.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.926+0000 7f047f337700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f047806d7e0 0x7f047815ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:41.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.926+0000 7f047f337700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f047806dbc0 0x7f0478143150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:41.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.926+0000 7f047f337700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f04780716c0 0x7f0478143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:41.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.926+0000 7f047f337700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f047815b1c0 con 0x7f04780716c0 2024-05-08T21:22:41.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.926+0000 7f047f337700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0478071c60 con 0x7f047806dbc0 2024-05-08T21:22:41.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.926+0000 7f047f337700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0478071f40 con 0x7f047806d7e0 2024-05-08T21:22:41.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.926+0000 7f047d8d4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f04780716c0 0x7f0478143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:41.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.927+0000 7f047d8d4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f04780716c0 0x7f0478143690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59866/0 (socket says 172.21.15.3:59866) 2024-05-08T21:22:41.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.927+0000 7f047d8d4700 1 -- 172.21.15.3:0/1970784372 learned_addr learned my addr 172.21.15.3:0/1970784372 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:41.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.927+0000 7f047c8d2700 1 --2- 172.21.15.3:0/1970784372 >> v2:172.21.15.186:3300/0 conn(0x7f047806dbc0 0x7f0478143150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:41.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.927+0000 7f047d0d3700 1 --2- 172.21.15.3:0/1970784372 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f047806d7e0 0x7f047815ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:41.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.927+0000 7f047d8d4700 1 -- 172.21.15.3:0/1970784372 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f047806d7e0 msgr2=0x7f047815ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:41.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.927+0000 7f047d8d4700 1 --2- 172.21.15.3:0/1970784372 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f047806d7e0 0x7f047815ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:41.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.927+0000 7f047d8d4700 1 -- 172.21.15.3:0/1970784372 >> v2:172.21.15.186:3300/0 conn(0x7f047806dbc0 msgr2=0x7f0478143150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:41.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.927+0000 7f047d8d4700 1 --2- 172.21.15.3:0/1970784372 >> v2:172.21.15.186:3300/0 conn(0x7f047806dbc0 0x7f0478143150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:41.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.927+0000 7f047d8d4700 1 -- 172.21.15.3:0/1970784372 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0478072220 con 0x7f04780716c0 2024-05-08T21:22:41.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.927+0000 7f047d8d4700 1 --2- 172.21.15.3:0/1970784372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04780716c0 0x7f0478143690 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f04780b3ee0 tx=0x7f047000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b77a3ce096a1e26 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:41.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.927+0000 7f046ffff700 1 -- 172.21.15.3:0/1970784372 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047000e070 con 0x7f04780716c0 2024-05-08T21:22:41.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.927+0000 7f046ffff700 1 -- 172.21.15.3:0/1970784372 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047000b910 con 0x7f04780716c0 2024-05-08T21:22:41.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.927+0000 7f046ffff700 1 -- 172.21.15.3:0/1970784372 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047000bcc0 con 0x7f04780716c0 2024-05-08T21:22:41.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.928+0000 7f047f337700 1 -- 172.21.15.3:0/1970784372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04780716c0 msgr2=0x7f0478143690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:41.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.928+0000 7f047f337700 1 --2- 172.21.15.3:0/1970784372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04780716c0 0x7f0478143690 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f04780b3ee0 tx=0x7f047000b330 comp rx=0 tx=0).stop 2024-05-08T21:22:41.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.928+0000 7f047f337700 1 -- 172.21.15.3:0/1970784372 shutdown_connections 2024-05-08T21:22:41.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.928+0000 7f047f337700 1 --2- 172.21.15.3:0/1970784372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04780716c0 0x7f0478143690 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:41.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.928+0000 7f047f337700 1 --2- 172.21.15.3:0/1970784372 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f047806d7e0 0x7f047815ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:41.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.928+0000 7f047f337700 1 --2- 172.21.15.3:0/1970784372 >> v2:172.21.15.186:3300/0 conn(0x7f047806dbc0 0x7f0478143150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:41.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.928+0000 7f047f337700 1 -- 172.21.15.3:0/1970784372 >> 172.21.15.3:0/1970784372 conn(0x7f04780b10c0 msgr2=0x7f04780b14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:41.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.928+0000 7f047f337700 1 -- 172.21.15.3:0/1970784372 shutdown_connections 2024-05-08T21:22:41.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.928+0000 7f047f337700 1 -- 172.21.15.3:0/1970784372 wait complete. 2024-05-08T21:22:41.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.929+0000 7f047f337700 1 Processor -- start 2024-05-08T21:22:41.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.929+0000 7f047f337700 1 -- start start 2024-05-08T21:22:41.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.930+0000 7f047f337700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f047806d7e0 0x7f047815b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:41.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.930+0000 7f047f337700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f047806dbc0 0x7f04781f6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:41.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.930+0000 7f047f337700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04780716c0 0x7f04781f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:41.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.930+0000 7f047f337700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047815c920 con 0x7f047806dbc0 2024-05-08T21:22:41.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.930+0000 7f047f337700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047815cdb0 con 0x7f04780716c0 2024-05-08T21:22:41.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.930+0000 7f047f337700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f047815bd10 con 0x7f047806d7e0 2024-05-08T21:22:41.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.930+0000 7f047c8d2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f047806dbc0 0x7f04781f6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:41.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.930+0000 7f047c8d2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f047806dbc0 0x7f04781f6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59882/0 (socket says 172.21.15.3:59882) 2024-05-08T21:22:41.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.930+0000 7f047c8d2700 1 -- 172.21.15.3:0/2298532670 learned_addr learned my addr 172.21.15.3:0/2298532670 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:41.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.930+0000 7f047d8d4700 1 --2- 172.21.15.3:0/2298532670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04780716c0 0x7f04781f6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:41.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.930+0000 7f047d0d3700 1 --2- 172.21.15.3:0/2298532670 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f047806d7e0 0x7f047815b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:41.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.931+0000 7f047d8d4700 1 -- 172.21.15.3:0/2298532670 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f047806d7e0 msgr2=0x7f047815b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:41.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.931+0000 7f047d8d4700 1 --2- 172.21.15.3:0/2298532670 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f047806d7e0 0x7f047815b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:41.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.931+0000 7f047d8d4700 1 -- 172.21.15.3:0/2298532670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f047806dbc0 msgr2=0x7f04781f6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:41.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.931+0000 7f047d8d4700 1 --2- 172.21.15.3:0/2298532670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f047806dbc0 0x7f04781f6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:41.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.931+0000 7f047d8d4700 1 -- 172.21.15.3:0/2298532670 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0470009b30 con 0x7f04780716c0 2024-05-08T21:22:41.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.931+0000 7f047c8d2700 1 --2- 172.21.15.3:0/2298532670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f047806dbc0 0x7f04781f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:41.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.931+0000 7f047d8d4700 1 --2- 172.21.15.3:0/2298532670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04780716c0 0x7f04781f6e80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0470015000 tx=0x7f0470004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:41.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.931+0000 7f046dffb700 1 -- 172.21.15.3:0/2298532670 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04700099b0 con 0x7f04780716c0 2024-05-08T21:22:41.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.931+0000 7f046dffb700 1 -- 172.21.15.3:0/2298532670 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047000ce80 con 0x7f04780716c0 2024-05-08T21:22:41.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.931+0000 7f047f337700 1 -- 172.21.15.3:0/2298532670 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f047815bff0 con 0x7f04780716c0 2024-05-08T21:22:41.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.931+0000 7f046dffb700 1 -- 172.21.15.3:0/2298532670 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0470004490 con 0x7f04780716c0 2024-05-08T21:22:41.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.931+0000 7f047f337700 1 -- 172.21.15.3:0/2298532670 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047815c5b0 con 0x7f04780716c0 2024-05-08T21:22:41.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.933+0000 7f047f337700 1 -- 172.21.15.3:0/2298532670 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f047815c740 con 0x7f04780716c0 2024-05-08T21:22:41.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.933+0000 7f046dffb700 1 -- 172.21.15.3:0/2298532670 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f04700045f0 con 0x7f04780716c0 2024-05-08T21:22:41.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.933+0000 7f046dffb700 1 --2- 172.21.15.3:0/2298532670 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f045c06ac90 0x7f045c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:41.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.933+0000 7f046dffb700 1 -- 172.21.15.3:0/2298532670 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0470089ed0 con 0x7f04780716c0 2024-05-08T21:22:41.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.934+0000 7f047d0d3700 1 --2- 172.21.15.3:0/2298532670 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f045c06ac90 0x7f045c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:41.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.934+0000 7f047d0d3700 1 --2- 172.21.15.3:0/2298532670 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f045c06ac90 0x7f045c06d150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f04780cd6c0 tx=0x7f0474009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:41.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:41.938+0000 7f046dffb700 1 -- 172.21.15.3:0/2298532670 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0470055440 con 0x7f04780716c0 2024-05-08T21:22:41.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-08T21:22:41.961 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-08T21:22:42.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:42.185+0000 7f047f337700 1 -- 172.21.15.3:0/2298532670 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f047815ca90 con 0x7f04780716c0 2024-05-08T21:22:42.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:42.906+0000 7f046dffb700 1 -- 172.21.15.3:0/2298532670 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v30) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0470058a60 con 0x7f04780716c0 2024-05-08T21:22:42.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:42.909+0000 7f047f337700 1 -- 172.21.15.3:0/2298532670 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f045c06ac90 msgr2=0x7f045c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:42.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:42.909+0000 7f047f337700 1 --2- 172.21.15.3:0/2298532670 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f045c06ac90 0x7f045c06d150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f04780cd6c0 tx=0x7f0474009210 comp rx=0 tx=0).stop 2024-05-08T21:22:42.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:42.909+0000 7f047f337700 1 -- 172.21.15.3:0/2298532670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04780716c0 msgr2=0x7f04781f6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:42.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:42.909+0000 7f047f337700 1 --2- 172.21.15.3:0/2298532670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04780716c0 0x7f04781f6e80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0470015000 tx=0x7f0470004ad0 comp rx=0 tx=0).stop 2024-05-08T21:22:42.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:42.909+0000 7f047f337700 1 -- 172.21.15.3:0/2298532670 shutdown_connections 2024-05-08T21:22:42.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:42.909+0000 7f047f337700 1 --2- 172.21.15.3:0/2298532670 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f047806d7e0 0x7f047815b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:42.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:42.909+0000 7f047f337700 1 --2- 172.21.15.3:0/2298532670 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f045c06ac90 0x7f045c06d150 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:42.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:42.909+0000 7f047f337700 1 --2- 172.21.15.3:0/2298532670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04780716c0 0x7f04781f6e80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:42.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:42.909+0000 7f047f337700 1 --2- 172.21.15.3:0/2298532670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f047806dbc0 0x7f04781f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:42.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:42.909+0000 7f047f337700 1 -- 172.21.15.3:0/2298532670 >> 172.21.15.3:0/2298532670 conn(0x7f04780b10c0 msgr2=0x7f0478098790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:42.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:42.909+0000 7f047f337700 1 -- 172.21.15.3:0/2298532670 shutdown_connections 2024-05-08T21:22:42.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:42.909+0000 7f047f337700 1 -- 172.21.15.3:0/2298532670 wait complete. 2024-05-08T21:22:42.912 INFO:teuthology.orchestra.run.smithi003.stderr: removed all blocklist entries 2024-05-08T21:22:42.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-08T21:22:43.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.065+0000 7f75d825c700 1 Processor -- start 2024-05-08T21:22:43.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.065+0000 7f75d825c700 1 -- start start 2024-05-08T21:22:43.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.066+0000 7f75d825c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d00d3f40 0x7f75d0161f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:43.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.066+0000 7f75d825c700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d0162450 con 0x7f75d015db80 2024-05-08T21:22:43.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.066+0000 7f75d825c700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d00cee00 con 0x7f75d015d7a0 2024-05-08T21:22:43.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.066+0000 7f75d825c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d00b27e0 con 0x7f75d00d3f40 2024-05-08T21:22:43.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.066+0000 7f75d67f9700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f75d015db80 0x7f75d00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34602/0 (socket says 172.21.15.3:34602) 2024-05-08T21:22:43.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.066+0000 7f75d67f9700 1 -- 172.21.15.3:0/1512641250 learned_addr learned my addr 172.21.15.3:0/1512641250 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:43.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.066+0000 7f75d57f7700 1 --2- 172.21.15.3:0/1512641250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d00d3f40 0x7f75d0161f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:43.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.066+0000 7f75d4ff6700 1 -- 172.21.15.3:0/1512641250 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551132473 0 0) 0x7f75c8001490 con 0x7f75d015db80 2024-05-08T21:22:43.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.066+0000 7f75d4ff6700 1 -- 172.21.15.3:0/1512641250 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75bc0035b0 con 0x7f75d015db80 2024-05-08T21:22:43.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.066+0000 7f75d4ff6700 1 -- 172.21.15.3:0/1512641250 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490465003 0 0) 0x7f75d00cee00 con 0x7f75d015d7a0 2024-05-08T21:22:43.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.066+0000 7f75d4ff6700 1 -- 172.21.15.3:0/1512641250 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c8001490 con 0x7f75d015d7a0 2024-05-08T21:22:43.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.066+0000 7f75d4ff6700 1 -- 172.21.15.3:0/1512641250 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4271885613 0 0) 0x7f75bc0035b0 con 0x7f75d015db80 2024-05-08T21:22:43.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.066+0000 7f75d4ff6700 1 -- 172.21.15.3:0/1512641250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d00d3f40 msgr2=0x7f75d0161f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.066+0000 7f75d4ff6700 1 --2- 172.21.15.3:0/1512641250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d00d3f40 0x7f75d0161f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d4ff6700 1 -- 172.21.15.3:0/1512641250 >> v1:172.21.15.186:6789/0 conn(0x7f75d015d7a0 legacy=0x7f75d00b0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d4ff6700 1 -- 172.21.15.3:0/1512641250 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d00d4400 con 0x7f75d015db80 2024-05-08T21:22:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d4ff6700 1 -- 172.21.15.3:0/1512641250 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f75c8001b80 con 0x7f75d015db80 2024-05-08T21:22:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d57f7700 1 --2- 172.21.15.3:0/1512641250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d00d3f40 0x7f75d0161f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d4ff6700 1 -- 172.21.15.3:0/1512641250 >> v1:172.21.15.3:6789/0 conn(0x7f75d015db80 legacy=0x7f75d00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d4ff6700 1 --2- 172.21.15.3:0/1512641250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc003bc0 0x7f75bc003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d4ff6700 1 --2- 172.21.15.3:0/1512641250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75bc004850 0x7f75bc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d4ff6700 1 --2- 172.21.15.3:0/1512641250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d00d3f40 0x7f75bc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d4ff6700 1 -- 172.21.15.3:0/1512641250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d00b27e0 con 0x7f75bc004850 2024-05-08T21:22:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d4ff6700 1 -- 172.21.15.3:0/1512641250 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc00ba70 con 0x7f75bc003bc0 2024-05-08T21:22:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d4ff6700 1 -- 172.21.15.3:0/1512641250 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc00bc30 con 0x7f75d00d3f40 2024-05-08T21:22:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d57f7700 1 --2- 172.21.15.3:0/1512641250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75bc004850 0x7f75bc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d5ff8700 1 --2- 172.21.15.3:0/1512641250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc003bc0 0x7f75bc003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d67f9700 1 --2- 172.21.15.3:0/1512641250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d00d3f40 0x7f75bc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:43.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d57f7700 1 -- 172.21.15.3:0/1512641250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d00d3f40 msgr2=0x7f75bc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:43.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d57f7700 1 --2- 172.21.15.3:0/1512641250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d00d3f40 0x7f75bc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d57f7700 1 -- 172.21.15.3:0/1512641250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc003bc0 msgr2=0x7f75bc003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:43.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d57f7700 1 --2- 172.21.15.3:0/1512641250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc003bc0 0x7f75bc003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d57f7700 1 -- 172.21.15.3:0/1512641250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c0009430 con 0x7f75bc004850 2024-05-08T21:22:43.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d67f9700 1 --2- 172.21.15.3:0/1512641250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d00d3f40 0x7f75bc005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:43.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d57f7700 1 --2- 172.21.15.3:0/1512641250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75bc004850 0x7f75bc004cd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009ce0 tx=0x7f75bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72233c2b85fb6465 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:43.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d4ff6700 1 -- 172.21.15.3:0/1512641250 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c0005a20 con 0x7f75bc004850 2024-05-08T21:22:43.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d4ff6700 1 -- 172.21.15.3:0/1512641250 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c0005b80 con 0x7f75bc004850 2024-05-08T21:22:43.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.067+0000 7f75d4ff6700 1 -- 172.21.15.3:0/1512641250 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c000e7e0 con 0x7f75bc004850 2024-05-08T21:22:43.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.068+0000 7f75d825c700 1 -- 172.21.15.3:0/1512641250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75bc004850 msgr2=0x7f75bc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:43.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.068+0000 7f75d825c700 1 --2- 172.21.15.3:0/1512641250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75bc004850 0x7f75bc004cd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009ce0 tx=0x7f75bc005880 comp rx=0 tx=0).stop 2024-05-08T21:22:43.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.068+0000 7f75d67f9700 1 -- 172.21.15.3:0/1512641250 reap_dead start 2024-05-08T21:22:43.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.068+0000 7f75d825c700 1 -- 172.21.15.3:0/1512641250 shutdown_connections 2024-05-08T21:22:43.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.068+0000 7f75d825c700 1 -- 172.21.15.3:0/1512641250 >> 172.21.15.3:0/1512641250 conn(0x7f75d008afe0 msgr2=0x7f75d00d3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:43.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.068+0000 7f75d825c700 1 -- 172.21.15.3:0/1512641250 shutdown_connections 2024-05-08T21:22:43.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.068+0000 7f75d825c700 1 -- 172.21.15.3:0/1512641250 wait complete. 2024-05-08T21:22:43.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.069+0000 7f75d825c700 1 Processor -- start 2024-05-08T21:22:43.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.070+0000 7f75d825c700 1 -- start start 2024-05-08T21:22:43.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.070+0000 7f75d825c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc004850 0x7f75d015e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:43.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.070+0000 7f75d825c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75d0157e20 0x7f75d0160510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:43.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.070+0000 7f75d825c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d01582a0 0x7f75d0160a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:43.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.070+0000 7f75d825c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d01f6c70 con 0x7f75d0157e20 2024-05-08T21:22:43.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.070+0000 7f75d825c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d00ca4b0 con 0x7f75bc004850 2024-05-08T21:22:43.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.070+0000 7f75d825c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d00caad0 con 0x7f75d01582a0 2024-05-08T21:22:43.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.070+0000 7f75d57f7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75d0157e20 0x7f75d0160510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:43.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.071+0000 7f75d57f7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75d0157e20 0x7f75d0160510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59906/0 (socket says 172.21.15.3:59906) 2024-05-08T21:22:43.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.071+0000 7f75d57f7700 1 -- 172.21.15.3:0/3731613569 learned_addr learned my addr 172.21.15.3:0/3731613569 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:43.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.071+0000 7f75d5ff8700 1 --2- 172.21.15.3:0/3731613569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc004850 0x7f75d015e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:43.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.071+0000 7f75d67f9700 1 --2- 172.21.15.3:0/3731613569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d01582a0 0x7f75d0160a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:43.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.071+0000 7f75d57f7700 1 -- 172.21.15.3:0/3731613569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d01582a0 msgr2=0x7f75d0160a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:43.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.071+0000 7f75d57f7700 1 --2- 172.21.15.3:0/3731613569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d01582a0 0x7f75d0160a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.071+0000 7f75d57f7700 1 -- 172.21.15.3:0/3731613569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc004850 msgr2=0x7f75d015e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:43.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.071+0000 7f75d57f7700 1 --2- 172.21.15.3:0/3731613569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc004850 0x7f75d015e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.071+0000 7f75d57f7700 1 -- 172.21.15.3:0/3731613569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d00cadb0 con 0x7f75d0157e20 2024-05-08T21:22:43.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.071+0000 7f75d57f7700 1 --2- 172.21.15.3:0/3731613569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75d0157e20 0x7f75d0160510 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f75c000e4b0 tx=0x7f75c00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:43.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.071+0000 7f75c67fc700 1 -- 172.21.15.3:0/3731613569 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c0006440 con 0x7f75d0157e20 2024-05-08T21:22:43.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.071+0000 7f75c67fc700 1 -- 172.21.15.3:0/3731613569 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c00065c0 con 0x7f75d0157e20 2024-05-08T21:22:43.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.071+0000 7f75d825c700 1 -- 172.21.15.3:0/3731613569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d00cb090 con 0x7f75d0157e20 2024-05-08T21:22:43.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.071+0000 7f75c67fc700 1 -- 172.21.15.3:0/3731613569 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c0006970 con 0x7f75d0157e20 2024-05-08T21:22:43.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.071+0000 7f75d825c700 1 -- 172.21.15.3:0/3731613569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d015ee60 con 0x7f75d0157e20 2024-05-08T21:22:43.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.072+0000 7f75d825c700 1 -- 172.21.15.3:0/3731613569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d015f020 con 0x7f75d0157e20 2024-05-08T21:22:43.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.073+0000 7f75c67fc700 1 -- 172.21.15.3:0/3731613569 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f75c0013470 con 0x7f75d0157e20 2024-05-08T21:22:43.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.073+0000 7f75c67fc700 1 --2- 172.21.15.3:0/3731613569 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f75b406ac90 0x7f75b406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:43.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.073+0000 7f75c67fc700 1 -- 172.21.15.3:0/3731613569 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f75c008aaa0 con 0x7f75d0157e20 2024-05-08T21:22:43.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.073+0000 7f75d5ff8700 1 --2- 172.21.15.3:0/3731613569 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f75b406ac90 0x7f75b406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:43.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.074+0000 7f75d5ff8700 1 --2- 172.21.15.3:0/3731613569 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f75b406ac90 0x7f75b406d150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f75cc007fd0 tx=0x7f75cc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:43.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.078+0000 7f75c67fc700 1 -- 172.21.15.3:0/3731613569 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f75c0056180 con 0x7f75d0157e20 2024-05-08T21:22:43.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.385+0000 7f75d825c700 1 -- 172.21.15.3:0/3731613569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f75d01f6ee0 con 0x7f75d0157e20 2024-05-08T21:22:43.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.385+0000 7f75c67fc700 1 -- 172.21.15.3:0/3731613569 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f75c0012e60 con 0x7f75d0157e20 2024-05-08T21:22:43.387 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:43.387 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:22:43.387 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_dump": [ 2024-05-08T21:22:43.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.387 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.a", 2024-05-08T21:22:43.387 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCs7Dtmcb2tHRAA2QzDU8WLD64M0ONuKl5Iog==", 2024-05-08T21:22:43.387 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.387 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2024-05-08T21:22:43.388 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2024-05-08T21:22:43.388 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2024-05-08T21:22:43.388 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2024-05-08T21:22:43.388 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.388 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.b", 2024-05-08T21:22:43.388 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCs7DtmVPZqKxAAp4iAjTWYy+DPBtuOQFNIXw==", 2024-05-08T21:22:43.388 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.388 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2024-05-08T21:22:43.388 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2024-05-08T21:22:43.388 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2024-05-08T21:22:43.388 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2024-05-08T21:22:43.388 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.389 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.c", 2024-05-08T21:22:43.389 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCs7Dtmaw4NJhAAWdciLu9JWl39vhcVAdnBHw==", 2024-05-08T21:22:43.389 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.389 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2024-05-08T21:22:43.389 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2024-05-08T21:22:43.389 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2024-05-08T21:22:43.389 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2024-05-08T21:22:43.389 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.389 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.d", 2024-05-08T21:22:43.389 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCs7Dtmw3nKMhAAGtM2BT5AWuVFdFTRwte3IQ==", 2024-05-08T21:22:43.390 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.390 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2024-05-08T21:22:43.390 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2024-05-08T21:22:43.390 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2024-05-08T21:22:43.390 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2024-05-08T21:22:43.390 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.390 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.0", 2024-05-08T21:22:43.390 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCw7Dtm250HFBAAS7AJ95J560z3oTTa9CFbOw==", 2024-05-08T21:22:43.390 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.390 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow profile osd", 2024-05-08T21:22:43.390 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile osd", 2024-05-08T21:22:43.390 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2024-05-08T21:22:43.391 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.391 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.1", 2024-05-08T21:22:43.391 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCz7DtmiR9oFBAAyJXjJxS/AtGBybRyKVoSAQ==", 2024-05-08T21:22:43.391 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.391 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow profile osd", 2024-05-08T21:22:43.391 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile osd", 2024-05-08T21:22:43.391 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2024-05-08T21:22:43.391 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.391 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.2", 2024-05-08T21:22:43.391 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC27Dtm8XYHFBAAhVst/mX9SsJAEA5fFUA6Kw==", 2024-05-08T21:22:43.391 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow profile osd", 2024-05-08T21:22:43.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile osd", 2024-05-08T21:22:43.392 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2024-05-08T21:22:43.392 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.392 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.3", 2024-05-08T21:22:43.392 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC57Dtm0rizExAA68YZqUs2zq1tBwVb8Q+tvw==", 2024-05-08T21:22:43.392 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow profile osd", 2024-05-08T21:22:43.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile osd", 2024-05-08T21:22:43.392 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2024-05-08T21:22:43.392 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.393 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.4", 2024-05-08T21:22:43.393 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC/7DtmA3+4GBAAaye0ak/pWNIdDz+UnzNvLA==", 2024-05-08T21:22:43.393 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow profile osd", 2024-05-08T21:22:43.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile osd", 2024-05-08T21:22:43.393 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2024-05-08T21:22:43.393 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.393 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.5", 2024-05-08T21:22:43.393 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDC7DtmRpbhFxAAIKofN/JzliaxKIXZLVGw5A==", 2024-05-08T21:22:43.393 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow profile osd", 2024-05-08T21:22:43.393 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile osd", 2024-05-08T21:22:43.394 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2024-05-08T21:22:43.394 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.394 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.6", 2024-05-08T21:22:43.394 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDF7DtmvB1hFhAAQowb3SH+rt7zIMK4Dz8FdA==", 2024-05-08T21:22:43.394 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.394 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow profile osd", 2024-05-08T21:22:43.394 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile osd", 2024-05-08T21:22:43.394 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2024-05-08T21:22:43.394 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.394 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.7", 2024-05-08T21:22:43.395 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDI7Dtm/8cFFhAAe+fAJVIoH1I9pwwnrqRssg==", 2024-05-08T21:22:43.395 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow profile osd", 2024-05-08T21:22:43.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile osd", 2024-05-08T21:22:43.395 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2024-05-08T21:22:43.395 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.395 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.0", 2024-05-08T21:22:43.395 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCs7DtmLtjINxAA2457rrNLNP06VzQmA55MMA==", 2024-05-08T21:22:43.395 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2024-05-08T21:22:43.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow r", 2024-05-08T21:22:43.395 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow rw", 2024-05-08T21:22:43.396 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rwx" 2024-05-08T21:22:43.396 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.396 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.1", 2024-05-08T21:22:43.396 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCt7DtmqYziBBAAOgnWbP2XqOMqQDiranNyjQ==", 2024-05-08T21:22:43.396 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2024-05-08T21:22:43.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow r", 2024-05-08T21:22:43.396 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow rw", 2024-05-08T21:22:43.397 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rwx" 2024-05-08T21:22:43.397 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.397 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.2", 2024-05-08T21:22:43.397 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCt7Dtm3idSARAAdMThdnq8J9CBIBB4Ja0zgQ==", 2024-05-08T21:22:43.397 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.397 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2024-05-08T21:22:43.397 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow r", 2024-05-08T21:22:43.397 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow rw", 2024-05-08T21:22:43.397 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rwx" 2024-05-08T21:22:43.397 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.398 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.3", 2024-05-08T21:22:43.398 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCt7DtmLyo2CBAAy/Em6jfx4vDVKXAeEhAO/Q==", 2024-05-08T21:22:43.398 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2024-05-08T21:22:43.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow r", 2024-05-08T21:22:43.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow rw", 2024-05-08T21:22:43.398 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rwx" 2024-05-08T21:22:43.398 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.398 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.admin", 2024-05-08T21:22:43.398 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCr7DtmMHvFOBAAJJC1EcB6cnJXBeNilmUTzw==", 2024-05-08T21:22:43.398 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2024-05-08T21:22:43.398 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2024-05-08T21:22:43.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2024-05-08T21:22:43.399 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2024-05-08T21:22:43.399 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.399 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.bootstrap-mds", 2024-05-08T21:22:43.399 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDY7DtmLcqSIBAAv4ghPDiOcpSaeVFSrUCRiw==", 2024-05-08T21:22:43.399 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile bootstrap-mds" 2024-05-08T21:22:43.399 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.399 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.bootstrap-mgr", 2024-05-08T21:22:43.400 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDY7Dtm2tWSIBAA9j8DujeiROpgvOi+vPPmYA==", 2024-05-08T21:22:43.400 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.400 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile bootstrap-mgr" 2024-05-08T21:22:43.400 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.400 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.bootstrap-osd", 2024-05-08T21:22:43.400 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDY7DtmbN+SIBAA8xIm8J94JFLbvcOvd7DoMA==", 2024-05-08T21:22:43.400 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.400 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile bootstrap-osd" 2024-05-08T21:22:43.400 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.400 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.bootstrap-rbd", 2024-05-08T21:22:43.401 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDY7DtmbACTIBAA0VYlkIfHzEHMMjqwCdL9OA==", 2024-05-08T21:22:43.401 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile bootstrap-rbd" 2024-05-08T21:22:43.401 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.401 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-05-08T21:22:43.401 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDY7Dtm+gqTIBAAaZN64/FDsnw4fQJZHJ8zsw==", 2024-05-08T21:22:43.401 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.401 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-05-08T21:22:43.401 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.401 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.bootstrap-rgw", 2024-05-08T21:22:43.401 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDY7DtmGxWTIBAAZlW8iNZ0FbMh0kyYEqmd3w==", 2024-05-08T21:22:43.402 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile bootstrap-rgw" 2024-05-08T21:22:43.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.402 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.x", 2024-05-08T21:22:43.402 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCs7DtmYF5dGRAAmBIxydvBNs7D+BQ5Gn2pSw==", 2024-05-08T21:22:43.402 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2024-05-08T21:22:43.402 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2024-05-08T21:22:43.402 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2024-05-08T21:22:43.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-05-08T21:22:43.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:43.403 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.y", 2024-05-08T21:22:43.403 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCs7DtmF9wvFRAArRqZRUxiZFut4m22n8Vw+A==", 2024-05-08T21:22:43.403 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2024-05-08T21:22:43.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2024-05-08T21:22:43.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2024-05-08T21:22:43.403 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2024-05-08T21:22:43.403 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.403 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:43.403 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-05-08T21:22:43.403 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:22:43.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.388+0000 7f75d825c700 1 -- 172.21.15.3:0/3731613569 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f75b406ac90 msgr2=0x7f75b406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:43.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.388+0000 7f75d825c700 1 --2- 172.21.15.3:0/3731613569 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f75b406ac90 0x7f75b406d150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f75cc007fd0 tx=0x7f75cc009040 comp rx=0 tx=0).stop 2024-05-08T21:22:43.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.388+0000 7f75d825c700 1 -- 172.21.15.3:0/3731613569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75d0157e20 msgr2=0x7f75d0160510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:43.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.388+0000 7f75d825c700 1 --2- 172.21.15.3:0/3731613569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75d0157e20 0x7f75d0160510 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f75c000e4b0 tx=0x7f75c00047c0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.388+0000 7f75d825c700 1 -- 172.21.15.3:0/3731613569 shutdown_connections 2024-05-08T21:22:43.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.388+0000 7f75d825c700 1 --2- 172.21.15.3:0/3731613569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d01582a0 0x7f75d0160a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.388+0000 7f75d825c700 1 --2- 172.21.15.3:0/3731613569 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f75b406ac90 0x7f75b406d150 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.388+0000 7f75d825c700 1 --2- 172.21.15.3:0/3731613569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc004850 0x7f75d015e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.388+0000 7f75d825c700 1 --2- 172.21.15.3:0/3731613569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75d0157e20 0x7f75d0160510 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.388+0000 7f75d825c700 1 -- 172.21.15.3:0/3731613569 >> 172.21.15.3:0/3731613569 conn(0x7f75d008afe0 msgr2=0x7f75d006bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:43.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.388+0000 7f75d825c700 1 -- 172.21.15.3:0/3731613569 shutdown_connections 2024-05-08T21:22:43.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.388+0000 7f75d825c700 1 -- 172.21.15.3:0/3731613569 wait complete. 2024-05-08T21:22:43.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-05-08T21:22:43.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.542+0000 7f1adc29c700 1 Processor -- start 2024-05-08T21:22:43.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.542+0000 7f1adc29c700 1 -- start start 2024-05-08T21:22:43.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.543+0000 7f1adc29c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1ad415ba00 0x7f1ad40b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:43.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.543+0000 7f1adc29c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1ad409a160 0x7f1ad40b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:43.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.543+0000 7f1adc29c700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad40b4280 con 0x7f1ad40b2510 2024-05-08T21:22:43.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.543+0000 7f1adc29c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad40b4560 con 0x7f1ad415ba00 2024-05-08T21:22:43.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.543+0000 7f1adc29c700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad40cccf0 con 0x7f1ad409a160 2024-05-08T21:22:43.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.543+0000 7f1ada038700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1ad415ba00 0x7f1ad40b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:43.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.543+0000 7f1ada038700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1ad415ba00 0x7f1ad40b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59918/0 (socket says 172.21.15.3:59918) 2024-05-08T21:22:43.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.543+0000 7f1ada038700 1 -- 172.21.15.3:0/459739411 learned_addr learned my addr 172.21.15.3:0/459739411 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:43.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.543+0000 7f1ada839700 1 --2- 172.21.15.3:0/459739411 >> v2:172.21.15.186:3300/0 conn(0x7f1ad409a160 0x7f1ad40b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:43.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.544+0000 7f1ada038700 1 -- 172.21.15.3:0/459739411 >> v2:172.21.15.186:3300/0 conn(0x7f1ad409a160 msgr2=0x7f1ad40b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:43.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.544+0000 7f1ada038700 1 --2- 172.21.15.3:0/459739411 >> v2:172.21.15.186:3300/0 conn(0x7f1ad409a160 0x7f1ad40b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.544+0000 7f1ada038700 1 -- 172.21.15.3:0/459739411 >> v1:172.21.15.186:6789/0 conn(0x7f1ad40b2510 legacy=0x7f1ad4099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:43.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.544+0000 7f1ada038700 1 -- 172.21.15.3:0/459739411 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad40ccfd0 con 0x7f1ad415ba00 2024-05-08T21:22:43.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.544+0000 7f1ada839700 1 --2- 172.21.15.3:0/459739411 >> v2:172.21.15.186:3300/0 conn(0x7f1ad409a160 0x7f1ad40b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:43.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.544+0000 7f1ada038700 1 --2- 172.21.15.3:0/459739411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad415ba00 0x7f1ad40b3a40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4073150 tx=0x7f1ad0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ee789dbea79cef2 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:43.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.544+0000 7f1ad9036700 1 -- 172.21.15.3:0/459739411 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad000b870 con 0x7f1ad415ba00 2024-05-08T21:22:43.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.544+0000 7f1ad9036700 1 -- 172.21.15.3:0/459739411 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad000b9d0 con 0x7f1ad415ba00 2024-05-08T21:22:43.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.544+0000 7f1ad9036700 1 -- 172.21.15.3:0/459739411 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad000bd60 con 0x7f1ad415ba00 2024-05-08T21:22:43.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.545+0000 7f1adc29c700 1 -- 172.21.15.3:0/459739411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad415ba00 msgr2=0x7f1ad40b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:43.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.545+0000 7f1adc29c700 1 --2- 172.21.15.3:0/459739411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad415ba00 0x7f1ad40b3a40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4073150 tx=0x7f1ad0009bd0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.545+0000 7f1adc29c700 1 -- 172.21.15.3:0/459739411 shutdown_connections 2024-05-08T21:22:43.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.545+0000 7f1adc29c700 1 --2- 172.21.15.3:0/459739411 >> v2:172.21.15.186:3300/0 conn(0x7f1ad409a160 0x7f1ad40b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.545+0000 7f1adc29c700 1 --2- 172.21.15.3:0/459739411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad415ba00 0x7f1ad40b3a40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.545+0000 7f1adc29c700 1 -- 172.21.15.3:0/459739411 >> 172.21.15.3:0/459739411 conn(0x7f1ad40b2f20 msgr2=0x7f1ad40b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:43.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.545+0000 7f1adc29c700 1 -- 172.21.15.3:0/459739411 shutdown_connections 2024-05-08T21:22:43.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.545+0000 7f1adc29c700 1 -- 172.21.15.3:0/459739411 wait complete. 2024-05-08T21:22:43.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.546+0000 7f1adc29c700 1 Processor -- start 2024-05-08T21:22:43.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.546+0000 7f1adc29c700 1 -- start start 2024-05-08T21:22:43.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1adc29c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad409a160 0x7f1ad40c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:43.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1adc29c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad40b2510 0x7f1ad4158780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:43.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1adc29c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad415ba00 0x7f1ad40cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:43.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1adc29c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad40ca070 con 0x7f1ad415ba00 2024-05-08T21:22:43.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1adc29c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad4158cc0 con 0x7f1ad409a160 2024-05-08T21:22:43.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1adc29c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad40cb2a0 con 0x7f1ad40b2510 2024-05-08T21:22:43.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1ada839700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad415ba00 0x7f1ad40cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:43.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1ada839700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad415ba00 0x7f1ad40cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59930/0 (socket says 172.21.15.3:59930) 2024-05-08T21:22:43.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1ada839700 1 -- 172.21.15.3:0/3093085067 learned_addr learned my addr 172.21.15.3:0/3093085067 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:43.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1ada038700 1 --2- 172.21.15.3:0/3093085067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad409a160 0x7f1ad40c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:43.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1ad9837700 1 --2- 172.21.15.3:0/3093085067 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad40b2510 0x7f1ad4158780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:43.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1ada839700 1 -- 172.21.15.3:0/3093085067 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad40b2510 msgr2=0x7f1ad4158780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:43.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1ada839700 1 --2- 172.21.15.3:0/3093085067 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad40b2510 0x7f1ad4158780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1ada839700 1 -- 172.21.15.3:0/3093085067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad409a160 msgr2=0x7f1ad40c8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:43.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1ada839700 1 --2- 172.21.15.3:0/3093085067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad409a160 0x7f1ad40c8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1ada839700 1 -- 172.21.15.3:0/3093085067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad0009550 con 0x7f1ad415ba00 2024-05-08T21:22:43.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.547+0000 7f1ad9837700 1 --2- 172.21.15.3:0/3093085067 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad40b2510 0x7f1ad4158780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:43.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.548+0000 7f1ada839700 1 --2- 172.21.15.3:0/3093085067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad415ba00 0x7f1ad40cad60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1acc007e30 tx=0x7f1acc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:43.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.548+0000 7f1acaffd700 1 -- 172.21.15.3:0/3093085067 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1acc00ddd0 con 0x7f1ad415ba00 2024-05-08T21:22:43.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.548+0000 7f1acaffd700 1 -- 172.21.15.3:0/3093085067 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1acc00f040 con 0x7f1ad415ba00 2024-05-08T21:22:43.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.548+0000 7f1adc29c700 1 -- 172.21.15.3:0/3093085067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad40c94f0 con 0x7f1ad415ba00 2024-05-08T21:22:43.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.548+0000 7f1acaffd700 1 -- 172.21.15.3:0/3093085067 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1acc003c60 con 0x7f1ad415ba00 2024-05-08T21:22:43.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.548+0000 7f1adc29c700 1 -- 172.21.15.3:0/3093085067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad40c9ab0 con 0x7f1ad415ba00 2024-05-08T21:22:43.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.549+0000 7f1adc29c700 1 -- 172.21.15.3:0/3093085067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad409b180 con 0x7f1ad415ba00 2024-05-08T21:22:43.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.549+0000 7f1acaffd700 1 -- 172.21.15.3:0/3093085067 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1acc004030 con 0x7f1ad415ba00 2024-05-08T21:22:43.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.550+0000 7f1acaffd700 1 --2- 172.21.15.3:0/3093085067 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1ab806ac90 0x7f1ab806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:43.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.550+0000 7f1acaffd700 1 -- 172.21.15.3:0/3093085067 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1acc089010 con 0x7f1ad415ba00 2024-05-08T21:22:43.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.550+0000 7f1ada038700 1 --2- 172.21.15.3:0/3093085067 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1ab806ac90 0x7f1ab806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:43.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.551+0000 7f1ada038700 1 --2- 172.21.15.3:0/3093085067 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1ab806ac90 0x7f1ab806d150 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0000c00 tx=0x7f1ad0008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:43.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.555+0000 7f1acaffd700 1 -- 172.21.15.3:0/3093085067 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f1ad409b180 con 0x7f1ad415ba00 2024-05-08T21:22:43.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.864+0000 7f1adc29c700 1 -- 172.21.15.3:0/3093085067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f1ad409b180 con 0x7f1ad415ba00 2024-05-08T21:22:43.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.868+0000 7f1acaffd700 1 -- 172.21.15.3:0/3093085067 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1ad409b180 con 0x7f1ad415ba00 2024-05-08T21:22:43.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.870+0000 7f1adc29c700 1 -- 172.21.15.3:0/3093085067 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1ab806ac90 msgr2=0x7f1ab806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:43.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.870+0000 7f1adc29c700 1 --2- 172.21.15.3:0/3093085067 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1ab806ac90 0x7f1ab806d150 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0000c00 tx=0x7f1ad0008030 comp rx=0 tx=0).stop 2024-05-08T21:22:43.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.870+0000 7f1adc29c700 1 -- 172.21.15.3:0/3093085067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad415ba00 msgr2=0x7f1ad40cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:43.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.870+0000 7f1adc29c700 1 --2- 172.21.15.3:0/3093085067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad415ba00 0x7f1ad40cad60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1acc007e30 tx=0x7f1acc00d3b0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.870+0000 7f1adc29c700 1 -- 172.21.15.3:0/3093085067 shutdown_connections 2024-05-08T21:22:43.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.870+0000 7f1adc29c700 1 --2- 172.21.15.3:0/3093085067 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f1ab806ac90 0x7f1ab806d150 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.870+0000 7f1adc29c700 1 --2- 172.21.15.3:0/3093085067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad415ba00 0x7f1ad40cad60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.870+0000 7f1adc29c700 1 --2- 172.21.15.3:0/3093085067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad409a160 0x7f1ad40c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.870+0000 7f1adc29c700 1 --2- 172.21.15.3:0/3093085067 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad40b2510 0x7f1ad4158780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:43.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.870+0000 7f1adc29c700 1 -- 172.21.15.3:0/3093085067 >> 172.21.15.3:0/3093085067 conn(0x7f1ad40b2f20 msgr2=0x7f1ad40b5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:43.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.870+0000 7f1adc29c700 1 -- 172.21.15.3:0/3093085067 shutdown_connections 2024-05-08T21:22:43.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:43.870+0000 7f1adc29c700 1 -- 172.21.15.3:0/3093085067 wait complete. 2024-05-08T21:22:43.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-05-08T21:22:44.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.021+0000 7ff6f1cc3700 1 Processor -- start 2024-05-08T21:22:44.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.021+0000 7ff6f1cc3700 1 -- start start 2024-05-08T21:22:44.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.022+0000 7ff6f1cc3700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff6ec1614f0 0x7ff6ec099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.022+0000 7ff6f1cc3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6ec0d0650 0x7ff6ec1618d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.022+0000 7ff6f1cc3700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec161e10 con 0x7ff6ec15d0a0 2024-05-08T21:22:44.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.022+0000 7ff6f1cc3700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6ec09a030 con 0x7ff6ec1614f0 2024-05-08T21:22:44.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.022+0000 7ff6f1cc3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec09a310 con 0x7ff6ec0d0650 2024-05-08T21:22:44.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.022+0000 7ff6e2ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6ec0d0650 0x7ff6ec1618d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.022+0000 7ff6ebfff700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff6ec15d0a0 0x7ff6ec15d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:52298/0 (socket says 172.21.15.3:52298) 2024-05-08T21:22:44.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.022+0000 7ff6eb7fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff6ec1614f0 0x7ff6ec099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.022+0000 7ff6ebfff700 1 -- 172.21.15.3:0/840107028 learned_addr learned my addr 172.21.15.3:0/840107028 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:44.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.023+0000 7ff6eaffd700 1 -- 172.21.15.3:0/840107028 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519239155 0 0) 0x7ff6ec161e10 con 0x7ff6ec15d0a0 2024-05-08T21:22:44.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.023+0000 7ff6eaffd700 1 -- 172.21.15.3:0/840107028 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d80035b0 con 0x7ff6ec15d0a0 2024-05-08T21:22:44.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.023+0000 7ff6e2ffd700 1 -- 172.21.15.3:0/840107028 >> v2:172.21.15.186:3300/0 conn(0x7ff6ec1614f0 msgr2=0x7ff6ec099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:44.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.023+0000 7ff6e2ffd700 1 --2- 172.21.15.3:0/840107028 >> v2:172.21.15.186:3300/0 conn(0x7ff6ec1614f0 0x7ff6ec099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.023+0000 7ff6e2ffd700 1 -- 172.21.15.3:0/840107028 >> v1:172.21.15.186:6789/0 conn(0x7ff6ec15d0a0 legacy=0x7ff6ec15d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:44.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.023+0000 7ff6e2ffd700 1 -- 172.21.15.3:0/840107028 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ec09a5f0 con 0x7ff6ec0d0650 2024-05-08T21:22:44.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.023+0000 7ff6eb7fe700 1 --2- 172.21.15.3:0/840107028 >> v2:172.21.15.186:3300/0 conn(0x7ff6ec1614f0 0x7ff6ec099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:44.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.023+0000 7ff6e2ffd700 1 --2- 172.21.15.3:0/840107028 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6ec0d0650 0x7ff6ec1618d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009840 tx=0x7ff6d4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53a560404e10308b server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:44.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.023+0000 7ff6eaffd700 1 -- 172.21.15.3:0/840107028 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4011830 con 0x7ff6ec0d0650 2024-05-08T21:22:44.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.023+0000 7ff6eaffd700 1 -- 172.21.15.3:0/840107028 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d4011990 con 0x7ff6ec0d0650 2024-05-08T21:22:44.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.023+0000 7ff6eaffd700 1 -- 172.21.15.3:0/840107028 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4011d20 con 0x7ff6ec0d0650 2024-05-08T21:22:44.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.024+0000 7ff6f1cc3700 1 -- 172.21.15.3:0/840107028 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6ec0d0650 msgr2=0x7ff6ec1618d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:44.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.024+0000 7ff6f1cc3700 1 --2- 172.21.15.3:0/840107028 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6ec0d0650 0x7ff6ec1618d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009840 tx=0x7ff6d4009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:44.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.024+0000 7ff6f1cc3700 1 -- 172.21.15.3:0/840107028 shutdown_connections 2024-05-08T21:22:44.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.024+0000 7ff6f1cc3700 1 --2- 172.21.15.3:0/840107028 >> v2:172.21.15.186:3300/0 conn(0x7ff6ec1614f0 0x7ff6ec099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.024+0000 7ff6f1cc3700 1 --2- 172.21.15.3:0/840107028 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6ec0d0650 0x7ff6ec1618d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.024+0000 7ff6f1cc3700 1 -- 172.21.15.3:0/840107028 >> 172.21.15.3:0/840107028 conn(0x7ff6ec0b2f20 msgr2=0x7ff6ec0b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:44.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.024+0000 7ff6f1cc3700 1 -- 172.21.15.3:0/840107028 shutdown_connections 2024-05-08T21:22:44.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.024+0000 7ff6f1cc3700 1 -- 172.21.15.3:0/840107028 wait complete. 2024-05-08T21:22:44.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.025+0000 7ff6f1cc3700 1 Processor -- start 2024-05-08T21:22:44.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.026+0000 7ff6f1cc3700 1 -- start start 2024-05-08T21:22:44.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.026+0000 7ff6f1cc3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6ec0d0650 0x7ff6ec1485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.026+0000 7ff6f1cc3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6ec15d0a0 0x7ff6ec1ecfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.026+0000 7ff6f1cc3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6ec1614f0 0x7ff6ec15dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.026+0000 7ff6f1cc3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec148b00 con 0x7ff6ec0d0650 2024-05-08T21:22:44.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.026+0000 7ff6f1cc3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec0c5fb0 con 0x7ff6ec15d0a0 2024-05-08T21:22:44.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.026+0000 7ff6f1cc3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec1ed500 con 0x7ff6ec1614f0 2024-05-08T21:22:44.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.026+0000 7ff6eb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6ec0d0650 0x7ff6ec1485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.026+0000 7ff6eb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6ec0d0650 0x7ff6ec1485c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59942/0 (socket says 172.21.15.3:59942) 2024-05-08T21:22:44.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.026+0000 7ff6eb7fe700 1 -- 172.21.15.3:0/78640581 learned_addr learned my addr 172.21.15.3:0/78640581 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:44.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.027+0000 7ff6e2ffd700 1 --2- 172.21.15.3:0/78640581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6ec15d0a0 0x7ff6ec1ecfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.027+0000 7ff6ebfff700 1 --2- 172.21.15.3:0/78640581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6ec1614f0 0x7ff6ec15dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.027+0000 7ff6eb7fe700 1 -- 172.21.15.3:0/78640581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6ec1614f0 msgr2=0x7ff6ec15dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:44.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.027+0000 7ff6eb7fe700 1 --2- 172.21.15.3:0/78640581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6ec1614f0 0x7ff6ec15dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.027+0000 7ff6eb7fe700 1 -- 172.21.15.3:0/78640581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6ec15d0a0 msgr2=0x7ff6ec1ecfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:44.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.027+0000 7ff6eb7fe700 1 --2- 172.21.15.3:0/78640581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6ec15d0a0 0x7ff6ec1ecfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.027+0000 7ff6eb7fe700 1 -- 172.21.15.3:0/78640581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc0094a0 con 0x7ff6ec0d0650 2024-05-08T21:22:44.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.027+0000 7ff6eb7fe700 1 --2- 172.21.15.3:0/78640581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6ec0d0650 0x7ff6ec1485c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00ebe0 tx=0x7ff6dc00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:44.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.027+0000 7ff6e8ff9700 1 -- 172.21.15.3:0/78640581 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc009760 con 0x7ff6ec0d0650 2024-05-08T21:22:44.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.027+0000 7ff6e8ff9700 1 -- 172.21.15.3:0/78640581 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc00c850 con 0x7ff6ec0d0650 2024-05-08T21:22:44.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.027+0000 7ff6f1cc3700 1 -- 172.21.15.3:0/78640581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d40094a0 con 0x7ff6ec0d0650 2024-05-08T21:22:44.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.027+0000 7ff6e8ff9700 1 -- 172.21.15.3:0/78640581 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc00cbe0 con 0x7ff6ec0d0650 2024-05-08T21:22:44.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.027+0000 7ff6f1cc3700 1 -- 172.21.15.3:0/78640581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6ec0c6250 con 0x7ff6ec0d0650 2024-05-08T21:22:44.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.028+0000 7ff6f1cc3700 1 -- 172.21.15.3:0/78640581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6ec068000 con 0x7ff6ec0d0650 2024-05-08T21:22:44.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.029+0000 7ff6e8ff9700 1 -- 172.21.15.3:0/78640581 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff6dc00cd40 con 0x7ff6ec0d0650 2024-05-08T21:22:44.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.029+0000 7ff6e8ff9700 1 --2- 172.21.15.3:0/78640581 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6cc06ac90 0x7ff6cc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.029+0000 7ff6e8ff9700 1 -- 172.21.15.3:0/78640581 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff6dc089030 con 0x7ff6ec0d0650 2024-05-08T21:22:44.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.029+0000 7ff6e2ffd700 1 --2- 172.21.15.3:0/78640581 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6cc06ac90 0x7ff6cc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.030+0000 7ff6e2ffd700 1 --2- 172.21.15.3:0/78640581 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6cc06ac90 0x7ff6cc06d150 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009fd0 tx=0x7ff6d401c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:44.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.034+0000 7ff6e8ff9700 1 -- 172.21.15.3:0/78640581 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff6dc0bc900 con 0x7ff6ec0d0650 2024-05-08T21:22:44.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.341+0000 7ff6f1cc3700 1 -- 172.21.15.3:0/78640581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7ff6ec15c2a0 con 0x7ff6ec0d0650 2024-05-08T21:22:44.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.345+0000 7ff6e8ff9700 1 -- 172.21.15.3:0/78640581 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff6dc0544c0 con 0x7ff6ec0d0650 2024-05-08T21:22:44.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.347+0000 7ff6f1cc3700 1 -- 172.21.15.3:0/78640581 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6cc06ac90 msgr2=0x7ff6cc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:44.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.347+0000 7ff6f1cc3700 1 --2- 172.21.15.3:0/78640581 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6cc06ac90 0x7ff6cc06d150 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009fd0 tx=0x7ff6d401c040 comp rx=0 tx=0).stop 2024-05-08T21:22:44.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.347+0000 7ff6f1cc3700 1 -- 172.21.15.3:0/78640581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6ec0d0650 msgr2=0x7ff6ec1485c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:44.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.347+0000 7ff6f1cc3700 1 --2- 172.21.15.3:0/78640581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6ec0d0650 0x7ff6ec1485c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00ebe0 tx=0x7ff6dc00eef0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.347+0000 7ff6f1cc3700 1 -- 172.21.15.3:0/78640581 shutdown_connections 2024-05-08T21:22:44.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.347+0000 7ff6f1cc3700 1 --2- 172.21.15.3:0/78640581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6ec1614f0 0x7ff6ec15dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.347+0000 7ff6f1cc3700 1 --2- 172.21.15.3:0/78640581 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6cc06ac90 0x7ff6cc06d150 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.347+0000 7ff6f1cc3700 1 --2- 172.21.15.3:0/78640581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6ec0d0650 0x7ff6ec1485c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.347+0000 7ff6f1cc3700 1 --2- 172.21.15.3:0/78640581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6ec15d0a0 0x7ff6ec1ecfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.347+0000 7ff6f1cc3700 1 -- 172.21.15.3:0/78640581 >> 172.21.15.3:0/78640581 conn(0x7ff6ec0b2f20 msgr2=0x7ff6ec15b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:44.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.347+0000 7ff6f1cc3700 1 -- 172.21.15.3:0/78640581 shutdown_connections 2024-05-08T21:22:44.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.347+0000 7ff6f1cc3700 1 -- 172.21.15.3:0/78640581 wait complete. 2024-05-08T21:22:44.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-05-08T21:22:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.497+0000 7f7df6484700 1 Processor -- start 2024-05-08T21:22:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.497+0000 7f7df6484700 1 -- start start 2024-05-08T21:22:44.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.498+0000 7f7df6484700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7df00cd4e0 0x7f7df006f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.498+0000 7f7df6484700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7df00cecf0 0x7f7df015c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.498+0000 7f7df6484700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df015c280 con 0x7f7df006f130 2024-05-08T21:22:44.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.498+0000 7f7df6484700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7df006e780 con 0x7f7df00cd4e0 2024-05-08T21:22:44.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.498+0000 7f7df6484700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7df006ea60 con 0x7f7df00cecf0 2024-05-08T21:22:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.498+0000 7f7deffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7df00cd4e0 0x7f7df006f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.498+0000 7f7deffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7df00cd4e0 0x7f7df006f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59950/0 (socket says 172.21.15.3:59950) 2024-05-08T21:22:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.498+0000 7f7deffff700 1 -- 172.21.15.3:0/2626335784 learned_addr learned my addr 172.21.15.3:0/2626335784 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.498+0000 7f7df4a21700 1 --2- 172.21.15.3:0/2626335784 >> v2:172.21.15.186:3300/0 conn(0x7f7df00cecf0 0x7f7df015c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.499+0000 7f7deffff700 1 -- 172.21.15.3:0/2626335784 >> v2:172.21.15.186:3300/0 conn(0x7f7df00cecf0 msgr2=0x7f7df015c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.499+0000 7f7deffff700 1 --2- 172.21.15.3:0/2626335784 >> v2:172.21.15.186:3300/0 conn(0x7f7df00cecf0 0x7f7df015c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.499+0000 7f7deffff700 1 -- 172.21.15.3:0/2626335784 >> v1:172.21.15.186:6789/0 conn(0x7f7df006f130 legacy=0x7f7df00ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.499+0000 7f7deffff700 1 -- 172.21.15.3:0/2626335784 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df006ed40 con 0x7f7df00cd4e0 2024-05-08T21:22:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.499+0000 7f7deffff700 1 --2- 172.21.15.3:0/2626335784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7df00cd4e0 0x7f7df006f910 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7df006ca30 tx=0x7f7de0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e6a8be2d06406a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.499+0000 7f7deeffd700 1 -- 172.21.15.3:0/2626335784 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de000b870 con 0x7f7df00cd4e0 2024-05-08T21:22:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.499+0000 7f7deeffd700 1 -- 172.21.15.3:0/2626335784 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de000b9d0 con 0x7f7df00cd4e0 2024-05-08T21:22:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.499+0000 7f7deeffd700 1 -- 172.21.15.3:0/2626335784 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de000bd60 con 0x7f7df00cd4e0 2024-05-08T21:22:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.499+0000 7f7df6484700 1 -- 172.21.15.3:0/2626335784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7df00cd4e0 msgr2=0x7f7df006f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.499+0000 7f7df6484700 1 --2- 172.21.15.3:0/2626335784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7df00cd4e0 0x7f7df006f910 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7df006ca30 tx=0x7f7de0009bd0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.500+0000 7f7df6484700 1 -- 172.21.15.3:0/2626335784 shutdown_connections 2024-05-08T21:22:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.500+0000 7f7df6484700 1 --2- 172.21.15.3:0/2626335784 >> v2:172.21.15.186:3300/0 conn(0x7f7df00cecf0 0x7f7df015c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.500+0000 7f7df6484700 1 --2- 172.21.15.3:0/2626335784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7df00cd4e0 0x7f7df006f910 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.500+0000 7f7df6484700 1 -- 172.21.15.3:0/2626335784 >> 172.21.15.3:0/2626335784 conn(0x7f7df00d0320 msgr2=0x7f7df00d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.500+0000 7f7df6484700 1 -- 172.21.15.3:0/2626335784 shutdown_connections 2024-05-08T21:22:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.500+0000 7f7df6484700 1 -- 172.21.15.3:0/2626335784 wait complete. 2024-05-08T21:22:44.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.501+0000 7f7df6484700 1 Processor -- start 2024-05-08T21:22:44.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.501+0000 7f7df6484700 1 -- start start 2024-05-08T21:22:44.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7df6484700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7df006f130 0x7f7df0167b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7df6484700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7df00cd4e0 0x7f7df0154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7df6484700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7df00cecf0 0x7f7df015b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7df6484700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df01680c0 con 0x7f7df00cd4e0 2024-05-08T21:22:44.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7df6484700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df01674b0 con 0x7f7df006f130 2024-05-08T21:22:44.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7df6484700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df0151650 con 0x7f7df00cecf0 2024-05-08T21:22:44.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7def7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7df00cd4e0 0x7f7df0154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7def7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7df00cd4e0 0x7f7df0154330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59964/0 (socket says 172.21.15.3:59964) 2024-05-08T21:22:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7def7fe700 1 -- 172.21.15.3:0/3710923167 learned_addr learned my addr 172.21.15.3:0/3710923167 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7deffff700 1 --2- 172.21.15.3:0/3710923167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7df006f130 0x7f7df0167b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7df4a21700 1 --2- 172.21.15.3:0/3710923167 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7df00cecf0 0x7f7df015b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7def7fe700 1 -- 172.21.15.3:0/3710923167 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7df00cecf0 msgr2=0x7f7df015b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7def7fe700 1 --2- 172.21.15.3:0/3710923167 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7df00cecf0 0x7f7df015b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7def7fe700 1 -- 172.21.15.3:0/3710923167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7df006f130 msgr2=0x7f7df0167b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7def7fe700 1 --2- 172.21.15.3:0/3710923167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7df006f130 0x7f7df0167b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7def7fe700 1 -- 172.21.15.3:0/3710923167 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de0009550 con 0x7f7df00cd4e0 2024-05-08T21:22:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.502+0000 7f7deffff700 1 --2- 172.21.15.3:0/3710923167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7df006f130 0x7f7df0167b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.503+0000 7f7def7fe700 1 --2- 172.21.15.3:0/3710923167 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7df00cd4e0 0x7f7df0154330 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7de400c4e0 tx=0x7f7de400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.503+0000 7f7decff9700 1 -- 172.21.15.3:0/3710923167 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de40052d0 con 0x7f7df00cd4e0 2024-05-08T21:22:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.503+0000 7f7decff9700 1 -- 172.21.15.3:0/3710923167 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de4005430 con 0x7f7df00cd4e0 2024-05-08T21:22:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.503+0000 7f7decff9700 1 -- 172.21.15.3:0/3710923167 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de40038c0 con 0x7f7df00cd4e0 2024-05-08T21:22:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.503+0000 7f7df6484700 1 -- 172.21.15.3:0/3710923167 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df0167940 con 0x7f7df00cd4e0 2024-05-08T21:22:44.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.503+0000 7f7df6484700 1 -- 172.21.15.3:0/3710923167 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df01518a0 con 0x7f7df00cd4e0 2024-05-08T21:22:44.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.504+0000 7f7df6484700 1 -- 172.21.15.3:0/3710923167 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df0068000 con 0x7f7df00cd4e0 2024-05-08T21:22:44.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.504+0000 7f7decff9700 1 -- 172.21.15.3:0/3710923167 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7de4003a20 con 0x7f7df00cd4e0 2024-05-08T21:22:44.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.505+0000 7f7decff9700 1 --2- 172.21.15.3:0/3710923167 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7ddc06ac90 0x7f7ddc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.505+0000 7f7decff9700 1 -- 172.21.15.3:0/3710923167 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7de40884a0 con 0x7f7df00cd4e0 2024-05-08T21:22:44.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.505+0000 7f7deffff700 1 --2- 172.21.15.3:0/3710923167 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7ddc06ac90 0x7f7ddc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.505+0000 7f7deffff700 1 --2- 172.21.15.3:0/3710923167 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7ddc06ac90 0x7f7ddc06d150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7de0000c00 tx=0x7f7de0008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:44.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.509+0000 7f7decff9700 1 -- 172.21.15.3:0/3710923167 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f7de40539d0 con 0x7f7df00cd4e0 2024-05-08T21:22:44.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.820+0000 7f7df6484700 1 -- 172.21.15.3:0/3710923167 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f7df006d440 con 0x7f7df00cd4e0 2024-05-08T21:22:44.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.826+0000 7f7decff9700 1 -- 172.21.15.3:0/3710923167 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f7de4056ff0 con 0x7f7df00cd4e0 2024-05-08T21:22:44.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.828+0000 7f7df6484700 1 -- 172.21.15.3:0/3710923167 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7ddc06ac90 msgr2=0x7f7ddc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:44.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.828+0000 7f7df6484700 1 --2- 172.21.15.3:0/3710923167 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7ddc06ac90 0x7f7ddc06d150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7de0000c00 tx=0x7f7de0008030 comp rx=0 tx=0).stop 2024-05-08T21:22:44.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.828+0000 7f7df6484700 1 -- 172.21.15.3:0/3710923167 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7df00cd4e0 msgr2=0x7f7df0154330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:44.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.828+0000 7f7df6484700 1 --2- 172.21.15.3:0/3710923167 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7df00cd4e0 0x7f7df0154330 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7de400c4e0 tx=0x7f7de400c8a0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.828+0000 7f7df6484700 1 -- 172.21.15.3:0/3710923167 shutdown_connections 2024-05-08T21:22:44.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.828+0000 7f7df6484700 1 --2- 172.21.15.3:0/3710923167 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7df00cecf0 0x7f7df015b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.828+0000 7f7df6484700 1 --2- 172.21.15.3:0/3710923167 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7ddc06ac90 0x7f7ddc06d150 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.828+0000 7f7df6484700 1 --2- 172.21.15.3:0/3710923167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7df006f130 0x7f7df0167b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.828+0000 7f7df6484700 1 --2- 172.21.15.3:0/3710923167 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7df00cd4e0 0x7f7df0154330 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.828+0000 7f7df6484700 1 -- 172.21.15.3:0/3710923167 >> 172.21.15.3:0/3710923167 conn(0x7f7df00d0320 msgr2=0x7f7df00cded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:44.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.829+0000 7f7df6484700 1 -- 172.21.15.3:0/3710923167 shutdown_connections 2024-05-08T21:22:44.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.829+0000 7f7df6484700 1 -- 172.21.15.3:0/3710923167 wait complete. 2024-05-08T21:22:44.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-05-08T21:22:44.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.979+0000 7f3ab12a1700 1 Processor -- start 2024-05-08T21:22:44.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.980+0000 7f3ab12a1700 1 -- start start 2024-05-08T21:22:44.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.980+0000 7f3ab12a1700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3aac1559f0 0x7f3aac06cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.980+0000 7f3ab12a1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac155dd0 0x7f3aac0ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.980+0000 7f3ab12a1700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aac15bf20 con 0x7f3aac0ae7f0 2024-05-08T21:22:44.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.980+0000 7f3ab12a1700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aac15c200 con 0x7f3aac1559f0 2024-05-08T21:22:44.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.980+0000 7f3ab12a1700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac15c4e0 con 0x7f3aac155dd0 2024-05-08T21:22:44.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.981+0000 7f3aab7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3aac0ae7f0 0x7f3aac06ad80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34610/0 (socket says 172.21.15.3:34610) 2024-05-08T21:22:44.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.981+0000 7f3aab7fe700 1 -- 172.21.15.3:0/856967722 learned_addr learned my addr 172.21.15.3:0/856967722 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:44.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.981+0000 7f3aaa7fc700 1 --2- 172.21.15.3:0/856967722 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac155dd0 0x7f3aac0ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.981+0000 7f3aaaffd700 1 --2- 172.21.15.3:0/856967722 >> v2:172.21.15.186:3300/0 conn(0x7f3aac1559f0 0x7f3aac06cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.981+0000 7f3aa9ffb700 1 -- 172.21.15.3:0/856967722 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508496769 0 0) 0x7f3aac15bf20 con 0x7f3aac0ae7f0 2024-05-08T21:22:44.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.981+0000 7f3aa9ffb700 1 -- 172.21.15.3:0/856967722 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a940035b0 con 0x7f3aac0ae7f0 2024-05-08T21:22:44.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.981+0000 7f3aaa7fc700 1 -- 172.21.15.3:0/856967722 >> v2:172.21.15.186:3300/0 conn(0x7f3aac1559f0 msgr2=0x7f3aac06cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:44.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.981+0000 7f3aaa7fc700 1 --2- 172.21.15.3:0/856967722 >> v2:172.21.15.186:3300/0 conn(0x7f3aac1559f0 0x7f3aac06cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.981+0000 7f3aaa7fc700 1 -- 172.21.15.3:0/856967722 >> v1:172.21.15.3:6789/0 conn(0x7f3aac0ae7f0 legacy=0x7f3aac06ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:44.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.981+0000 7f3aaa7fc700 1 -- 172.21.15.3:0/856967722 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac15c7c0 con 0x7f3aac155dd0 2024-05-08T21:22:44.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.981+0000 7f3aaaffd700 1 --2- 172.21.15.3:0/856967722 >> v2:172.21.15.186:3300/0 conn(0x7f3aac1559f0 0x7f3aac06cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:44.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.981+0000 7f3aaa7fc700 1 --2- 172.21.15.3:0/856967722 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac155dd0 0x7f3aac0ae2b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009840 tx=0x7f3aa0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ae31b225c8ad97b server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:44.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.982+0000 7f3aa9ffb700 1 -- 172.21.15.3:0/856967722 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0011830 con 0x7f3aac155dd0 2024-05-08T21:22:44.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.982+0000 7f3aa9ffb700 1 -- 172.21.15.3:0/856967722 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa0011990 con 0x7f3aac155dd0 2024-05-08T21:22:44.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.982+0000 7f3aa9ffb700 1 -- 172.21.15.3:0/856967722 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0011d20 con 0x7f3aac155dd0 2024-05-08T21:22:44.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.982+0000 7f3ab12a1700 1 -- 172.21.15.3:0/856967722 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac155dd0 msgr2=0x7f3aac0ae2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:44.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.982+0000 7f3ab12a1700 1 --2- 172.21.15.3:0/856967722 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac155dd0 0x7f3aac0ae2b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009840 tx=0x7f3aa0009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:44.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.982+0000 7f3ab12a1700 1 -- 172.21.15.3:0/856967722 shutdown_connections 2024-05-08T21:22:44.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.982+0000 7f3ab12a1700 1 --2- 172.21.15.3:0/856967722 >> v2:172.21.15.186:3300/0 conn(0x7f3aac1559f0 0x7f3aac06cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.982+0000 7f3ab12a1700 1 --2- 172.21.15.3:0/856967722 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac155dd0 0x7f3aac0ae2b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.982+0000 7f3ab12a1700 1 -- 172.21.15.3:0/856967722 >> 172.21.15.3:0/856967722 conn(0x7f3aac0b1290 msgr2=0x7f3aac0b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:44.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.982+0000 7f3ab12a1700 1 -- 172.21.15.3:0/856967722 shutdown_connections 2024-05-08T21:22:44.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.982+0000 7f3ab12a1700 1 -- 172.21.15.3:0/856967722 wait complete. 2024-05-08T21:22:44.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.984+0000 7f3ab12a1700 1 Processor -- start 2024-05-08T21:22:44.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.984+0000 7f3ab12a1700 1 -- start start 2024-05-08T21:22:44.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3ab12a1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac0ae7f0 0x7f3aac1f4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3ab12a1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3aac1559f0 0x7f3aac14f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3ab12a1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac155dd0 0x7f3aac0cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3ab12a1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac0cb580 con 0x7f3aac1559f0 2024-05-08T21:22:44.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3ab12a1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac1f4cf0 con 0x7f3aac0ae7f0 2024-05-08T21:22:44.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3ab12a1700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac1f4fd0 con 0x7f3aac155dd0 2024-05-08T21:22:44.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3aaa7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3aac1559f0 0x7f3aac14f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3aaa7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3aac1559f0 0x7f3aac14f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59974/0 (socket says 172.21.15.3:59974) 2024-05-08T21:22:44.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3aaa7fc700 1 -- 172.21.15.3:0/2392466768 learned_addr learned my addr 172.21.15.3:0/2392466768 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:44.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3aaaffd700 1 --2- 172.21.15.3:0/2392466768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac0ae7f0 0x7f3aac1f4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3aab7fe700 1 --2- 172.21.15.3:0/2392466768 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac155dd0 0x7f3aac0cb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3aaa7fc700 1 -- 172.21.15.3:0/2392466768 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac155dd0 msgr2=0x7f3aac0cb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:44.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3aaa7fc700 1 --2- 172.21.15.3:0/2392466768 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac155dd0 0x7f3aac0cb7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3aaa7fc700 1 -- 172.21.15.3:0/2392466768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac0ae7f0 msgr2=0x7f3aac1f4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:44.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3aaa7fc700 1 --2- 172.21.15.3:0/2392466768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac0ae7f0 0x7f3aac1f4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:44.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3aaa7fc700 1 -- 172.21.15.3:0/2392466768 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9c0094a0 con 0x7f3aac1559f0 2024-05-08T21:22:44.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3aaaffd700 1 --2- 172.21.15.3:0/2392466768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac0ae7f0 0x7f3aac1f4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:44.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3aaa7fc700 1 --2- 172.21.15.3:0/2392466768 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3aac1559f0 0x7f3aac14f770 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009fd0 tx=0x7f3aa0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:44.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.985+0000 7f3a937fe700 1 -- 172.21.15.3:0/2392466768 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa00036a0 con 0x7f3aac1559f0 2024-05-08T21:22:44.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.986+0000 7f3a937fe700 1 -- 172.21.15.3:0/2392466768 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa001e070 con 0x7f3aac1559f0 2024-05-08T21:22:44.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.986+0000 7f3ab12a1700 1 -- 172.21.15.3:0/2392466768 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa00094a0 con 0x7f3aac1559f0 2024-05-08T21:22:44.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.986+0000 7f3a937fe700 1 -- 172.21.15.3:0/2392466768 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0003a60 con 0x7f3aac1559f0 2024-05-08T21:22:44.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.986+0000 7f3ab12a1700 1 -- 172.21.15.3:0/2392466768 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aac153180 con 0x7f3aac1559f0 2024-05-08T21:22:44.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.987+0000 7f3ab12a1700 1 -- 172.21.15.3:0/2392466768 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aac152fe0 con 0x7f3aac1559f0 2024-05-08T21:22:44.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.987+0000 7f3a937fe700 1 -- 172.21.15.3:0/2392466768 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3aa0003bc0 con 0x7f3aac1559f0 2024-05-08T21:22:44.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.988+0000 7f3a937fe700 1 --2- 172.21.15.3:0/2392466768 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3a9806acc0 0x7f3a9806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:44.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.988+0000 7f3a937fe700 1 -- 172.21.15.3:0/2392466768 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3aa0089e00 con 0x7f3aac1559f0 2024-05-08T21:22:44.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.988+0000 7f3aaaffd700 1 --2- 172.21.15.3:0/2392466768 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3a9806acc0 0x7f3a9806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:44.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.988+0000 7f3aaaffd700 1 --2- 172.21.15.3:0/2392466768 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3a9806acc0 0x7f3a9806d180 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c009dc0 tx=0x7f3a9c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:44.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:44.992+0000 7f3a937fe700 1 -- 172.21.15.3:0/2392466768 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f3aac152fe0 con 0x7f3aac1559f0 2024-05-08T21:22:45.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.304+0000 7f3ab12a1700 1 -- 172.21.15.3:0/2392466768 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f3aac152fe0 con 0x7f3aac1559f0 2024-05-08T21:22:45.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.308+0000 7f3a937fe700 1 -- 172.21.15.3:0/2392466768 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f3aac152fe0 con 0x7f3aac1559f0 2024-05-08T21:22:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.311+0000 7f3ab12a1700 1 -- 172.21.15.3:0/2392466768 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3a9806acc0 msgr2=0x7f3a9806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.311+0000 7f3ab12a1700 1 --2- 172.21.15.3:0/2392466768 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3a9806acc0 0x7f3a9806d180 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c009dc0 tx=0x7f3a9c008040 comp rx=0 tx=0).stop 2024-05-08T21:22:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.311+0000 7f3ab12a1700 1 -- 172.21.15.3:0/2392466768 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3aac1559f0 msgr2=0x7f3aac14f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.311+0000 7f3ab12a1700 1 --2- 172.21.15.3:0/2392466768 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3aac1559f0 0x7f3aac14f770 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009fd0 tx=0x7f3aa0004750 comp rx=0 tx=0).stop 2024-05-08T21:22:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.311+0000 7f3ab12a1700 1 -- 172.21.15.3:0/2392466768 shutdown_connections 2024-05-08T21:22:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.311+0000 7f3ab12a1700 1 --2- 172.21.15.3:0/2392466768 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac155dd0 0x7f3aac0cb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.311+0000 7f3ab12a1700 1 --2- 172.21.15.3:0/2392466768 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3a9806acc0 0x7f3a9806d180 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.311+0000 7f3ab12a1700 1 --2- 172.21.15.3:0/2392466768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac0ae7f0 0x7f3aac1f4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.311+0000 7f3ab12a1700 1 --2- 172.21.15.3:0/2392466768 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3aac1559f0 0x7f3aac14f770 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.311+0000 7f3ab12a1700 1 -- 172.21.15.3:0/2392466768 >> 172.21.15.3:0/2392466768 conn(0x7f3aac0b1290 msgr2=0x7f3aac0ac360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.311+0000 7f3ab12a1700 1 -- 172.21.15.3:0/2392466768 shutdown_connections 2024-05-08T21:22:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.311+0000 7f3ab12a1700 1 -- 172.21.15.3:0/2392466768 wait complete. 2024-05-08T21:22:45.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-05-08T21:22:45.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.462+0000 7fc4a739e700 1 Processor -- start 2024-05-08T21:22:45.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.462+0000 7fc4a739e700 1 -- start start 2024-05-08T21:22:45.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.463+0000 7fc4a739e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc4a015eb60 0x7fc4a00b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:45.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a739e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a00b50a0 con 0x7fc4a015e6b0 2024-05-08T21:22:45.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a739e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a006fb50 con 0x7fc4a00b3dc0 2024-05-08T21:22:45.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a739e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a00b48d0 con 0x7fc4a015eb60 2024-05-08T21:22:45.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a5b9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc4a015e6b0 0x7fc4a0150be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34616/0 (socket says 172.21.15.3:34616) 2024-05-08T21:22:45.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a5b9b700 1 -- 172.21.15.3:0/2275331266 learned_addr learned my addr 172.21.15.3:0/2275331266 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:45.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a6b9d700 1 --2- 172.21.15.3:0/2275331266 >> v2:172.21.15.186:3300/0 conn(0x7fc4a015eb60 0x7fc4a00b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:45.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a539a700 1 -- 172.21.15.3:0/2275331266 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456335707 0 0) 0x7fc4a00b50a0 con 0x7fc4a015e6b0 2024-05-08T21:22:45.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a539a700 1 -- 172.21.15.3:0/2275331266 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc48c0035b0 con 0x7fc4a015e6b0 2024-05-08T21:22:45.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a539a700 1 -- 172.21.15.3:0/2275331266 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2054747266 0 0) 0x7fc48c0035b0 con 0x7fc4a015e6b0 2024-05-08T21:22:45.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a539a700 1 -- 172.21.15.3:0/2275331266 >> v2:172.21.15.186:3300/0 conn(0x7fc4a015eb60 msgr2=0x7fc4a00b41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:45.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a539a700 1 --2- 172.21.15.3:0/2275331266 >> v2:172.21.15.186:3300/0 conn(0x7fc4a015eb60 0x7fc4a00b41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a539a700 1 -- 172.21.15.3:0/2275331266 >> v1:172.21.15.186:6789/0 conn(0x7fc4a00b3dc0 legacy=0x7fc4a015dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:45.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a539a700 1 -- 172.21.15.3:0/2275331266 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a00b4ae0 con 0x7fc4a015e6b0 2024-05-08T21:22:45.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a539a700 1 -- 172.21.15.3:0/2275331266 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7fc490001670 con 0x7fc4a015e6b0 2024-05-08T21:22:45.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a539a700 1 -- 172.21.15.3:0/2275331266 >> v1:172.21.15.3:6789/0 conn(0x7fc4a015e6b0 legacy=0x7fc4a0150be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:45.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a539a700 1 --2- 172.21.15.3:0/2275331266 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc48c003a70 0x7fc48c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:45.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a539a700 1 --2- 172.21.15.3:0/2275331266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc48c004850 0x7fc48c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:45.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a539a700 1 --2- 172.21.15.3:0/2275331266 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc48c005210 0x7fc48c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:45.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a539a700 1 -- 172.21.15.3:0/2275331266 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a00b48d0 con 0x7fc48c003a70 2024-05-08T21:22:45.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a539a700 1 -- 172.21.15.3:0/2275331266 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c00c140 con 0x7fc48c004850 2024-05-08T21:22:45.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.464+0000 7fc4a539a700 1 -- 172.21.15.3:0/2275331266 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c00c300 con 0x7fc48c005210 2024-05-08T21:22:45.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.465+0000 7fc4a639c700 1 --2- 172.21.15.3:0/2275331266 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc48c003a70 0x7fc48c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:45.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.465+0000 7fc4a5b9b700 1 --2- 172.21.15.3:0/2275331266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc48c004850 0x7fc48c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:45.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.465+0000 7fc4a6b9d700 1 --2- 172.21.15.3:0/2275331266 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc48c005210 0x7fc48c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:45.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.465+0000 7fc4a639c700 1 -- 172.21.15.3:0/2275331266 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc48c005210 msgr2=0x7fc48c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:45.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.465+0000 7fc4a639c700 1 --2- 172.21.15.3:0/2275331266 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc48c005210 0x7fc48c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.465+0000 7fc4a639c700 1 -- 172.21.15.3:0/2275331266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc48c004850 msgr2=0x7fc48c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:45.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.465+0000 7fc4a639c700 1 --2- 172.21.15.3:0/2275331266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc48c004850 0x7fc48c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.465+0000 7fc4a639c700 1 -- 172.21.15.3:0/2275331266 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c0035b0 con 0x7fc48c003a70 2024-05-08T21:22:45.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.465+0000 7fc4a6b9d700 1 --2- 172.21.15.3:0/2275331266 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc48c005210 0x7fc48c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:45.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.465+0000 7fc4a6b9d700 1 -- 172.21.15.3:0/2275331266 reap_dead start 2024-05-08T21:22:45.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.465+0000 7fc4a639c700 1 --2- 172.21.15.3:0/2275331266 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc48c003a70 0x7fc48c003e50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc49c001980 tx=0x7fc48c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495314c835a7ab9f server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:45.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.465+0000 7fc4a539a700 1 -- 172.21.15.3:0/2275331266 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c001420 con 0x7fc48c003a70 2024-05-08T21:22:45.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.465+0000 7fc4a539a700 1 -- 172.21.15.3:0/2275331266 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49c0052d0 con 0x7fc48c003a70 2024-05-08T21:22:45.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.465+0000 7fc4a539a700 1 -- 172.21.15.3:0/2275331266 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c003680 con 0x7fc48c003a70 2024-05-08T21:22:45.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.466+0000 7fc4a739e700 1 -- 172.21.15.3:0/2275331266 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc48c003a70 msgr2=0x7fc48c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:45.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.466+0000 7fc4a739e700 1 --2- 172.21.15.3:0/2275331266 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc48c003a70 0x7fc48c003e50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc49c001980 tx=0x7fc48c005880 comp rx=0 tx=0).stop 2024-05-08T21:22:45.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.466+0000 7fc4a739e700 1 -- 172.21.15.3:0/2275331266 shutdown_connections 2024-05-08T21:22:45.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.466+0000 7fc4a739e700 1 --2- 172.21.15.3:0/2275331266 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc48c003a70 0x7fc48c003e50 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.466+0000 7fc4a739e700 1 -- 172.21.15.3:0/2275331266 >> 172.21.15.3:0/2275331266 conn(0x7fc4a00cc5d0 msgr2=0x7fc4a00cc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:45.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.466+0000 7fc4a739e700 1 -- 172.21.15.3:0/2275331266 shutdown_connections 2024-05-08T21:22:45.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.466+0000 7fc4a739e700 1 -- 172.21.15.3:0/2275331266 wait complete. 2024-05-08T21:22:45.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.467+0000 7fc4a739e700 1 Processor -- start 2024-05-08T21:22:45.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.467+0000 7fc4a739e700 1 -- start start 2024-05-08T21:22:45.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.468+0000 7fc4a739e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc48c003a70 0x7fc4a014e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:45.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.468+0000 7fc4a739e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4a00ad120 0x7fc4a00b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:45.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.468+0000 7fc4a739e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a00b1060 0x7fc4a01514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:45.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.468+0000 7fc4a739e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a00b1510 con 0x7fc4a00b1060 2024-05-08T21:22:45.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.468+0000 7fc4a739e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a00ac360 con 0x7fc4a00ad120 2024-05-08T21:22:45.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.468+0000 7fc4a739e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a00ac640 con 0x7fc48c003a70 2024-05-08T21:22:45.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.468+0000 7fc4a6b9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a00b1060 0x7fc4a01514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:45.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.468+0000 7fc4a6b9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a00b1060 0x7fc4a01514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59986/0 (socket says 172.21.15.3:59986) 2024-05-08T21:22:45.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.468+0000 7fc4a6b9d700 1 -- 172.21.15.3:0/170825640 learned_addr learned my addr 172.21.15.3:0/170825640 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:45.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.469+0000 7fc4a5b9b700 1 --2- 172.21.15.3:0/170825640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4a00ad120 0x7fc4a00b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:45.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.469+0000 7fc4a639c700 1 --2- 172.21.15.3:0/170825640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc48c003a70 0x7fc4a014e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:45.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.469+0000 7fc4a6b9d700 1 -- 172.21.15.3:0/170825640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc48c003a70 msgr2=0x7fc4a014e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:45.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.469+0000 7fc4a6b9d700 1 --2- 172.21.15.3:0/170825640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc48c003a70 0x7fc4a014e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.469+0000 7fc4a6b9d700 1 -- 172.21.15.3:0/170825640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4a00ad120 msgr2=0x7fc4a00b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:45.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.469+0000 7fc4a6b9d700 1 --2- 172.21.15.3:0/170825640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4a00ad120 0x7fc4a00b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.469+0000 7fc4a6b9d700 1 -- 172.21.15.3:0/170825640 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a00ac920 con 0x7fc4a00b1060 2024-05-08T21:22:45.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.469+0000 7fc4a639c700 1 --2- 172.21.15.3:0/170825640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc48c003a70 0x7fc4a014e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:45.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.469+0000 7fc4a5b9b700 1 --2- 172.21.15.3:0/170825640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4a00ad120 0x7fc4a00b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:45.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.469+0000 7fc4a6b9d700 1 --2- 172.21.15.3:0/170825640 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a00b1060 0x7fc4a01514b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00cf5f0 tx=0x7fc4980104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:45.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.469+0000 7fc496ffd700 1 -- 172.21.15.3:0/170825640 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49801b070 con 0x7fc4a00b1060 2024-05-08T21:22:45.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.469+0000 7fc496ffd700 1 -- 172.21.15.3:0/170825640 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49800f040 con 0x7fc4a00b1060 2024-05-08T21:22:45.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.469+0000 7fc4a739e700 1 -- 172.21.15.3:0/170825640 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a00acc00 con 0x7fc4a00b1060 2024-05-08T21:22:45.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.469+0000 7fc496ffd700 1 -- 172.21.15.3:0/170825640 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49800e050 con 0x7fc4a00b1060 2024-05-08T21:22:45.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.469+0000 7fc4a739e700 1 -- 172.21.15.3:0/170825640 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a01519f0 con 0x7fc4a00b1060 2024-05-08T21:22:45.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.470+0000 7fc4a739e700 1 -- 172.21.15.3:0/170825640 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a0074b50 con 0x7fc4a00b1060 2024-05-08T21:22:45.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.471+0000 7fc496ffd700 1 -- 172.21.15.3:0/170825640 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc498003a20 con 0x7fc4a00b1060 2024-05-08T21:22:45.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.471+0000 7fc496ffd700 1 --2- 172.21.15.3:0/170825640 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc48406acc0 0x7fc48406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:45.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.471+0000 7fc496ffd700 1 -- 172.21.15.3:0/170825640 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc4980892b0 con 0x7fc4a00b1060 2024-05-08T21:22:45.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.471+0000 7fc4a639c700 1 --2- 172.21.15.3:0/170825640 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc48406acc0 0x7fc48406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:45.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.472+0000 7fc4a639c700 1 --2- 172.21.15.3:0/170825640 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc48406acc0 0x7fc48406d180 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc49c015040 tx=0x7fc49c00a000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:45.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.476+0000 7fc496ffd700 1 -- 172.21.15.3:0/170825640 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc498054750 con 0x7fc4a00b1060 2024-05-08T21:22:45.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.788+0000 7fc4a739e700 1 -- 172.21.15.3:0/170825640 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fc4a00aeec0 con 0x7fc4a00b1060 2024-05-08T21:22:45.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.792+0000 7fc496ffd700 1 -- 172.21.15.3:0/170825640 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc498057d70 con 0x7fc4a00b1060 2024-05-08T21:22:45.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.794+0000 7fc4a739e700 1 -- 172.21.15.3:0/170825640 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc48406acc0 msgr2=0x7fc48406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:45.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.794+0000 7fc4a739e700 1 --2- 172.21.15.3:0/170825640 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc48406acc0 0x7fc48406d180 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc49c015040 tx=0x7fc49c00a000 comp rx=0 tx=0).stop 2024-05-08T21:22:45.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.794+0000 7fc4a739e700 1 -- 172.21.15.3:0/170825640 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a00b1060 msgr2=0x7fc4a01514b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:45.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.794+0000 7fc4a739e700 1 --2- 172.21.15.3:0/170825640 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a00b1060 0x7fc4a01514b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00cf5f0 tx=0x7fc4980104a0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.794+0000 7fc4a739e700 1 -- 172.21.15.3:0/170825640 shutdown_connections 2024-05-08T21:22:45.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.794+0000 7fc4a739e700 1 --2- 172.21.15.3:0/170825640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc48c003a70 0x7fc4a014e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.794+0000 7fc4a739e700 1 --2- 172.21.15.3:0/170825640 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc48406acc0 0x7fc48406d180 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.794+0000 7fc4a739e700 1 --2- 172.21.15.3:0/170825640 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a00b1060 0x7fc4a01514b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.794+0000 7fc4a739e700 1 --2- 172.21.15.3:0/170825640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4a00ad120 0x7fc4a00b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.794+0000 7fc4a739e700 1 -- 172.21.15.3:0/170825640 >> 172.21.15.3:0/170825640 conn(0x7fc4a00cc5d0 msgr2=0x7fc4a015dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:45.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.794+0000 7fc4a739e700 1 -- 172.21.15.3:0/170825640 shutdown_connections 2024-05-08T21:22:45.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.794+0000 7fc4a739e700 1 -- 172.21.15.3:0/170825640 wait complete. 2024-05-08T21:22:45.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-05-08T21:22:45.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.944+0000 7f54cde3a700 1 Processor -- start 2024-05-08T21:22:45.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.945+0000 7f54cde3a700 1 -- start start 2024-05-08T21:22:45.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.946+0000 7f54cde3a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f54c815ba00 0x7f54c80b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:45.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.946+0000 7f54cde3a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c80b4280 con 0x7f54c80b2510 2024-05-08T21:22:45.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.946+0000 7f54cde3a700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c80b4560 con 0x7f54c809a160 2024-05-08T21:22:45.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.946+0000 7f54cde3a700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c80cccf0 con 0x7f54c815ba00 2024-05-08T21:22:45.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.946+0000 7f54c6ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f54c80b2510 0x7f54c8099a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34618/0 (socket says 172.21.15.3:34618) 2024-05-08T21:22:45.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.946+0000 7f54c6ffd700 1 -- 172.21.15.3:0/1465565726 learned_addr learned my addr 172.21.15.3:0/1465565726 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:45.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.946+0000 7f54c77fe700 1 --2- 172.21.15.3:0/1465565726 >> v2:172.21.15.186:3300/0 conn(0x7f54c815ba00 0x7f54c80b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:45.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.946+0000 7f54c67fc700 1 -- 172.21.15.3:0/1465565726 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262244153 0 0) 0x7f54c80b4280 con 0x7f54c80b2510 2024-05-08T21:22:45.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.946+0000 7f54c67fc700 1 -- 172.21.15.3:0/1465565726 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b00035b0 con 0x7f54c80b2510 2024-05-08T21:22:45.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.946+0000 7f54c67fc700 1 -- 172.21.15.3:0/1465565726 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561469380 0 0) 0x7f54c80b4560 con 0x7f54c809a160 2024-05-08T21:22:45.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.946+0000 7f54c67fc700 1 -- 172.21.15.3:0/1465565726 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c80b4280 con 0x7f54c809a160 2024-05-08T21:22:45.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.947+0000 7f54c77fe700 1 -- 172.21.15.3:0/1465565726 >> v1:172.21.15.186:6789/0 conn(0x7f54c809a160 legacy=0x7f54c80b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:45.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.947+0000 7f54c77fe700 1 -- 172.21.15.3:0/1465565726 >> v1:172.21.15.3:6789/0 conn(0x7f54c80b2510 legacy=0x7f54c8099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:45.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.947+0000 7f54c77fe700 1 -- 172.21.15.3:0/1465565726 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c80ccfd0 con 0x7f54c815ba00 2024-05-08T21:22:45.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.947+0000 7f54c77fe700 1 --2- 172.21.15.3:0/1465565726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54c815ba00 0x7f54c80b3a40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009840 tx=0x7f54b8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2209f2406dd76cb6 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:45.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.947+0000 7f54c67fc700 1 -- 172.21.15.3:0/1465565726 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8011830 con 0x7f54c815ba00 2024-05-08T21:22:45.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.947+0000 7f54c67fc700 1 -- 172.21.15.3:0/1465565726 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b8011990 con 0x7f54c815ba00 2024-05-08T21:22:45.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.947+0000 7f54c67fc700 1 -- 172.21.15.3:0/1465565726 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8011d20 con 0x7f54c815ba00 2024-05-08T21:22:45.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.948+0000 7f54cde3a700 1 -- 172.21.15.3:0/1465565726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54c815ba00 msgr2=0x7f54c80b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:45.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.948+0000 7f54cde3a700 1 --2- 172.21.15.3:0/1465565726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54c815ba00 0x7f54c80b3a40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009840 tx=0x7f54b8009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:45.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.948+0000 7f54cde3a700 1 -- 172.21.15.3:0/1465565726 shutdown_connections 2024-05-08T21:22:45.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.948+0000 7f54cde3a700 1 --2- 172.21.15.3:0/1465565726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54c815ba00 0x7f54c80b3a40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.948+0000 7f54cde3a700 1 -- 172.21.15.3:0/1465565726 >> 172.21.15.3:0/1465565726 conn(0x7f54c80b2f20 msgr2=0x7f54c80b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:45.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.948+0000 7f54cde3a700 1 -- 172.21.15.3:0/1465565726 shutdown_connections 2024-05-08T21:22:45.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.948+0000 7f54cde3a700 1 -- 172.21.15.3:0/1465565726 wait complete. 2024-05-08T21:22:45.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.949+0000 7f54cde3a700 1 Processor -- start 2024-05-08T21:22:45.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.950+0000 7f54cde3a700 1 -- start start 2024-05-08T21:22:45.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54cde3a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54c809a160 0x7f54c8159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:45.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54cde3a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54c80b2510 0x7f54c809beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:45.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54cde3a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54c815ba00 0x7f54c809ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:45.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54cde3a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c80b22e0 con 0x7f54c80b2510 2024-05-08T21:22:45.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54cde3a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c81f5250 con 0x7f54c815ba00 2024-05-08T21:22:45.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54cde3a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c809b0f0 con 0x7f54c809a160 2024-05-08T21:22:45.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54c6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54c80b2510 0x7f54c809beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:45.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54c6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54c80b2510 0x7f54c809beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60002/0 (socket says 172.21.15.3:60002) 2024-05-08T21:22:45.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54c6ffd700 1 -- 172.21.15.3:0/554368517 learned_addr learned my addr 172.21.15.3:0/554368517 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:45.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54c77fe700 1 --2- 172.21.15.3:0/554368517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54c809a160 0x7f54c8159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:45.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54c7fff700 1 --2- 172.21.15.3:0/554368517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54c815ba00 0x7f54c809ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:45.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54c6ffd700 1 -- 172.21.15.3:0/554368517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54c809a160 msgr2=0x7f54c8159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:45.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54c6ffd700 1 --2- 172.21.15.3:0/554368517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54c809a160 0x7f54c8159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54c6ffd700 1 -- 172.21.15.3:0/554368517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54c815ba00 msgr2=0x7f54c809ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:45.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54c6ffd700 1 --2- 172.21.15.3:0/554368517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54c815ba00 0x7f54c809ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:45.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54c6ffd700 1 -- 172.21.15.3:0/554368517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b80094a0 con 0x7f54c80b2510 2024-05-08T21:22:45.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54c6ffd700 1 --2- 172.21.15.3:0/554368517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54c80b2510 0x7f54c809beb0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00c700 tx=0x7f54bc00cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:45.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.951+0000 7f54affff700 1 -- 172.21.15.3:0/554368517 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc005d70 con 0x7f54c80b2510 2024-05-08T21:22:45.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.952+0000 7f54affff700 1 -- 172.21.15.3:0/554368517 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54bc0070d0 con 0x7f54c80b2510 2024-05-08T21:22:45.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.952+0000 7f54cde3a700 1 -- 172.21.15.3:0/554368517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c809b3d0 con 0x7f54c80b2510 2024-05-08T21:22:45.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.952+0000 7f54affff700 1 -- 172.21.15.3:0/554368517 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc007460 con 0x7f54c80b2510 2024-05-08T21:22:45.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.952+0000 7f54cde3a700 1 -- 172.21.15.3:0/554368517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c809b990 con 0x7f54c80b2510 2024-05-08T21:22:45.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.953+0000 7f54cde3a700 1 -- 172.21.15.3:0/554368517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c809bb80 con 0x7f54c80b2510 2024-05-08T21:22:45.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.953+0000 7f54affff700 1 -- 172.21.15.3:0/554368517 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f54bc0075c0 con 0x7f54c80b2510 2024-05-08T21:22:45.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.954+0000 7f54affff700 1 --2- 172.21.15.3:0/554368517 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f54b406ac90 0x7f54b406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:45.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.954+0000 7f54affff700 1 -- 172.21.15.3:0/554368517 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f54bc088660 con 0x7f54c80b2510 2024-05-08T21:22:45.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.954+0000 7f54c77fe700 1 --2- 172.21.15.3:0/554368517 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f54b406ac90 0x7f54b406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:45.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.954+0000 7f54c77fe700 1 --2- 172.21.15.3:0/554368517 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f54b406ac90 0x7f54b406d150 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009fd0 tx=0x7f54b801c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:45.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:45.958+0000 7f54affff700 1 -- 172.21.15.3:0/554368517 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f54bc053ba0 con 0x7f54c80b2510 2024-05-08T21:22:46.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.263+0000 7f54cde3a700 1 -- 172.21.15.3:0/554368517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f54c8068000 con 0x7f54c80b2510 2024-05-08T21:22:46.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.268+0000 7f54affff700 1 -- 172.21.15.3:0/554368517 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f54bc0571c0 con 0x7f54c80b2510 2024-05-08T21:22:46.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.271+0000 7f54cde3a700 1 -- 172.21.15.3:0/554368517 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f54b406ac90 msgr2=0x7f54b406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:46.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.271+0000 7f54cde3a700 1 --2- 172.21.15.3:0/554368517 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f54b406ac90 0x7f54b406d150 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009fd0 tx=0x7f54b801c040 comp rx=0 tx=0).stop 2024-05-08T21:22:46.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.271+0000 7f54cde3a700 1 -- 172.21.15.3:0/554368517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54c80b2510 msgr2=0x7f54c809beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:46.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.271+0000 7f54cde3a700 1 --2- 172.21.15.3:0/554368517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54c80b2510 0x7f54c809beb0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00c700 tx=0x7f54bc00cac0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.271+0000 7f54cde3a700 1 -- 172.21.15.3:0/554368517 shutdown_connections 2024-05-08T21:22:46.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.271+0000 7f54cde3a700 1 --2- 172.21.15.3:0/554368517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54c809a160 0x7f54c8159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.271+0000 7f54cde3a700 1 --2- 172.21.15.3:0/554368517 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f54b406ac90 0x7f54b406d150 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.271+0000 7f54cde3a700 1 --2- 172.21.15.3:0/554368517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54c815ba00 0x7f54c809ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.271+0000 7f54cde3a700 1 --2- 172.21.15.3:0/554368517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54c80b2510 0x7f54c809beb0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.271+0000 7f54cde3a700 1 -- 172.21.15.3:0/554368517 >> 172.21.15.3:0/554368517 conn(0x7f54c80b2f20 msgr2=0x7f54c80b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:46.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.271+0000 7f54cde3a700 1 -- 172.21.15.3:0/554368517 shutdown_connections 2024-05-08T21:22:46.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.271+0000 7f54cde3a700 1 -- 172.21.15.3:0/554368517 wait complete. 2024-05-08T21:22:46.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:46.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.420+0000 7f2cd45ac700 1 Processor -- start 2024-05-08T21:22:46.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.420+0000 7f2cd45ac700 1 -- start start 2024-05-08T21:22:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.421+0000 7f2cd45ac700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ccc15a630 0x7f2ccc0d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.421+0000 7f2cd45ac700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2ccc15e200 0x7f2ccc15e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.421+0000 7f2cd45ac700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ccc0ce740 con 0x7f2ccc0cdef0 2024-05-08T21:22:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.421+0000 7f2cd45ac700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ccc15ebd0 con 0x7f2ccc15e200 2024-05-08T21:22:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.421+0000 7f2cd45ac700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc0cb670 con 0x7f2ccc15a630 2024-05-08T21:22:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.421+0000 7f2cd1b47700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2ccc0cdef0 0x7f2ccc0cbb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34632/0 (socket says 172.21.15.3:34632) 2024-05-08T21:22:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.421+0000 7f2cd1b47700 1 -- 172.21.15.3:0/1263730491 learned_addr learned my addr 172.21.15.3:0/1263730491 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.421+0000 7f2cd2b49700 1 --2- 172.21.15.3:0/1263730491 >> v2:172.21.15.186:3300/0 conn(0x7f2ccc15e200 0x7f2ccc15e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd2348700 1 --2- 172.21.15.3:0/1263730491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ccc15a630 0x7f2ccc0d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1346700 1 -- 172.21.15.3:0/1263730491 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853960637 0 0) 0x7f2cbc001110 con 0x7f2ccc0cdef0 2024-05-08T21:22:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1346700 1 -- 172.21.15.3:0/1263730491 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb80035b0 con 0x7f2ccc0cdef0 2024-05-08T21:22:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1346700 1 -- 172.21.15.3:0/1263730491 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 602775018 0 0) 0x7f2cb80035b0 con 0x7f2ccc0cdef0 2024-05-08T21:22:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1346700 1 -- 172.21.15.3:0/1263730491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ccc15a630 msgr2=0x7f2ccc0d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1346700 1 --2- 172.21.15.3:0/1263730491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ccc15a630 0x7f2ccc0d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1346700 1 -- 172.21.15.3:0/1263730491 >> v2:172.21.15.186:3300/0 conn(0x7f2ccc15e200 msgr2=0x7f2ccc15e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1346700 1 --2- 172.21.15.3:0/1263730491 >> v2:172.21.15.186:3300/0 conn(0x7f2ccc15e200 0x7f2ccc15e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1346700 1 -- 172.21.15.3:0/1263730491 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ccc0cadc0 con 0x7f2ccc0cdef0 2024-05-08T21:22:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1346700 1 -- 172.21.15.3:0/1263730491 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f2cbc001800 con 0x7f2ccc0cdef0 2024-05-08T21:22:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1346700 1 -- 172.21.15.3:0/1263730491 >> v1:172.21.15.3:6789/0 conn(0x7f2ccc0cdef0 legacy=0x7f2ccc0cbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1346700 1 --2- 172.21.15.3:0/1263730491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb8003a00 0x7f2cb8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1346700 1 --2- 172.21.15.3:0/1263730491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb8004850 0x7f2cb8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1346700 1 --2- 172.21.15.3:0/1263730491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ccc15a630 0x7f2cb8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1346700 1 -- 172.21.15.3:0/1263730491 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc15ebd0 con 0x7f2cb8004850 2024-05-08T21:22:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1346700 1 -- 172.21.15.3:0/1263730491 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc0cb670 con 0x7f2cb8003a00 2024-05-08T21:22:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1346700 1 -- 172.21.15.3:0/1263730491 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb800ba10 con 0x7f2ccc15a630 2024-05-08T21:22:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1b47700 1 --2- 172.21.15.3:0/1263730491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb8004850 0x7f2cb8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd2348700 1 --2- 172.21.15.3:0/1263730491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb8003a00 0x7f2cb8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd2b49700 1 --2- 172.21.15.3:0/1263730491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ccc15a630 0x7f2cb8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1b47700 1 -- 172.21.15.3:0/1263730491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ccc15a630 msgr2=0x7f2cb8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1b47700 1 --2- 172.21.15.3:0/1263730491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ccc15a630 0x7f2cb8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1b47700 1 -- 172.21.15.3:0/1263730491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb8003a00 msgr2=0x7f2cb8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1b47700 1 --2- 172.21.15.3:0/1263730491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb8003a00 0x7f2cb8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd1b47700 1 -- 172.21.15.3:0/1263730491 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb80035b0 con 0x7f2cb8004850 2024-05-08T21:22:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.422+0000 7f2cd2b49700 1 --2- 172.21.15.3:0/1263730491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ccc15a630 0x7f2cb8005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.423+0000 7f2cd1b47700 1 --2- 172.21.15.3:0/1263730491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb8004850 0x7f2cb8004cd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc00c240 tx=0x7f2cb8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780b33cceec0a3a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.423+0000 7f2cd1346700 1 -- 172.21.15.3:0/1263730491 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc00cde0 con 0x7f2cb8004850 2024-05-08T21:22:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.423+0000 7f2cd1346700 1 -- 172.21.15.3:0/1263730491 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cbc005960 con 0x7f2cb8004850 2024-05-08T21:22:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.423+0000 7f2cd1346700 1 -- 172.21.15.3:0/1263730491 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc005d10 con 0x7f2cb8004850 2024-05-08T21:22:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.423+0000 7f2cd45ac700 1 -- 172.21.15.3:0/1263730491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb8004850 msgr2=0x7f2cb8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.423+0000 7f2cd45ac700 1 --2- 172.21.15.3:0/1263730491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb8004850 0x7f2cb8004cd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc00c240 tx=0x7f2cb8005880 comp rx=0 tx=0).stop 2024-05-08T21:22:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.423+0000 7f2cd2b49700 1 -- 172.21.15.3:0/1263730491 reap_dead start 2024-05-08T21:22:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.423+0000 7f2cd45ac700 1 -- 172.21.15.3:0/1263730491 shutdown_connections 2024-05-08T21:22:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.423+0000 7f2cd45ac700 1 -- 172.21.15.3:0/1263730491 >> 172.21.15.3:0/1263730491 conn(0x7f2ccc099a50 msgr2=0x7f2ccc0d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.423+0000 7f2cd45ac700 1 -- 172.21.15.3:0/1263730491 shutdown_connections 2024-05-08T21:22:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.423+0000 7f2cd45ac700 1 -- 172.21.15.3:0/1263730491 wait complete. 2024-05-08T21:22:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.425+0000 7f2cd45ac700 1 Processor -- start 2024-05-08T21:22:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.425+0000 7f2cd45ac700 1 -- start start 2024-05-08T21:22:46.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd45ac700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb8004850 0x7f2ccc15c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:46.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd45ac700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ccc1512c0 0x7f2ccc150ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:46.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd45ac700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ccc15b460 0x7f2ccc0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:46.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd45ac700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc15aa10 con 0x7f2ccc15b460 2024-05-08T21:22:46.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd45ac700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc15cf10 con 0x7f2cb8004850 2024-05-08T21:22:46.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd45ac700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc15d1f0 con 0x7f2ccc1512c0 2024-05-08T21:22:46.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd2b49700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ccc15b460 0x7f2ccc0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:46.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd2b49700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ccc15b460 0x7f2ccc0b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60016/0 (socket says 172.21.15.3:60016) 2024-05-08T21:22:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd2b49700 1 -- 172.21.15.3:0/3334884427 learned_addr learned my addr 172.21.15.3:0/3334884427 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd2348700 1 --2- 172.21.15.3:0/3334884427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb8004850 0x7f2ccc15c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd1b47700 1 --2- 172.21.15.3:0/3334884427 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ccc1512c0 0x7f2ccc150ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd2b49700 1 -- 172.21.15.3:0/3334884427 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ccc1512c0 msgr2=0x7f2ccc150ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd2b49700 1 --2- 172.21.15.3:0/3334884427 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ccc1512c0 0x7f2ccc150ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd2b49700 1 -- 172.21.15.3:0/3334884427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb8004850 msgr2=0x7f2ccc15c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd2b49700 1 --2- 172.21.15.3:0/3334884427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb8004850 0x7f2ccc15c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd2b49700 1 -- 172.21.15.3:0/3334884427 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ccc15ac60 con 0x7f2ccc15b460 2024-05-08T21:22:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cd2b49700 1 --2- 172.21.15.3:0/3334884427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ccc15b460 0x7f2ccc0b0b90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc15e5e0 tx=0x7f2cc40104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cc2ffd700 1 -- 172.21.15.3:0/3334884427 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc401b070 con 0x7f2ccc15b460 2024-05-08T21:22:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.426+0000 7f2cc2ffd700 1 -- 172.21.15.3:0/3334884427 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc400f040 con 0x7f2ccc15b460 2024-05-08T21:22:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.427+0000 7f2cd45ac700 1 -- 172.21.15.3:0/3334884427 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ccc15af40 con 0x7f2ccc15b460 2024-05-08T21:22:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.427+0000 7f2cc2ffd700 1 -- 172.21.15.3:0/3334884427 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc400e050 con 0x7f2ccc15b460 2024-05-08T21:22:46.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.427+0000 7f2cd45ac700 1 -- 172.21.15.3:0/3334884427 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ccc151020 con 0x7f2ccc15b460 2024-05-08T21:22:46.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.427+0000 7f2cd45ac700 1 -- 172.21.15.3:0/3334884427 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ccc151f80 con 0x7f2ccc15b460 2024-05-08T21:22:46.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.428+0000 7f2cc2ffd700 1 -- 172.21.15.3:0/3334884427 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2cc4003a20 con 0x7f2ccc15b460 2024-05-08T21:22:46.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.428+0000 7f2cc2ffd700 1 --2- 172.21.15.3:0/3334884427 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2cb006ac90 0x7f2cb006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:46.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.429+0000 7f2cc2ffd700 1 -- 172.21.15.3:0/3334884427 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2cc40893e0 con 0x7f2ccc15b460 2024-05-08T21:22:46.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.429+0000 7f2cd2348700 1 --2- 172.21.15.3:0/3334884427 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2cb006ac90 0x7f2cb006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:46.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.429+0000 7f2cd2348700 1 --2- 172.21.15.3:0/3334884427 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2cb006ac90 0x7f2cb006d150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800bda0 tx=0x7f2cc8013040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:46.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.433+0000 7f2cc2ffd700 1 -- 172.21.15.3:0/3334884427 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2cc4054880 con 0x7f2ccc15b460 2024-05-08T21:22:46.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.650+0000 7f2cd45ac700 1 -- 172.21.15.3:0/3334884427 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ccc068000 con 0x7f2ccc15b460 2024-05-08T21:22:46.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.651+0000 7f2cc2ffd700 1 -- 172.21.15.3:0/3334884427 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12341 (secure 0 0 0) 0x7f2cc4057ea0 con 0x7f2ccc15b460 2024-05-08T21:22:46.653 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:46.653 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":30,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:22:42.900200+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:46.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.654+0000 7f2cd45ac700 1 -- 172.21.15.3:0/3334884427 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2cb006ac90 msgr2=0x7f2cb006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:46.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.654+0000 7f2cd45ac700 1 --2- 172.21.15.3:0/3334884427 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2cb006ac90 0x7f2cb006d150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800bda0 tx=0x7f2cc8013040 comp rx=0 tx=0).stop 2024-05-08T21:22:46.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.654+0000 7f2cd45ac700 1 -- 172.21.15.3:0/3334884427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ccc15b460 msgr2=0x7f2ccc0b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:46.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.654+0000 7f2cd45ac700 1 --2- 172.21.15.3:0/3334884427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ccc15b460 0x7f2ccc0b0b90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc15e5e0 tx=0x7f2cc40104a0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.654+0000 7f2cd45ac700 1 -- 172.21.15.3:0/3334884427 shutdown_connections 2024-05-08T21:22:46.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.654+0000 7f2cd45ac700 1 --2- 172.21.15.3:0/3334884427 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2cb006ac90 0x7f2cb006d150 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.654+0000 7f2cd45ac700 1 --2- 172.21.15.3:0/3334884427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ccc15b460 0x7f2ccc0b0b90 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.654+0000 7f2cd45ac700 1 --2- 172.21.15.3:0/3334884427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb8004850 0x7f2ccc15c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.654+0000 7f2cd45ac700 1 --2- 172.21.15.3:0/3334884427 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ccc1512c0 0x7f2ccc150ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.654+0000 7f2cd45ac700 1 -- 172.21.15.3:0/3334884427 >> 172.21.15.3:0/3334884427 conn(0x7f2ccc099a50 msgr2=0x7f2ccc1e3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:46.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.654+0000 7f2cd45ac700 1 -- 172.21.15.3:0/3334884427 shutdown_connections 2024-05-08T21:22:46.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.654+0000 7f2cd45ac700 1 -- 172.21.15.3:0/3334884427 wait complete. 2024-05-08T21:22:46.669 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:22:46.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:22:46.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.807+0000 7efcfb921700 1 Processor -- start 2024-05-08T21:22:46.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.807+0000 7efcfb921700 1 -- start start 2024-05-08T21:22:46.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.808+0000 7efcfb921700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf415b4e0 0x7efcf40b1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:46.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.808+0000 7efcfb921700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf40cf560 con 0x7efcf40d1910 2024-05-08T21:22:46.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.808+0000 7efcfb921700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf40ccff0 con 0x7efcf415b8c0 2024-05-08T21:22:46.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.808+0000 7efcfb921700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf40cefe0 con 0x7efcf415b4e0 2024-05-08T21:22:46.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.808+0000 7efcf9ebe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7efcf40d1910 0x7efcf40cc2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34634/0 (socket says 172.21.15.3:34634) 2024-05-08T21:22:46.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.808+0000 7efcf9ebe700 1 -- 172.21.15.3:0/233053655 learned_addr learned my addr 172.21.15.3:0/233053655 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:46.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.808+0000 7efcf96bd700 1 --2- 172.21.15.3:0/233053655 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf415b4e0 0x7efcf40b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:46.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.809+0000 7efcebfff700 1 -- 172.21.15.3:0/233053655 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420751792 0 0) 0x7efcf40cf560 con 0x7efcf40d1910 2024-05-08T21:22:46.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.809+0000 7efcebfff700 1 -- 172.21.15.3:0/233053655 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efce00035b0 con 0x7efcf40d1910 2024-05-08T21:22:46.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.809+0000 7efcf96bd700 1 -- 172.21.15.3:0/233053655 >> v1:172.21.15.186:6789/0 conn(0x7efcf415b8c0 legacy=0x7efcf40d1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:46.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.809+0000 7efcf96bd700 1 -- 172.21.15.3:0/233053655 >> v1:172.21.15.3:6789/0 conn(0x7efcf40d1910 legacy=0x7efcf40cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:46.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.809+0000 7efcebfff700 1 -- 172.21.15.3:0/233053655 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3530882515 0 0) 0x7efce00035b0 con 0x7efcf40d1910 2024-05-08T21:22:46.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.809+0000 7efcf96bd700 1 -- 172.21.15.3:0/233053655 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf40cd2d0 con 0x7efcf415b4e0 2024-05-08T21:22:46.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.809+0000 7efcf96bd700 1 --2- 172.21.15.3:0/233053655 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf415b4e0 0x7efcf40b1200 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efcf0009870 tx=0x7efcf0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=367bf1beac895966 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:46.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.809+0000 7efcebfff700 1 -- 172.21.15.3:0/233053655 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec0019f0 con 0x7efcf415b4e0 2024-05-08T21:22:46.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.809+0000 7efcebfff700 1 -- 172.21.15.3:0/233053655 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf0004030 con 0x7efcf415b4e0 2024-05-08T21:22:46.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.809+0000 7efcebfff700 1 -- 172.21.15.3:0/233053655 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf00043c0 con 0x7efcf415b4e0 2024-05-08T21:22:46.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.810+0000 7efcfb921700 1 -- 172.21.15.3:0/233053655 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf415b4e0 msgr2=0x7efcf40b1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:46.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.810+0000 7efcfb921700 1 --2- 172.21.15.3:0/233053655 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf415b4e0 0x7efcf40b1200 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efcf0009870 tx=0x7efcf0009b80 comp rx=0 tx=0).stop 2024-05-08T21:22:46.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.810+0000 7efcfb921700 1 -- 172.21.15.3:0/233053655 shutdown_connections 2024-05-08T21:22:46.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.810+0000 7efcfb921700 1 --2- 172.21.15.3:0/233053655 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf415b4e0 0x7efcf40b1200 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.810+0000 7efcfb921700 1 -- 172.21.15.3:0/233053655 >> 172.21.15.3:0/233053655 conn(0x7efcf408afe0 msgr2=0x7efcf408b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:46.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.810+0000 7efcfb921700 1 -- 172.21.15.3:0/233053655 shutdown_connections 2024-05-08T21:22:46.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.810+0000 7efcfb921700 1 -- 172.21.15.3:0/233053655 wait complete. 2024-05-08T21:22:46.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.811+0000 7efcfb921700 1 Processor -- start 2024-05-08T21:22:46.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.811+0000 7efcfb921700 1 -- start start 2024-05-08T21:22:46.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.812+0000 7efcfb921700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcf40d1910 0x7efcf406ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:46.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.812+0000 7efcfb921700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf415b4e0 0x7efcf41f7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:46.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.812+0000 7efcfb921700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efcf415b8c0 0x7efcf414f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:46.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.812+0000 7efcfb921700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf41f8210 con 0x7efcf40d1910 2024-05-08T21:22:46.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.812+0000 7efcfb921700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf406c130 con 0x7efcf415b8c0 2024-05-08T21:22:46.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.812+0000 7efcfb921700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf406c410 con 0x7efcf415b4e0 2024-05-08T21:22:46.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.812+0000 7efcf96bd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcf40d1910 0x7efcf406ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:46.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.812+0000 7efcf96bd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcf40d1910 0x7efcf406ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60028/0 (socket says 172.21.15.3:60028) 2024-05-08T21:22:46.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.812+0000 7efcf96bd700 1 -- 172.21.15.3:0/1162590073 learned_addr learned my addr 172.21.15.3:0/1162590073 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:46.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.813+0000 7efcf96bd700 1 -- 172.21.15.3:0/1162590073 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf415b4e0 msgr2=0x7efcf41f7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:46.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.813+0000 7efcf8ebc700 1 --2- 172.21.15.3:0/1162590073 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf415b4e0 0x7efcf41f7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:46.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.813+0000 7efcf9ebe700 1 --2- 172.21.15.3:0/1162590073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efcf415b8c0 0x7efcf414f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:46.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.813+0000 7efcf96bd700 1 --2- 172.21.15.3:0/1162590073 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf415b4e0 0x7efcf41f7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.813+0000 7efcf96bd700 1 -- 172.21.15.3:0/1162590073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efcf415b8c0 msgr2=0x7efcf414f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:46.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.813+0000 7efcf96bd700 1 --2- 172.21.15.3:0/1162590073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efcf415b8c0 0x7efcf414f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:46.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.813+0000 7efcf96bd700 1 -- 172.21.15.3:0/1162590073 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf00094a0 con 0x7efcf40d1910 2024-05-08T21:22:46.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.813+0000 7efcf96bd700 1 --2- 172.21.15.3:0/1162590073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcf40d1910 0x7efcf406ba70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7efcf40c8490 tx=0x7efcf0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:46.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.813+0000 7efce9ffb700 1 -- 172.21.15.3:0/1162590073 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf0010590 con 0x7efcf40d1910 2024-05-08T21:22:46.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.813+0000 7efce9ffb700 1 -- 172.21.15.3:0/1162590073 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf00106f0 con 0x7efcf40d1910 2024-05-08T21:22:46.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.813+0000 7efce9ffb700 1 -- 172.21.15.3:0/1162590073 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf0010aa0 con 0x7efcf40d1910 2024-05-08T21:22:46.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.813+0000 7efcfb921700 1 -- 172.21.15.3:0/1162590073 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf406c6f0 con 0x7efcf40d1910 2024-05-08T21:22:46.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.813+0000 7efcfb921700 1 -- 172.21.15.3:0/1162590073 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf40b40d0 con 0x7efcf40d1910 2024-05-08T21:22:46.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.814+0000 7efcd77fe700 1 -- 172.21.15.3:0/1162590073 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf4074bc0 con 0x7efcf40d1910 2024-05-08T21:22:46.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.815+0000 7efce9ffb700 1 -- 172.21.15.3:0/1162590073 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efcf0010c00 con 0x7efcf40d1910 2024-05-08T21:22:46.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.815+0000 7efce9ffb700 1 --2- 172.21.15.3:0/1162590073 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efcd806ac90 0x7efcd806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:46.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.816+0000 7efce9ffb700 1 -- 172.21.15.3:0/1162590073 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efcf008a860 con 0x7efcf40d1910 2024-05-08T21:22:46.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.816+0000 7efcf8ebc700 1 --2- 172.21.15.3:0/1162590073 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efcd806ac90 0x7efcd806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:46.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.816+0000 7efcf8ebc700 1 --2- 172.21.15.3:0/1162590073 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efcd806ac90 0x7efcd806d150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efce4007fd0 tx=0x7efce4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:46.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:46.820+0000 7efce9ffb700 1 -- 172.21.15.3:0/1162590073 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7efcf0055f40 con 0x7efcf40d1910 2024-05-08T21:22:47.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.039+0000 7efcd77fe700 1 -- 172.21.15.3:0/1162590073 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efcf40b0d10 con 0x7efcf40d1910 2024-05-08T21:22:47.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.039+0000 7efce9ffb700 1 -- 172.21.15.3:0/1162590073 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7efcf00200a0 con 0x7efcf40d1910 2024-05-08T21:22:47.042 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:22:47.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.041+0000 7efcd77fe700 1 -- 172.21.15.3:0/1162590073 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efcd806ac90 msgr2=0x7efcd806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:47.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.041+0000 7efcd77fe700 1 --2- 172.21.15.3:0/1162590073 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efcd806ac90 0x7efcd806d150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efce4007fd0 tx=0x7efce4009040 comp rx=0 tx=0).stop 2024-05-08T21:22:47.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.041+0000 7efcd77fe700 1 -- 172.21.15.3:0/1162590073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcf40d1910 msgr2=0x7efcf406ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:47.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.041+0000 7efcd77fe700 1 --2- 172.21.15.3:0/1162590073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcf40d1910 0x7efcf406ba70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7efcf40c8490 tx=0x7efcf0004750 comp rx=0 tx=0).stop 2024-05-08T21:22:47.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.041+0000 7efcd77fe700 1 -- 172.21.15.3:0/1162590073 shutdown_connections 2024-05-08T21:22:47.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.041+0000 7efcd77fe700 1 --2- 172.21.15.3:0/1162590073 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efcd806ac90 0x7efcd806d150 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.041+0000 7efcd77fe700 1 --2- 172.21.15.3:0/1162590073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efcf415b8c0 0x7efcf414f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.041+0000 7efcd77fe700 1 --2- 172.21.15.3:0/1162590073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcf40d1910 0x7efcf406ba70 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.041+0000 7efcd77fe700 1 --2- 172.21.15.3:0/1162590073 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf415b4e0 0x7efcf41f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.041+0000 7efcd77fe700 1 -- 172.21.15.3:0/1162590073 >> 172.21.15.3:0/1162590073 conn(0x7efcf408afe0 msgr2=0x7efcf40cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:47.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.042+0000 7efcd77fe700 1 -- 172.21.15.3:0/1162590073 shutdown_connections 2024-05-08T21:22:47.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.042+0000 7efcd77fe700 1 -- 172.21.15.3:0/1162590073 wait complete. 2024-05-08T21:22:47.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-08T21:22:47.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.191+0000 7fdb1dd2d700 1 Processor -- start 2024-05-08T21:22:47.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.191+0000 7fdb1dd2d700 1 -- start start 2024-05-08T21:22:47.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.192+0000 7fdb1dd2d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdb1815a750 0x7fdb180d0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:47.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.192+0000 7fdb1dd2d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb180af860 0x7fdb180ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:47.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.192+0000 7fdb1dd2d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb180c5720 0x7fdb180c5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:47.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.192+0000 7fdb1dd2d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb180c6110 con 0x7fdb1815a750 2024-05-08T21:22:47.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.192+0000 7fdb1dd2d700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb180cfbd0 con 0x7fdb180af860 2024-05-08T21:22:47.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.192+0000 7fdb1dd2d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb180cb780 con 0x7fdb180c5720 2024-05-08T21:22:47.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.192+0000 7fdb177fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdb1815a750 0x7fdb180d0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:47.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.192+0000 7fdb177fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdb1815a750 0x7fdb180d0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60044/0 (socket says 172.21.15.3:60044) 2024-05-08T21:22:47.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.192+0000 7fdb177fe700 1 -- 172.21.15.3:0/811346934 learned_addr learned my addr 172.21.15.3:0/811346934 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:47.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.192+0000 7fdb0effd700 1 --2- 172.21.15.3:0/811346934 >> v2:172.21.15.186:3300/0 conn(0x7fdb180af860 0x7fdb180ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:47.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.192+0000 7fdb17fff700 1 --2- 172.21.15.3:0/811346934 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb180c5720 0x7fdb180c5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:47.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.193+0000 7fdb177fe700 1 -- 172.21.15.3:0/811346934 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb180c5720 msgr2=0x7fdb180c5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:47.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.193+0000 7fdb177fe700 1 --2- 172.21.15.3:0/811346934 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb180c5720 0x7fdb180c5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.193+0000 7fdb177fe700 1 -- 172.21.15.3:0/811346934 >> v2:172.21.15.186:3300/0 conn(0x7fdb180af860 msgr2=0x7fdb180ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:47.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.193+0000 7fdb177fe700 1 --2- 172.21.15.3:0/811346934 >> v2:172.21.15.186:3300/0 conn(0x7fdb180af860 0x7fdb180ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.193+0000 7fdb177fe700 1 -- 172.21.15.3:0/811346934 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb180cf650 con 0x7fdb1815a750 2024-05-08T21:22:47.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.193+0000 7fdb177fe700 1 --2- 172.21.15.3:0/811346934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb1815a750 0x7fdb180d0ec0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdb18073100 tx=0x7fdb08009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ce068566e5f21f server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:47.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.193+0000 7fdb16ffd700 1 -- 172.21.15.3:0/811346934 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb08011830 con 0x7fdb1815a750 2024-05-08T21:22:47.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.193+0000 7fdb16ffd700 1 -- 172.21.15.3:0/811346934 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb08011990 con 0x7fdb1815a750 2024-05-08T21:22:47.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.193+0000 7fdb16ffd700 1 -- 172.21.15.3:0/811346934 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb08011d20 con 0x7fdb1815a750 2024-05-08T21:22:47.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.194+0000 7fdb1dd2d700 1 -- 172.21.15.3:0/811346934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb1815a750 msgr2=0x7fdb180d0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:47.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.194+0000 7fdb1dd2d700 1 --2- 172.21.15.3:0/811346934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb1815a750 0x7fdb180d0ec0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdb18073100 tx=0x7fdb08009bd0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.194+0000 7fdb1dd2d700 1 -- 172.21.15.3:0/811346934 shutdown_connections 2024-05-08T21:22:47.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.194+0000 7fdb1dd2d700 1 --2- 172.21.15.3:0/811346934 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb180c5720 0x7fdb180c5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.194+0000 7fdb1dd2d700 1 --2- 172.21.15.3:0/811346934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb1815a750 0x7fdb180d0ec0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.194+0000 7fdb1dd2d700 1 --2- 172.21.15.3:0/811346934 >> v2:172.21.15.186:3300/0 conn(0x7fdb180af860 0x7fdb180ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.194+0000 7fdb1dd2d700 1 -- 172.21.15.3:0/811346934 >> 172.21.15.3:0/811346934 conn(0x7fdb180d03a0 msgr2=0x7fdb180d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:47.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.194+0000 7fdb1dd2d700 1 -- 172.21.15.3:0/811346934 shutdown_connections 2024-05-08T21:22:47.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.194+0000 7fdb1dd2d700 1 -- 172.21.15.3:0/811346934 wait complete. 2024-05-08T21:22:47.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.195+0000 7fdb1dd2d700 1 Processor -- start 2024-05-08T21:22:47.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.196+0000 7fdb1dd2d700 1 -- start start 2024-05-08T21:22:47.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb1dd2d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb180af860 0x7fdb1806abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:47.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb1dd2d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb180c5720 0x7fdb180c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:47.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb1dd2d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb1815a750 0x7fdb180c8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:47.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb1dd2d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb180b09c0 con 0x7fdb1815a750 2024-05-08T21:22:47.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb1dd2d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb180c9250 con 0x7fdb180af860 2024-05-08T21:22:47.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb1dd2d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb180c9530 con 0x7fdb180c5720 2024-05-08T21:22:47.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb17fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb1815a750 0x7fdb180c8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:47.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb177fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb180af860 0x7fdb1806abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:47.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb17fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb1815a750 0x7fdb180c8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60048/0 (socket says 172.21.15.3:60048) 2024-05-08T21:22:47.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb17fff700 1 -- 172.21.15.3:0/3552266894 learned_addr learned my addr 172.21.15.3:0/3552266894 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:47.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb0effd700 1 --2- 172.21.15.3:0/3552266894 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb180c5720 0x7fdb180c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:47.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb17fff700 1 -- 172.21.15.3:0/3552266894 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb180c5720 msgr2=0x7fdb180c86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:47.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb17fff700 1 --2- 172.21.15.3:0/3552266894 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb180c5720 0x7fdb180c86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb17fff700 1 -- 172.21.15.3:0/3552266894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb180af860 msgr2=0x7fdb1806abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:47.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb17fff700 1 --2- 172.21.15.3:0/3552266894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb180af860 0x7fdb1806abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb17fff700 1 -- 172.21.15.3:0/3552266894 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb08009550 con 0x7fdb1815a750 2024-05-08T21:22:47.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb17fff700 1 --2- 172.21.15.3:0/3552266894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb1815a750 0x7fdb180c8c20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdb1000efc0 tx=0x7fdb1000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:47.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb14ff9700 1 -- 172.21.15.3:0/3552266894 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb10009ea0 con 0x7fdb1815a750 2024-05-08T21:22:47.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb14ff9700 1 -- 172.21.15.3:0/3552266894 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1000f040 con 0x7fdb1815a750 2024-05-08T21:22:47.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb1dd2d700 1 -- 172.21.15.3:0/3552266894 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb180c9810 con 0x7fdb1815a750 2024-05-08T21:22:47.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.197+0000 7fdb14ff9700 1 -- 172.21.15.3:0/3552266894 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1000ced0 con 0x7fdb1815a750 2024-05-08T21:22:47.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.198+0000 7fdb1dd2d700 1 -- 172.21.15.3:0/3552266894 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb180c9dd0 con 0x7fdb1815a750 2024-05-08T21:22:47.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.198+0000 7fdb1dd2d700 1 -- 172.21.15.3:0/3552266894 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb180c9fc0 con 0x7fdb1815a750 2024-05-08T21:22:47.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.199+0000 7fdb14ff9700 1 -- 172.21.15.3:0/3552266894 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdb10014480 con 0x7fdb1815a750 2024-05-08T21:22:47.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.199+0000 7fdb14ff9700 1 --2- 172.21.15.3:0/3552266894 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdaf806ac90 0x7fdaf806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:47.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.200+0000 7fdb14ff9700 1 -- 172.21.15.3:0/3552266894 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdb10089a00 con 0x7fdb1815a750 2024-05-08T21:22:47.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.200+0000 7fdb177fe700 1 --2- 172.21.15.3:0/3552266894 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdaf806ac90 0x7fdaf806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:47.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.200+0000 7fdb177fe700 1 --2- 172.21.15.3:0/3552266894 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdaf806ac90 0x7fdaf806d150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009840 tx=0x7fdb08008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:47.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.204+0000 7fdb14ff9700 1 -- 172.21.15.3:0/3552266894 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fdb10054fc0 con 0x7fdb1815a750 2024-05-08T21:22:47.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.427+0000 7fdb1dd2d700 1 -- 172.21.15.3:0/3552266894 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fdb18068000 con 0x7fdb1815a750 2024-05-08T21:22:47.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.427+0000 7fdb14ff9700 1 -- 172.21.15.3:0/3552266894 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7fdb10018090 con 0x7fdb1815a750 2024-05-08T21:22:47.430 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:47.430 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2024-05-08T21:22:47.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-05-08T21:22:47.430 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2024-05-08T21:22:47.430 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": ".mgr" 2024-05-08T21:22:47.430 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:22:47.430 INFO:teuthology.orchestra.run.smithi003.stdout:] 2024-05-08T21:22:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.430+0000 7fdb1dd2d700 1 -- 172.21.15.3:0/3552266894 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdaf806ac90 msgr2=0x7fdaf806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.430+0000 7fdb1dd2d700 1 --2- 172.21.15.3:0/3552266894 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdaf806ac90 0x7fdaf806d150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009840 tx=0x7fdb08008030 comp rx=0 tx=0).stop 2024-05-08T21:22:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.430+0000 7fdb1dd2d700 1 -- 172.21.15.3:0/3552266894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb1815a750 msgr2=0x7fdb180c8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.430+0000 7fdb1dd2d700 1 --2- 172.21.15.3:0/3552266894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb1815a750 0x7fdb180c8c20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdb1000efc0 tx=0x7fdb1000c3a0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.430+0000 7fdb1dd2d700 1 -- 172.21.15.3:0/3552266894 shutdown_connections 2024-05-08T21:22:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.430+0000 7fdb1dd2d700 1 --2- 172.21.15.3:0/3552266894 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdaf806ac90 0x7fdaf806d150 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.430+0000 7fdb1dd2d700 1 --2- 172.21.15.3:0/3552266894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb1815a750 0x7fdb180c8c20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.430+0000 7fdb1dd2d700 1 --2- 172.21.15.3:0/3552266894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb180af860 0x7fdb1806abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.430+0000 7fdb1dd2d700 1 --2- 172.21.15.3:0/3552266894 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb180c5720 0x7fdb180c86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.430+0000 7fdb1dd2d700 1 -- 172.21.15.3:0/3552266894 >> 172.21.15.3:0/3552266894 conn(0x7fdb180d03a0 msgr2=0x7fdb180aeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.430+0000 7fdb1dd2d700 1 -- 172.21.15.3:0/3552266894 shutdown_connections 2024-05-08T21:22:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.430+0000 7fdb1dd2d700 1 -- 172.21.15.3:0/3552266894 wait complete. 2024-05-08T21:22:47.446 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-08T21:22:47.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-08T21:22:47.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.585+0000 7fa98a6de700 1 Processor -- start 2024-05-08T21:22:47.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.585+0000 7fa98a6de700 1 -- start start 2024-05-08T21:22:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.585+0000 7fa98a6de700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa98415d880 0x7fa9840af990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.585+0000 7fa98a6de700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa98406e7e0 0x7fa98406ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.585+0000 7fa98a6de700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9840b3870 con 0x7fa98406f8c0 2024-05-08T21:22:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.585+0000 7fa98a6de700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98406f150 con 0x7fa98415d880 2024-05-08T21:22:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.585+0000 7fa98a6de700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840b32f0 con 0x7fa98406e7e0 2024-05-08T21:22:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa9837fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa98406f8c0 0x7fa9840ad260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34650/0 (socket says 172.21.15.3:34650) 2024-05-08T21:22:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa9837fe700 1 -- 172.21.15.3:0/1092996433 learned_addr learned my addr 172.21.15.3:0/1092996433 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa983fff700 1 --2- 172.21.15.3:0/1092996433 >> v2:172.21.15.186:3300/0 conn(0x7fa98415d880 0x7fa9840af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa988c7b700 1 --2- 172.21.15.3:0/1092996433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa98406e7e0 0x7fa98406ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa982ffd700 1 -- 172.21.15.3:0/1092996433 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587529453 0 0) 0x7fa9840b3870 con 0x7fa98406f8c0 2024-05-08T21:22:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa982ffd700 1 -- 172.21.15.3:0/1092996433 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa96c0035b0 con 0x7fa98406f8c0 2024-05-08T21:22:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa982ffd700 1 -- 172.21.15.3:0/1092996433 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3601864231 0 0) 0x7fa96c0035b0 con 0x7fa98406f8c0 2024-05-08T21:22:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa982ffd700 1 -- 172.21.15.3:0/1092996433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa98406e7e0 msgr2=0x7fa98406ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa982ffd700 1 --2- 172.21.15.3:0/1092996433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa98406e7e0 0x7fa98406ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa982ffd700 1 -- 172.21.15.3:0/1092996433 >> v2:172.21.15.186:3300/0 conn(0x7fa98415d880 msgr2=0x7fa9840af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa982ffd700 1 --2- 172.21.15.3:0/1092996433 >> v2:172.21.15.186:3300/0 conn(0x7fa98415d880 0x7fa9840af990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa982ffd700 1 -- 172.21.15.3:0/1092996433 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9840b4ee0 con 0x7fa98406f8c0 2024-05-08T21:22:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa982ffd700 1 -- 172.21.15.3:0/1092996433 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7fa978001670 con 0x7fa98406f8c0 2024-05-08T21:22:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa982ffd700 1 -- 172.21.15.3:0/1092996433 >> v1:172.21.15.3:6789/0 conn(0x7fa98406f8c0 legacy=0x7fa9840ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa982ffd700 1 --2- 172.21.15.3:0/1092996433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa98406e7e0 0x7fa96c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa982ffd700 1 --2- 172.21.15.3:0/1092996433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa96c003f80 0x7fa96c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa982ffd700 1 --2- 172.21.15.3:0/1092996433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa96c004d90 0x7fa96c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa982ffd700 1 -- 172.21.15.3:0/1092996433 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98406f150 con 0x7fa96c004d90 2024-05-08T21:22:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa982ffd700 1 -- 172.21.15.3:0/1092996433 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840b32f0 con 0x7fa96c003f80 2024-05-08T21:22:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.586+0000 7fa982ffd700 1 -- 172.21.15.3:0/1092996433 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa96c00ba10 con 0x7fa98406e7e0 2024-05-08T21:22:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.587+0000 7fa988c7b700 1 --2- 172.21.15.3:0/1092996433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa96c004d90 0x7fa96c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.587+0000 7fa983fff700 1 --2- 172.21.15.3:0/1092996433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa98406e7e0 0x7fa96c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.587+0000 7fa9837fe700 1 --2- 172.21.15.3:0/1092996433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa96c003f80 0x7fa96c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.587+0000 7fa988c7b700 1 -- 172.21.15.3:0/1092996433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa98406e7e0 msgr2=0x7fa96c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.587+0000 7fa988c7b700 1 --2- 172.21.15.3:0/1092996433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa98406e7e0 0x7fa96c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.587+0000 7fa988c7b700 1 -- 172.21.15.3:0/1092996433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa96c003f80 msgr2=0x7fa96c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.587+0000 7fa988c7b700 1 --2- 172.21.15.3:0/1092996433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa96c003f80 0x7fa96c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.587+0000 7fa988c7b700 1 -- 172.21.15.3:0/1092996433 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96c0035b0 con 0x7fa96c004d90 2024-05-08T21:22:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.587+0000 7fa988c7b700 1 --2- 172.21.15.3:0/1092996433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa96c004d90 0x7fa96c005210 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009fc0 tx=0x7fa96c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c3e056746c23e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.587+0000 7fa982ffd700 1 -- 172.21.15.3:0/1092996433 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c00ebd0 con 0x7fa96c004d90 2024-05-08T21:22:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.587+0000 7fa982ffd700 1 -- 172.21.15.3:0/1092996433 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97c00f040 con 0x7fa96c004d90 2024-05-08T21:22:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.587+0000 7fa982ffd700 1 -- 172.21.15.3:0/1092996433 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c014480 con 0x7fa96c004d90 2024-05-08T21:22:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.588+0000 7fa98a6de700 1 -- 172.21.15.3:0/1092996433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa96c004d90 msgr2=0x7fa96c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.588+0000 7fa98a6de700 1 --2- 172.21.15.3:0/1092996433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa96c004d90 0x7fa96c005210 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009fc0 tx=0x7fa96c005880 comp rx=0 tx=0).stop 2024-05-08T21:22:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.588+0000 7fa988c7b700 1 -- 172.21.15.3:0/1092996433 reap_dead start 2024-05-08T21:22:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.588+0000 7fa98a6de700 1 -- 172.21.15.3:0/1092996433 shutdown_connections 2024-05-08T21:22:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.588+0000 7fa98a6de700 1 -- 172.21.15.3:0/1092996433 >> 172.21.15.3:0/1092996433 conn(0x7fa9840b5810 msgr2=0x7fa9840b5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.588+0000 7fa98a6de700 1 -- 172.21.15.3:0/1092996433 shutdown_connections 2024-05-08T21:22:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.588+0000 7fa98a6de700 1 -- 172.21.15.3:0/1092996433 wait complete. 2024-05-08T21:22:47.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.589+0000 7fa98a6de700 1 Processor -- start 2024-05-08T21:22:47.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.589+0000 7fa98a6de700 1 -- start start 2024-05-08T21:22:47.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.590+0000 7fa98a6de700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa984140290 0x7fa984140c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:47.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.590+0000 7fa98a6de700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa984140670 0x7fa9841423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:47.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.590+0000 7fa98a6de700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9841440f0 0x7fa984144570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:47.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.590+0000 7fa98a6de700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa984141140 con 0x7fa984140290 2024-05-08T21:22:47.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.590+0000 7fa98a6de700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa984141330 con 0x7fa9841440f0 2024-05-08T21:22:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.590+0000 7fa98a6de700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa984141610 con 0x7fa984140670 2024-05-08T21:22:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.590+0000 7fa983fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa984140290 0x7fa984140c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.590+0000 7fa983fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa984140290 0x7fa984140c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60054/0 (socket says 172.21.15.3:60054) 2024-05-08T21:22:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.590+0000 7fa983fff700 1 -- 172.21.15.3:0/2966437535 learned_addr learned my addr 172.21.15.3:0/2966437535 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.591+0000 7fa988c7b700 1 --2- 172.21.15.3:0/2966437535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9841440f0 0x7fa984144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.591+0000 7fa9837fe700 1 --2- 172.21.15.3:0/2966437535 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa984140670 0x7fa9841423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.591+0000 7fa983fff700 1 -- 172.21.15.3:0/2966437535 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa984140670 msgr2=0x7fa9841423d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.591+0000 7fa983fff700 1 --2- 172.21.15.3:0/2966437535 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa984140670 0x7fa9841423d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.591+0000 7fa983fff700 1 -- 172.21.15.3:0/2966437535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9841440f0 msgr2=0x7fa984144570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.591+0000 7fa983fff700 1 --2- 172.21.15.3:0/2966437535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9841440f0 0x7fa984144570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.591+0000 7fa983fff700 1 -- 172.21.15.3:0/2966437535 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9841418f0 con 0x7fa984140290 2024-05-08T21:22:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.591+0000 7fa983fff700 1 --2- 172.21.15.3:0/2966437535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa984140290 0x7fa984140c00 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa974011aa0 tx=0x7fa974011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.591+0000 7fa980ff9700 1 -- 172.21.15.3:0/2966437535 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97400f9e0 con 0x7fa984140290 2024-05-08T21:22:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.591+0000 7fa980ff9700 1 -- 172.21.15.3:0/2966437535 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97400fb40 con 0x7fa984140290 2024-05-08T21:22:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.591+0000 7fa980ff9700 1 -- 172.21.15.3:0/2966437535 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa974014440 con 0x7fa984140290 2024-05-08T21:22:47.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.591+0000 7fa98a6de700 1 -- 172.21.15.3:0/2966437535 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa984141bd0 con 0x7fa984140290 2024-05-08T21:22:47.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.591+0000 7fa98a6de700 1 -- 172.21.15.3:0/2966437535 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa984144ab0 con 0x7fa984140290 2024-05-08T21:22:47.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.592+0000 7fa98a6de700 1 -- 172.21.15.3:0/2966437535 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa984074bc0 con 0x7fa984140290 2024-05-08T21:22:47.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.593+0000 7fa980ff9700 1 -- 172.21.15.3:0/2966437535 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa9740145a0 con 0x7fa984140290 2024-05-08T21:22:47.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.593+0000 7fa980ff9700 1 --2- 172.21.15.3:0/2966437535 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa97006ac90 0x7fa97006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:47.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.593+0000 7fa980ff9700 1 -- 172.21.15.3:0/2966437535 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa974089790 con 0x7fa984140290 2024-05-08T21:22:47.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.593+0000 7fa9837fe700 1 --2- 172.21.15.3:0/2966437535 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa97006ac90 0x7fa97006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:47.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.594+0000 7fa9837fe700 1 --2- 172.21.15.3:0/2966437535 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa97006ac90 0x7fa97006d150 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa9780048d0 tx=0x7fa978009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:47.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.598+0000 7fa980ff9700 1 -- 172.21.15.3:0/2966437535 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fa974054cc0 con 0x7fa984140290 2024-05-08T21:22:47.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:47.820+0000 7fa98a6de700 1 -- 172.21.15.3:0/2966437535 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa9840b5e30 con 0x7fa984140290 2024-05-08T21:22:48.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.719+0000 7fa980ff9700 1 -- 172.21.15.3:0/2966437535 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa9740582e0 con 0x7fa984140290 2024-05-08T21:22:48.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.726+0000 7fa98a6de700 1 -- 172.21.15.3:0/2966437535 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa97006ac90 msgr2=0x7fa97006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:48.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.726+0000 7fa98a6de700 1 --2- 172.21.15.3:0/2966437535 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa97006ac90 0x7fa97006d150 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa9780048d0 tx=0x7fa978009040 comp rx=0 tx=0).stop 2024-05-08T21:22:48.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.726+0000 7fa98a6de700 1 -- 172.21.15.3:0/2966437535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa984140290 msgr2=0x7fa984140c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:48.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.726+0000 7fa98a6de700 1 --2- 172.21.15.3:0/2966437535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa984140290 0x7fa984140c00 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa974011aa0 tx=0x7fa974011e60 comp rx=0 tx=0).stop 2024-05-08T21:22:48.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.726+0000 7fa98a6de700 1 -- 172.21.15.3:0/2966437535 shutdown_connections 2024-05-08T21:22:48.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.726+0000 7fa98a6de700 1 --2- 172.21.15.3:0/2966437535 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa97006ac90 0x7fa97006d150 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:48.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.726+0000 7fa98a6de700 1 --2- 172.21.15.3:0/2966437535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9841440f0 0x7fa984144570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:48.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.726+0000 7fa98a6de700 1 --2- 172.21.15.3:0/2966437535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa984140290 0x7fa984140c00 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:48.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.726+0000 7fa98a6de700 1 --2- 172.21.15.3:0/2966437535 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa984140670 0x7fa9841423d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:48.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.726+0000 7fa98a6de700 1 -- 172.21.15.3:0/2966437535 >> 172.21.15.3:0/2966437535 conn(0x7fa9840b5810 msgr2=0x7fa9840af7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:48.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.726+0000 7fa98a6de700 1 -- 172.21.15.3:0/2966437535 shutdown_connections 2024-05-08T21:22:48.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.726+0000 7fa98a6de700 1 -- 172.21.15.3:0/2966437535 wait complete. 2024-05-08T21:22:48.730 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2024-05-08T21:22:48.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-08T21:22:48.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.891+0000 7fec7b9a9700 1 Processor -- start 2024-05-08T21:22:48.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.891+0000 7fec7b9a9700 1 -- start start 2024-05-08T21:22:48.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.892+0000 7fec7b9a9700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fec74154df0 0x7fec740d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:48.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.892+0000 7fec7b9a9700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fec740aea70 0x7fec740b4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:48.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.892+0000 7fec7b9a9700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec7415add0 con 0x7fec740d0890 2024-05-08T21:22:48.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.892+0000 7fec7b9a9700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7415abe0 con 0x7fec740aea70 2024-05-08T21:22:48.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.892+0000 7fec7b9a9700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec740d0d20 con 0x7fec74154df0 2024-05-08T21:22:48.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.892+0000 7fec78f44700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fec740d0890 0x7fec740b3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34652/0 (socket says 172.21.15.3:34652) 2024-05-08T21:22:48.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.892+0000 7fec78f44700 1 -- 172.21.15.3:0/3700248241 learned_addr learned my addr 172.21.15.3:0/3700248241 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:48.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.892+0000 7fec79f46700 1 --2- 172.21.15.3:0/3700248241 >> v2:172.21.15.3:3300/0 conn(0x7fec740aea70 0x7fec740b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:48.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.892+0000 7fec79745700 1 --2- 172.21.15.3:0/3700248241 >> v2:172.21.15.186:3300/0 conn(0x7fec74154df0 0x7fec740d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:48.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.892+0000 7fec6bfff700 1 -- 172.21.15.3:0/3700248241 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370232929 0 0) 0x7fec7415add0 con 0x7fec740d0890 2024-05-08T21:22:48.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.892+0000 7fec6bfff700 1 -- 172.21.15.3:0/3700248241 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec600035b0 con 0x7fec740d0890 2024-05-08T21:22:48.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.892+0000 7fec79f46700 1 -- 172.21.15.3:0/3700248241 >> v2:172.21.15.186:3300/0 conn(0x7fec74154df0 msgr2=0x7fec740d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:48.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.892+0000 7fec79f46700 1 --2- 172.21.15.3:0/3700248241 >> v2:172.21.15.186:3300/0 conn(0x7fec74154df0 0x7fec740d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:48.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.892+0000 7fec79f46700 1 -- 172.21.15.3:0/3700248241 >> v1:172.21.15.3:6789/0 conn(0x7fec740d0890 legacy=0x7fec740b3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:48.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.892+0000 7fec79f46700 1 -- 172.21.15.3:0/3700248241 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7415b070 con 0x7fec740aea70 2024-05-08T21:22:48.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.893+0000 7fec79f46700 1 --2- 172.21.15.3:0/3700248241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec740aea70 0x7fec740b4580 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fec740675f0 tx=0x7fec6c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc80db046d5671f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:48.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.893+0000 7fec6bfff700 1 -- 172.21.15.3:0/3700248241 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c00e070 con 0x7fec740aea70 2024-05-08T21:22:48.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.893+0000 7fec6bfff700 1 -- 172.21.15.3:0/3700248241 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6c00bae0 con 0x7fec740aea70 2024-05-08T21:22:48.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.893+0000 7fec6bfff700 1 -- 172.21.15.3:0/3700248241 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c00be50 con 0x7fec740aea70 2024-05-08T21:22:48.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.893+0000 7fec7b9a9700 1 -- 172.21.15.3:0/3700248241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec740aea70 msgr2=0x7fec740b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:48.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.893+0000 7fec7b9a9700 1 --2- 172.21.15.3:0/3700248241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec740aea70 0x7fec740b4580 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fec740675f0 tx=0x7fec6c00b330 comp rx=0 tx=0).stop 2024-05-08T21:22:48.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.893+0000 7fec7b9a9700 1 -- 172.21.15.3:0/3700248241 shutdown_connections 2024-05-08T21:22:48.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.893+0000 7fec7b9a9700 1 --2- 172.21.15.3:0/3700248241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec740aea70 0x7fec740b4580 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:48.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.893+0000 7fec7b9a9700 1 --2- 172.21.15.3:0/3700248241 >> v2:172.21.15.186:3300/0 conn(0x7fec74154df0 0x7fec740d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:48.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.893+0000 7fec7b9a9700 1 -- 172.21.15.3:0/3700248241 >> 172.21.15.3:0/3700248241 conn(0x7fec74099a50 msgr2=0x7fec740b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:48.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.893+0000 7fec7b9a9700 1 -- 172.21.15.3:0/3700248241 shutdown_connections 2024-05-08T21:22:48.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.894+0000 7fec7b9a9700 1 -- 172.21.15.3:0/3700248241 wait complete. 2024-05-08T21:22:48.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.895+0000 7fec7b9a9700 1 Processor -- start 2024-05-08T21:22:48.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.895+0000 7fec7b9a9700 1 -- start start 2024-05-08T21:22:48.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec7b9a9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec740aea70 0x7fec740cc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:48.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec7b9a9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec740d0890 0x7fec7413c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:48.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec7b9a9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec74154df0 0x7fec740adc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:48.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec7b9a9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec740cdb80 con 0x7fec74154df0 2024-05-08T21:22:48.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec7b9a9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec740ccf70 con 0x7fec740d0890 2024-05-08T21:22:48.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec7b9a9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec740cd250 con 0x7fec740aea70 2024-05-08T21:22:48.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec79f46700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec74154df0 0x7fec740adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:48.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec79f46700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec74154df0 0x7fec740adc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60058/0 (socket says 172.21.15.3:60058) 2024-05-08T21:22:48.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec79f46700 1 -- 172.21.15.3:0/1370238945 learned_addr learned my addr 172.21.15.3:0/1370238945 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:48.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec79745700 1 --2- 172.21.15.3:0/1370238945 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec740aea70 0x7fec740cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:48.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec78f44700 1 --2- 172.21.15.3:0/1370238945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec740d0890 0x7fec7413c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:48.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec79f46700 1 -- 172.21.15.3:0/1370238945 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec740aea70 msgr2=0x7fec740cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:48.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec79f46700 1 --2- 172.21.15.3:0/1370238945 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec740aea70 0x7fec740cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:48.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec79f46700 1 -- 172.21.15.3:0/1370238945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec740d0890 msgr2=0x7fec7413c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:48.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec79f46700 1 --2- 172.21.15.3:0/1370238945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec740d0890 0x7fec7413c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:48.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec79f46700 1 -- 172.21.15.3:0/1370238945 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6c009a70 con 0x7fec74154df0 2024-05-08T21:22:48.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec78f44700 1 --2- 172.21.15.3:0/1370238945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec740d0890 0x7fec7413c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:48.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.896+0000 7fec79f46700 1 --2- 172.21.15.3:0/1370238945 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec74154df0 0x7fec740adc20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00d010 tx=0x7fec6c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:48.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.897+0000 7fec69ffb700 1 -- 172.21.15.3:0/1370238945 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c004450 con 0x7fec74154df0 2024-05-08T21:22:48.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.897+0000 7fec69ffb700 1 -- 172.21.15.3:0/1370238945 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6c01f070 con 0x7fec74154df0 2024-05-08T21:22:48.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.897+0000 7fec7b9a9700 1 -- 172.21.15.3:0/1370238945 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec740cd530 con 0x7fec74154df0 2024-05-08T21:22:48.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.897+0000 7fec69ffb700 1 -- 172.21.15.3:0/1370238945 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c0047f0 con 0x7fec74154df0 2024-05-08T21:22:48.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.897+0000 7fec7b9a9700 1 -- 172.21.15.3:0/1370238945 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec740ccc60 con 0x7fec74154df0 2024-05-08T21:22:48.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.898+0000 7fec7b9a9700 1 -- 172.21.15.3:0/1370238945 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec74074bc0 con 0x7fec74154df0 2024-05-08T21:22:48.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.898+0000 7fec69ffb700 1 -- 172.21.15.3:0/1370238945 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fec6c012430 con 0x7fec74154df0 2024-05-08T21:22:48.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.899+0000 7fec69ffb700 1 --2- 172.21.15.3:0/1370238945 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fec5806ac90 0x7fec5806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:48.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.899+0000 7fec69ffb700 1 -- 172.21.15.3:0/1370238945 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fec6c08a060 con 0x7fec74154df0 2024-05-08T21:22:48.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.899+0000 7fec79745700 1 --2- 172.21.15.3:0/1370238945 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fec5806ac90 0x7fec5806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:48.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.900+0000 7fec79745700 1 --2- 172.21.15.3:0/1370238945 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fec5806ac90 0x7fec5806d150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fec70009b40 tx=0x7fec70009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:48.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:48.904+0000 7fec69ffb700 1 -- 172.21.15.3:0/1370238945 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fec6c055560 con 0x7fec74154df0 2024-05-08T21:22:49.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.120+0000 7fec7b9a9700 1 -- 172.21.15.3:0/1370238945 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fec740ae680 con 0x7fec74154df0 2024-05-08T21:22:49.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.723+0000 7fec69ffb700 1 -- 172.21.15.3:0/1370238945 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7fec6c058b80 con 0x7fec74154df0 2024-05-08T21:22:49.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.731+0000 7fec7b9a9700 1 -- 172.21.15.3:0/1370238945 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fec5806ac90 msgr2=0x7fec5806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:49.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.731+0000 7fec7b9a9700 1 --2- 172.21.15.3:0/1370238945 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fec5806ac90 0x7fec5806d150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fec70009b40 tx=0x7fec70009210 comp rx=0 tx=0).stop 2024-05-08T21:22:49.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.731+0000 7fec7b9a9700 1 -- 172.21.15.3:0/1370238945 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec74154df0 msgr2=0x7fec740adc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:49.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.731+0000 7fec7b9a9700 1 --2- 172.21.15.3:0/1370238945 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec74154df0 0x7fec740adc20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00d010 tx=0x7fec6c004ad0 comp rx=0 tx=0).stop 2024-05-08T21:22:49.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.732+0000 7fec7b9a9700 1 -- 172.21.15.3:0/1370238945 shutdown_connections 2024-05-08T21:22:49.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.732+0000 7fec7b9a9700 1 --2- 172.21.15.3:0/1370238945 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec740aea70 0x7fec740cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:49.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.732+0000 7fec7b9a9700 1 --2- 172.21.15.3:0/1370238945 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fec5806ac90 0x7fec5806d150 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:49.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.732+0000 7fec7b9a9700 1 --2- 172.21.15.3:0/1370238945 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec74154df0 0x7fec740adc20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:49.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.732+0000 7fec7b9a9700 1 --2- 172.21.15.3:0/1370238945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec740d0890 0x7fec7413c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:49.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.732+0000 7fec7b9a9700 1 -- 172.21.15.3:0/1370238945 >> 172.21.15.3:0/1370238945 conn(0x7fec74099a50 msgr2=0x7fec7406dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:49.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.737+0000 7fec7b9a9700 1 -- 172.21.15.3:0/1370238945 shutdown_connections 2024-05-08T21:22:49.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.738+0000 7fec7b9a9700 1 -- 172.21.15.3:0/1370238945 wait complete. 2024-05-08T21:22:49.746 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2024-05-08T21:22:49.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-08T21:22:49.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.899+0000 7f7b03cc6700 1 Processor -- start 2024-05-08T21:22:49.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.899+0000 7f7b03cc6700 1 -- start start 2024-05-08T21:22:49.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.900+0000 7f7b03cc6700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7afc06f2d0 0x7f7afc0ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:49.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.900+0000 7f7b03cc6700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc165970 con 0x7f7afc1652f0 2024-05-08T21:22:49.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.900+0000 7f7b03cc6700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc1656d0 con 0x7f7afc0c39e0 2024-05-08T21:22:49.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.900+0000 7f7b03cc6700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7afc165bc0 con 0x7f7afc06f2d0 2024-05-08T21:22:49.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.900+0000 7f7b01a62700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7afc1652f0 0x7f7afc0c32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34658/0 (socket says 172.21.15.3:34658) 2024-05-08T21:22:49.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.900+0000 7f7b01a62700 1 -- 172.21.15.3:0/1807025256 learned_addr learned my addr 172.21.15.3:0/1807025256 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:49.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b01261700 1 --2- 172.21.15.3:0/1807025256 >> v2:172.21.15.186:3300/0 conn(0x7f7afc06f2d0 0x7f7afc0ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:49.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b00a60700 1 -- 172.21.15.3:0/1807025256 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773801686 0 0) 0x7f7afc165970 con 0x7f7afc1652f0 2024-05-08T21:22:49.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b00a60700 1 -- 172.21.15.3:0/1807025256 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ae80035b0 con 0x7f7afc1652f0 2024-05-08T21:22:49.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b00a60700 1 -- 172.21.15.3:0/1807025256 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 574075836 0 0) 0x7f7ae80035b0 con 0x7f7afc1652f0 2024-05-08T21:22:49.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b00a60700 1 -- 172.21.15.3:0/1807025256 >> v2:172.21.15.186:3300/0 conn(0x7f7afc06f2d0 msgr2=0x7f7afc0ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b00a60700 1 --2- 172.21.15.3:0/1807025256 >> v2:172.21.15.186:3300/0 conn(0x7f7afc06f2d0 0x7f7afc0ce9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b00a60700 1 -- 172.21.15.3:0/1807025256 >> v1:172.21.15.186:6789/0 conn(0x7f7afc0c39e0 legacy=0x7f7afc0cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b00a60700 1 -- 172.21.15.3:0/1807025256 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc165ea0 con 0x7f7afc1652f0 2024-05-08T21:22:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b00a60700 1 -- 172.21.15.3:0/1807025256 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f7af8001670 con 0x7f7afc1652f0 2024-05-08T21:22:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b00a60700 1 -- 172.21.15.3:0/1807025256 >> v1:172.21.15.3:6789/0 conn(0x7f7afc1652f0 legacy=0x7f7afc0c32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b00a60700 1 --2- 172.21.15.3:0/1807025256 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ae8003a00 0x7f7ae8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b00a60700 1 --2- 172.21.15.3:0/1807025256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ae8004850 0x7f7ae8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b00a60700 1 --2- 172.21.15.3:0/1807025256 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ae8005210 0x7f7ae800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b00a60700 1 -- 172.21.15.3:0/1807025256 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc165bc0 con 0x7f7ae8003a00 2024-05-08T21:22:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b00a60700 1 -- 172.21.15.3:0/1807025256 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae800c0b0 con 0x7f7ae8004850 2024-05-08T21:22:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b00a60700 1 -- 172.21.15.3:0/1807025256 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae800c270 con 0x7f7ae8005210 2024-05-08T21:22:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b01a62700 1 --2- 172.21.15.3:0/1807025256 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ae8003a00 0x7f7ae8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b01261700 1 --2- 172.21.15.3:0/1807025256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ae8004850 0x7f7ae8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.901+0000 7f7b02263700 1 --2- 172.21.15.3:0/1807025256 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ae8005210 0x7f7ae800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:49.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.902+0000 7f7b01a62700 1 -- 172.21.15.3:0/1807025256 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ae8005210 msgr2=0x7f7ae800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:49.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.902+0000 7f7b01a62700 1 --2- 172.21.15.3:0/1807025256 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ae8005210 0x7f7ae800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:49.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.902+0000 7f7b01a62700 1 -- 172.21.15.3:0/1807025256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ae8004850 msgr2=0x7f7ae8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:49.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.902+0000 7f7b01a62700 1 --2- 172.21.15.3:0/1807025256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ae8004850 0x7f7ae8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:49.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.902+0000 7f7b01a62700 1 -- 172.21.15.3:0/1807025256 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae80035b0 con 0x7f7ae8003a00 2024-05-08T21:22:49.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.902+0000 7f7b02263700 1 -- 172.21.15.3:0/1807025256 reap_dead start 2024-05-08T21:22:49.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.902+0000 7f7b01261700 1 --2- 172.21.15.3:0/1807025256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ae8004850 0x7f7ae8004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:49.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.902+0000 7f7b01a62700 1 --2- 172.21.15.3:0/1807025256 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ae8003a00 0x7f7ae8003e20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7af800c2a0 tx=0x7f7ae8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172f27ad54ef9a16 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:49.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.902+0000 7f7b00a60700 1 -- 172.21.15.3:0/1807025256 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8007c10 con 0x7f7ae8003a00 2024-05-08T21:22:49.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.902+0000 7f7b00a60700 1 -- 172.21.15.3:0/1807025256 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af800cde0 con 0x7f7ae8003a00 2024-05-08T21:22:49.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.902+0000 7f7b00a60700 1 -- 172.21.15.3:0/1807025256 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8005ba0 con 0x7f7ae8003a00 2024-05-08T21:22:49.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.902+0000 7f7b03cc6700 1 -- 172.21.15.3:0/1807025256 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ae8003a00 msgr2=0x7f7ae8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:49.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.903+0000 7f7b03cc6700 1 --2- 172.21.15.3:0/1807025256 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ae8003a00 0x7f7ae8003e20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7af800c2a0 tx=0x7f7ae8005880 comp rx=0 tx=0).stop 2024-05-08T21:22:49.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.903+0000 7f7b03cc6700 1 -- 172.21.15.3:0/1807025256 shutdown_connections 2024-05-08T21:22:49.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.903+0000 7f7b03cc6700 1 --2- 172.21.15.3:0/1807025256 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ae8003a00 0x7f7ae8003e20 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:49.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.903+0000 7f7b03cc6700 1 -- 172.21.15.3:0/1807025256 >> 172.21.15.3:0/1807025256 conn(0x7f7afc099a50 msgr2=0x7f7afc0b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:49.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.903+0000 7f7b03cc6700 1 -- 172.21.15.3:0/1807025256 shutdown_connections 2024-05-08T21:22:49.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.903+0000 7f7b03cc6700 1 -- 172.21.15.3:0/1807025256 wait complete. 2024-05-08T21:22:49.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.904+0000 7f7b03cc6700 1 Processor -- start 2024-05-08T21:22:49.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.904+0000 7f7b03cc6700 1 -- start start 2024-05-08T21:22:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.905+0000 7f7b03cc6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ae8003a00 0x7f7afc152d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.905+0000 7f7b03cc6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7afc0cd350 0x7f7afc0c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.905+0000 7f7b03cc6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7afc0c5f70 0x7f7afc0c6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.905+0000 7f7b03cc6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0c5870 con 0x7f7afc0cd350 2024-05-08T21:22:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.905+0000 7f7b03cc6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0c5ae0 con 0x7f7afc0c5f70 2024-05-08T21:22:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.905+0000 7f7b03cc6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0c7910 con 0x7f7ae8003a00 2024-05-08T21:22:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.905+0000 7f7b01261700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7afc0cd350 0x7f7afc0c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.905+0000 7f7b01261700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7afc0cd350 0x7f7afc0c73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60068/0 (socket says 172.21.15.3:60068) 2024-05-08T21:22:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.905+0000 7f7b01261700 1 -- 172.21.15.3:0/1090069990 learned_addr learned my addr 172.21.15.3:0/1090069990 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.905+0000 7f7b01a62700 1 --2- 172.21.15.3:0/1090069990 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ae8003a00 0x7f7afc152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.905+0000 7f7b02263700 1 --2- 172.21.15.3:0/1090069990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7afc0c5f70 0x7f7afc0c6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.905+0000 7f7b01261700 1 -- 172.21.15.3:0/1090069990 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ae8003a00 msgr2=0x7f7afc152d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.905+0000 7f7b01261700 1 --2- 172.21.15.3:0/1090069990 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ae8003a00 0x7f7afc152d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.905+0000 7f7b01261700 1 -- 172.21.15.3:0/1090069990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7afc0c5f70 msgr2=0x7f7afc0c6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.905+0000 7f7b01261700 1 --2- 172.21.15.3:0/1090069990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7afc0c5f70 0x7f7afc0c6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:49.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.905+0000 7f7b01261700 1 -- 172.21.15.3:0/1090069990 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc0cd7e0 con 0x7f7afc0cd350 2024-05-08T21:22:49.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.906+0000 7f7b01261700 1 --2- 172.21.15.3:0/1090069990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7afc0cd350 0x7f7afc0c73d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7aec011bd0 tx=0x7f7aec011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:49.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.906+0000 7f7af27fc700 1 -- 172.21.15.3:0/1090069990 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec00fa00 con 0x7f7afc0cd350 2024-05-08T21:22:49.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.906+0000 7f7af27fc700 1 -- 172.21.15.3:0/1090069990 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aec00fb60 con 0x7f7afc0cd350 2024-05-08T21:22:49.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.906+0000 7f7b03cc6700 1 -- 172.21.15.3:0/1090069990 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7afc0c9f10 con 0x7f7afc0cd350 2024-05-08T21:22:49.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.906+0000 7f7af27fc700 1 -- 172.21.15.3:0/1090069990 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec004020 con 0x7f7afc0cd350 2024-05-08T21:22:49.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.906+0000 7f7b03cc6700 1 -- 172.21.15.3:0/1090069990 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7afc1535c0 con 0x7f7afc0cd350 2024-05-08T21:22:49.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.907+0000 7f7b03cc6700 1 -- 172.21.15.3:0/1090069990 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7afc0ca430 con 0x7f7afc0cd350 2024-05-08T21:22:49.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.907+0000 7f7af27fc700 1 -- 172.21.15.3:0/1090069990 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7aec004180 con 0x7f7afc0cd350 2024-05-08T21:22:49.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.908+0000 7f7af27fc700 1 --2- 172.21.15.3:0/1090069990 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7ae006ac90 0x7f7ae006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:49.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.908+0000 7f7af27fc700 1 -- 172.21.15.3:0/1090069990 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f7aec089570 con 0x7f7afc0cd350 2024-05-08T21:22:49.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.908+0000 7f7b01a62700 1 --2- 172.21.15.3:0/1090069990 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7ae006ac90 0x7f7ae006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:49.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.909+0000 7f7b01a62700 1 --2- 172.21.15.3:0/1090069990 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7ae006ac90 0x7f7ae006d150 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7af8015040 tx=0x7f7af8007000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:49.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:49.913+0000 7f7af27fc700 1 -- 172.21.15.3:0/1090069990 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f7afc0ca430 con 0x7f7afc0cd350 2024-05-08T21:22:50.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.181+0000 7f7b03cc6700 1 -- 172.21.15.3:0/1090069990 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7afc0ca430 con 0x7f7afc0cd350 2024-05-08T21:22:50.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.187+0000 7f7af27fc700 1 -- 172.21.15.3:0/1090069990 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7afc0ca430 con 0x7f7afc0cd350 2024-05-08T21:22:50.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.207+0000 7f7b03cc6700 1 -- 172.21.15.3:0/1090069990 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7ae006ac90 msgr2=0x7f7ae006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:50.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.207+0000 7f7b03cc6700 1 --2- 172.21.15.3:0/1090069990 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7ae006ac90 0x7f7ae006d150 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7af8015040 tx=0x7f7af8007000 comp rx=0 tx=0).stop 2024-05-08T21:22:50.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.207+0000 7f7b03cc6700 1 -- 172.21.15.3:0/1090069990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7afc0cd350 msgr2=0x7f7afc0c73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:50.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.207+0000 7f7b03cc6700 1 --2- 172.21.15.3:0/1090069990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7afc0cd350 0x7f7afc0c73d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7aec011bd0 tx=0x7f7aec011f90 comp rx=0 tx=0).stop 2024-05-08T21:22:50.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.208+0000 7f7b03cc6700 1 -- 172.21.15.3:0/1090069990 shutdown_connections 2024-05-08T21:22:50.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.208+0000 7f7b03cc6700 1 --2- 172.21.15.3:0/1090069990 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ae8003a00 0x7f7afc152d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.208+0000 7f7b03cc6700 1 --2- 172.21.15.3:0/1090069990 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7ae006ac90 0x7f7ae006d150 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.208+0000 7f7b03cc6700 1 --2- 172.21.15.3:0/1090069990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7afc0c5f70 0x7f7afc0c6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.208+0000 7f7b03cc6700 1 --2- 172.21.15.3:0/1090069990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7afc0cd350 0x7f7afc0c73d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.208+0000 7f7b03cc6700 1 -- 172.21.15.3:0/1090069990 >> 172.21.15.3:0/1090069990 conn(0x7f7afc099a50 msgr2=0x7f7afc0ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:50.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.208+0000 7f7b03cc6700 1 -- 172.21.15.3:0/1090069990 shutdown_connections 2024-05-08T21:22:50.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.209+0000 7f7b03cc6700 1 -- 172.21.15.3:0/1090069990 wait complete. 2024-05-08T21:22:50.215 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 4 and data pool 5 2024-05-08T21:22:50.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:50.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.362+0000 7f8d4c888700 1 Processor -- start 2024-05-08T21:22:50.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.362+0000 7f8d4c888700 1 -- start start 2024-05-08T21:22:50.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.363+0000 7f8d4c888700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8d480cf3c0 0x7f8d4806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:50.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.363+0000 7f8d4c888700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d480b2b10 con 0x7f8d4806f4a0 2024-05-08T21:22:50.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.363+0000 7f8d4c888700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d480ce340 con 0x7f8d4815a380 2024-05-08T21:22:50.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.363+0000 7f8d4c888700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d480ceba0 con 0x7f8d480cf3c0 2024-05-08T21:22:50.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.363+0000 7f8d45d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8d480cf3c0 0x7f8d4806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:50.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.363+0000 7f8d46d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f8d4806f4a0 0x7f8d4806d8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34670/0 (socket says 172.21.15.3:34670) 2024-05-08T21:22:50.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.363+0000 7f8d46d9d700 1 -- 172.21.15.3:0/677591326 learned_addr learned my addr 172.21.15.3:0/677591326 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:50.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.364+0000 7f8d45d9b700 1 -- 172.21.15.3:0/677591326 >> v1:172.21.15.186:6789/0 conn(0x7f8d4815a380 legacy=0x7f8d480cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:50.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.364+0000 7f8d45d9b700 1 -- 172.21.15.3:0/677591326 >> v1:172.21.15.3:6789/0 conn(0x7f8d4806f4a0 legacy=0x7f8d4806d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:50.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.364+0000 7f8d45d9b700 1 -- 172.21.15.3:0/677591326 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d480ce5e0 con 0x7f8d480cf3c0 2024-05-08T21:22:50.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.364+0000 7f8d45d9b700 1 --2- 172.21.15.3:0/677591326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d480cf3c0 0x7f8d4806ef60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00bae0 tx=0x7f8d3c00bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1a000fa18eb2c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:50.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.364+0000 7f8d4559a700 1 -- 172.21.15.3:0/677591326 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c00dd30 con 0x7f8d480cf3c0 2024-05-08T21:22:50.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.364+0000 7f8d4559a700 1 -- 172.21.15.3:0/677591326 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3c00de90 con 0x7f8d480cf3c0 2024-05-08T21:22:50.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.364+0000 7f8d4559a700 1 -- 172.21.15.3:0/677591326 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c010690 con 0x7f8d480cf3c0 2024-05-08T21:22:50.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.365+0000 7f8d4c888700 1 -- 172.21.15.3:0/677591326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d480cf3c0 msgr2=0x7f8d4806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:50.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.365+0000 7f8d4c888700 1 --2- 172.21.15.3:0/677591326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d480cf3c0 0x7f8d4806ef60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00bae0 tx=0x7f8d3c00bdf0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.365+0000 7f8d4c888700 1 -- 172.21.15.3:0/677591326 shutdown_connections 2024-05-08T21:22:50.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.365+0000 7f8d4c888700 1 --2- 172.21.15.3:0/677591326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d480cf3c0 0x7f8d4806ef60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.365+0000 7f8d4c888700 1 -- 172.21.15.3:0/677591326 >> 172.21.15.3:0/677591326 conn(0x7f8d480cfce0 msgr2=0x7f8d480d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:50.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.365+0000 7f8d4c888700 1 -- 172.21.15.3:0/677591326 shutdown_connections 2024-05-08T21:22:50.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.365+0000 7f8d4c888700 1 -- 172.21.15.3:0/677591326 wait complete. 2024-05-08T21:22:50.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.366+0000 7f8d4c888700 1 Processor -- start 2024-05-08T21:22:50.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.366+0000 7f8d4c888700 1 -- start start 2024-05-08T21:22:50.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.367+0000 7f8d4c888700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d4806f4a0 0x7f8d48145780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:50.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.367+0000 7f8d4c888700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d480cf3c0 0x7f8d481493d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:50.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.367+0000 7f8d4c888700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d4815a380 0x7f8d48151880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:50.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.367+0000 7f8d4c888700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d48145cc0 con 0x7f8d480cf3c0 2024-05-08T21:22:50.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.367+0000 7f8d4c888700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d48157990 con 0x7f8d4806f4a0 2024-05-08T21:22:50.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.367+0000 7f8d4c888700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d48149910 con 0x7f8d4815a380 2024-05-08T21:22:50.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.367+0000 7f8d45d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d480cf3c0 0x7f8d481493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:50.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.367+0000 7f8d45d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d480cf3c0 0x7f8d481493d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60090/0 (socket says 172.21.15.3:60090) 2024-05-08T21:22:50.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.367+0000 7f8d45d9b700 1 -- 172.21.15.3:0/3624048637 learned_addr learned my addr 172.21.15.3:0/3624048637 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:50.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.367+0000 7f8d46d9d700 1 --2- 172.21.15.3:0/3624048637 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d4815a380 0x7f8d48151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:50.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.367+0000 7f8d4659c700 1 --2- 172.21.15.3:0/3624048637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d4806f4a0 0x7f8d48145780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:50.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.368+0000 7f8d45d9b700 1 -- 172.21.15.3:0/3624048637 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d4815a380 msgr2=0x7f8d48151880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:50.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.368+0000 7f8d45d9b700 1 --2- 172.21.15.3:0/3624048637 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d4815a380 0x7f8d48151880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.368+0000 7f8d45d9b700 1 -- 172.21.15.3:0/3624048637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d4806f4a0 msgr2=0x7f8d48145780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:50.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.368+0000 7f8d45d9b700 1 --2- 172.21.15.3:0/3624048637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d4806f4a0 0x7f8d48145780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.368+0000 7f8d45d9b700 1 -- 172.21.15.3:0/3624048637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c00b740 con 0x7f8d480cf3c0 2024-05-08T21:22:50.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.368+0000 7f8d45d9b700 1 --2- 172.21.15.3:0/3624048637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d480cf3c0 0x7f8d481493d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00b2c0 tx=0x7f8d3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:50.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.368+0000 7f8d36ffd700 1 -- 172.21.15.3:0/3624048637 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c003dd0 con 0x7f8d480cf3c0 2024-05-08T21:22:50.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.368+0000 7f8d36ffd700 1 -- 172.21.15.3:0/3624048637 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3c010470 con 0x7f8d480cf3c0 2024-05-08T21:22:50.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.368+0000 7f8d4c888700 1 -- 172.21.15.3:0/3624048637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d48157be0 con 0x7f8d480cf3c0 2024-05-08T21:22:50.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.368+0000 7f8d36ffd700 1 -- 172.21.15.3:0/3624048637 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c010820 con 0x7f8d480cf3c0 2024-05-08T21:22:50.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.368+0000 7f8d4c888700 1 -- 172.21.15.3:0/3624048637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d481581a0 con 0x7f8d480cf3c0 2024-05-08T21:22:50.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.369+0000 7f8d4c888700 1 -- 172.21.15.3:0/3624048637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d48068000 con 0x7f8d480cf3c0 2024-05-08T21:22:50.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.370+0000 7f8d36ffd700 1 -- 172.21.15.3:0/3624048637 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8d3c019400 con 0x7f8d480cf3c0 2024-05-08T21:22:50.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.370+0000 7f8d36ffd700 1 --2- 172.21.15.3:0/3624048637 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f8d2806ac90 0x7f8d2806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:50.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.370+0000 7f8d36ffd700 1 -- 172.21.15.3:0/3624048637 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d3c08afc0 con 0x7f8d480cf3c0 2024-05-08T21:22:50.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.370+0000 7f8d4659c700 1 --2- 172.21.15.3:0/3624048637 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f8d2806ac90 0x7f8d2806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:50.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.371+0000 7f8d4659c700 1 --2- 172.21.15.3:0/3624048637 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f8d2806ac90 0x7f8d2806d150 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8d38000f60 tx=0x7f8d38009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:50.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.375+0000 7f8d36ffd700 1 -- 172.21.15.3:0/3624048637 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f8d3c056190 con 0x7f8d480cf3c0 2024-05-08T21:22:50.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.596+0000 7f8d4c888700 1 -- 172.21.15.3:0/3624048637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d4806c1e0 con 0x7f8d480cf3c0 2024-05-08T21:22:50.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.597+0000 7f8d36ffd700 1 -- 172.21.15.3:0/3624048637 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16066 (secure 0 0 0) 0x7f8d3c0597b0 con 0x7f8d480cf3c0 2024-05-08T21:22:50.600 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:50.601 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":33,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:22:50.183905+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:22:47.853786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-08T21:22:49.153759+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:50.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.599+0000 7f8d4c888700 1 -- 172.21.15.3:0/3624048637 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f8d2806ac90 msgr2=0x7f8d2806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:50.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.599+0000 7f8d4c888700 1 --2- 172.21.15.3:0/3624048637 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f8d2806ac90 0x7f8d2806d150 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8d38000f60 tx=0x7f8d38009040 comp rx=0 tx=0).stop 2024-05-08T21:22:50.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.599+0000 7f8d4c888700 1 -- 172.21.15.3:0/3624048637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d480cf3c0 msgr2=0x7f8d481493d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:50.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.599+0000 7f8d4c888700 1 --2- 172.21.15.3:0/3624048637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d480cf3c0 0x7f8d481493d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00b2c0 tx=0x7f8d3c004750 comp rx=0 tx=0).stop 2024-05-08T21:22:50.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.600+0000 7f8d4c888700 1 -- 172.21.15.3:0/3624048637 shutdown_connections 2024-05-08T21:22:50.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.600+0000 7f8d4c888700 1 --2- 172.21.15.3:0/3624048637 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d4815a380 0x7f8d48151880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.600+0000 7f8d4c888700 1 --2- 172.21.15.3:0/3624048637 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f8d2806ac90 0x7f8d2806d150 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.600+0000 7f8d4c888700 1 --2- 172.21.15.3:0/3624048637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d4806f4a0 0x7f8d48145780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.600+0000 7f8d4c888700 1 --2- 172.21.15.3:0/3624048637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d480cf3c0 0x7f8d481493d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.600+0000 7f8d4c888700 1 -- 172.21.15.3:0/3624048637 >> 172.21.15.3:0/3624048637 conn(0x7f8d480cfce0 msgr2=0x7f8d4806d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:50.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.600+0000 7f8d4c888700 1 -- 172.21.15.3:0/3624048637 shutdown_connections 2024-05-08T21:22:50.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.600+0000 7f8d4c888700 1 -- 172.21.15.3:0/3624048637 wait complete. 2024-05-08T21:22:50.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:50.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.751+0000 7f0b83067700 1 Processor -- start 2024-05-08T21:22:50.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.751+0000 7f0b83067700 1 -- start start 2024-05-08T21:22:50.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.752+0000 7f0b83067700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0b7c06f920 0x7f0b7c0cc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:50.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.752+0000 7f0b83067700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7c0cf530 con 0x7f0b7c06b070 2024-05-08T21:22:50.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.752+0000 7f0b83067700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7c0cf760 con 0x7f0b7c0cc470 2024-05-08T21:22:50.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.752+0000 7f0b83067700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7c0cfa40 con 0x7f0b7c06f920 2024-05-08T21:22:50.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.753+0000 7f0b81604700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0b7c06b070 0x7f0b7c15e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34678/0 (socket says 172.21.15.3:34678) 2024-05-08T21:22:50.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.753+0000 7f0b81604700 1 -- 172.21.15.3:0/3139065290 learned_addr learned my addr 172.21.15.3:0/3139065290 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:50.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.753+0000 7f0b80e03700 1 --2- 172.21.15.3:0/3139065290 >> v2:172.21.15.186:3300/0 conn(0x7f0b7c06f920 0x7f0b7c0cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:50.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.753+0000 7f0b737fe700 1 -- 172.21.15.3:0/3139065290 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801010789 0 0) 0x7f0b7c0cf530 con 0x7f0b7c06b070 2024-05-08T21:22:50.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.753+0000 7f0b737fe700 1 -- 172.21.15.3:0/3139065290 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b640035b0 con 0x7f0b7c06b070 2024-05-08T21:22:50.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.753+0000 7f0b737fe700 1 -- 172.21.15.3:0/3139065290 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152639358 0 0) 0x7f0b7c0cf760 con 0x7f0b7c0cc470 2024-05-08T21:22:50.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.753+0000 7f0b737fe700 1 -- 172.21.15.3:0/3139065290 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b7c0cf530 con 0x7f0b7c0cc470 2024-05-08T21:22:50.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.753+0000 7f0b80e03700 1 -- 172.21.15.3:0/3139065290 >> v1:172.21.15.186:6789/0 conn(0x7f0b7c0cc470 legacy=0x7f0b7c06a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:50.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.753+0000 7f0b80e03700 1 -- 172.21.15.3:0/3139065290 >> v1:172.21.15.3:6789/0 conn(0x7f0b7c06b070 legacy=0x7f0b7c15e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:50.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.753+0000 7f0b80e03700 1 -- 172.21.15.3:0/3139065290 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c0cfd20 con 0x7f0b7c06f920 2024-05-08T21:22:50.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.753+0000 7f0b80e03700 1 --2- 172.21.15.3:0/3139065290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b7c06f920 0x7f0b7c0cc970 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0b78009840 tx=0x7f0b78009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56cc9db24bd94cbb server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:50.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.753+0000 7f0b737fe700 1 -- 172.21.15.3:0/3139065290 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78011830 con 0x7f0b7c06f920 2024-05-08T21:22:50.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.754+0000 7f0b737fe700 1 -- 172.21.15.3:0/3139065290 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b78011990 con 0x7f0b7c06f920 2024-05-08T21:22:50.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.754+0000 7f0b737fe700 1 -- 172.21.15.3:0/3139065290 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78011d20 con 0x7f0b7c06f920 2024-05-08T21:22:50.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.754+0000 7f0b83067700 1 -- 172.21.15.3:0/3139065290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b7c06f920 msgr2=0x7f0b7c0cc970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:50.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.754+0000 7f0b83067700 1 --2- 172.21.15.3:0/3139065290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b7c06f920 0x7f0b7c0cc970 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0b78009840 tx=0x7f0b78009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:50.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.754+0000 7f0b83067700 1 -- 172.21.15.3:0/3139065290 shutdown_connections 2024-05-08T21:22:50.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.754+0000 7f0b83067700 1 --2- 172.21.15.3:0/3139065290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b7c06f920 0x7f0b7c0cc970 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.754+0000 7f0b83067700 1 -- 172.21.15.3:0/3139065290 >> 172.21.15.3:0/3139065290 conn(0x7f0b7c0b1e10 msgr2=0x7f0b7c0b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:50.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.754+0000 7f0b83067700 1 -- 172.21.15.3:0/3139065290 shutdown_connections 2024-05-08T21:22:50.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.754+0000 7f0b83067700 1 -- 172.21.15.3:0/3139065290 wait complete. 2024-05-08T21:22:50.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.756+0000 7f0b83067700 1 Processor -- start 2024-05-08T21:22:50.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.756+0000 7f0b83067700 1 -- start start 2024-05-08T21:22:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.757+0000 7f0b83067700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c06b070 0x7f0b7c1f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.757+0000 7f0b83067700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b7c06f920 0x7f0b7c0c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.757+0000 7f0b83067700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b7c0cc470 0x7f0b7c0b0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.757+0000 7f0b83067700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c1f31b0 con 0x7f0b7c06b070 2024-05-08T21:22:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.757+0000 7f0b83067700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0c7910 con 0x7f0b7c0cc470 2024-05-08T21:22:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.757+0000 7f0b83067700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c1f7720 con 0x7f0b7c06f920 2024-05-08T21:22:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.758+0000 7f0b80e03700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c06b070 0x7f0b7c1f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.758+0000 7f0b80e03700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c06b070 0x7f0b7c1f2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60096/0 (socket says 172.21.15.3:60096) 2024-05-08T21:22:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.758+0000 7f0b80e03700 1 -- 172.21.15.3:0/661561292 learned_addr learned my addr 172.21.15.3:0/661561292 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.758+0000 7f0b81604700 1 --2- 172.21.15.3:0/661561292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b7c0cc470 0x7f0b7c0b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.758+0000 7f0b73fff700 1 --2- 172.21.15.3:0/661561292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b7c06f920 0x7f0b7c0c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.758+0000 7f0b80e03700 1 -- 172.21.15.3:0/661561292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b7c06f920 msgr2=0x7f0b7c0c73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.758+0000 7f0b80e03700 1 --2- 172.21.15.3:0/661561292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b7c06f920 0x7f0b7c0c73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.758+0000 7f0b80e03700 1 -- 172.21.15.3:0/661561292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b7c0cc470 msgr2=0x7f0b7c0b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.758+0000 7f0b80e03700 1 --2- 172.21.15.3:0/661561292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b7c0cc470 0x7f0b7c0b0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.758+0000 7f0b80e03700 1 -- 172.21.15.3:0/661561292 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b780094a0 con 0x7f0b7c06b070 2024-05-08T21:22:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.758+0000 7f0b80e03700 1 --2- 172.21.15.3:0/661561292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c06b070 0x7f0b7c1f2c70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0b780090b0 tx=0x7f0b78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.758+0000 7f0b717fa700 1 -- 172.21.15.3:0/661561292 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7801a6e0 con 0x7f0b7c06b070 2024-05-08T21:22:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.758+0000 7f0b717fa700 1 -- 172.21.15.3:0/661561292 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7801a840 con 0x7f0b7c06b070 2024-05-08T21:22:50.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.758+0000 7f0b83067700 1 -- 172.21.15.3:0/661561292 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7c0c5b00 con 0x7f0b7c06b070 2024-05-08T21:22:50.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.758+0000 7f0b717fa700 1 -- 172.21.15.3:0/661561292 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7801abb0 con 0x7f0b7c06b070 2024-05-08T21:22:50.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.758+0000 7f0b83067700 1 -- 172.21.15.3:0/661561292 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c0c60c0 con 0x7f0b7c06b070 2024-05-08T21:22:50.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.760+0000 7f0b83067700 1 -- 172.21.15.3:0/661561292 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7c068000 con 0x7f0b7c06b070 2024-05-08T21:22:50.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.760+0000 7f0b717fa700 1 -- 172.21.15.3:0/661561292 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0b78022070 con 0x7f0b7c06b070 2024-05-08T21:22:50.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.761+0000 7f0b717fa700 1 --2- 172.21.15.3:0/661561292 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0b6806ac90 0x7f0b6806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:50.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.761+0000 7f0b717fa700 1 -- 172.21.15.3:0/661561292 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b78089e20 con 0x7f0b7c06b070 2024-05-08T21:22:50.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.761+0000 7f0b73fff700 1 --2- 172.21.15.3:0/661561292 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0b6806ac90 0x7f0b6806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:50.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.761+0000 7f0b73fff700 1 --2- 172.21.15.3:0/661561292 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0b6806ac90 0x7f0b6806d150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c0b5a30 tx=0x7f0b6c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:50.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.765+0000 7f0b717fa700 1 -- 172.21.15.3:0/661561292 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0b78055260 con 0x7f0b7c06b070 2024-05-08T21:22:50.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.984+0000 7f0b83067700 1 -- 172.21.15.3:0/661561292 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b7c0b6280 con 0x7f0b7c06b070 2024-05-08T21:22:50.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.985+0000 7f0b717fa700 1 -- 172.21.15.3:0/661561292 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16066 (secure 0 0 0) 0x7f0b78029090 con 0x7f0b7c06b070 2024-05-08T21:22:50.988 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:50.988 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":33,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:22:50.183905+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:22:47.853786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-08T21:22:49.153759+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:50.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.987+0000 7f0b83067700 1 -- 172.21.15.3:0/661561292 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0b6806ac90 msgr2=0x7f0b6806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:50.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.987+0000 7f0b83067700 1 --2- 172.21.15.3:0/661561292 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0b6806ac90 0x7f0b6806d150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c0b5a30 tx=0x7f0b6c009040 comp rx=0 tx=0).stop 2024-05-08T21:22:50.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.987+0000 7f0b83067700 1 -- 172.21.15.3:0/661561292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c06b070 msgr2=0x7f0b7c1f2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:50.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.987+0000 7f0b83067700 1 --2- 172.21.15.3:0/661561292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c06b070 0x7f0b7c1f2c70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0b780090b0 tx=0x7f0b78004750 comp rx=0 tx=0).stop 2024-05-08T21:22:50.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.987+0000 7f0b83067700 1 -- 172.21.15.3:0/661561292 shutdown_connections 2024-05-08T21:22:50.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.987+0000 7f0b83067700 1 --2- 172.21.15.3:0/661561292 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0b6806ac90 0x7f0b6806d150 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.987+0000 7f0b83067700 1 --2- 172.21.15.3:0/661561292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b7c0cc470 0x7f0b7c0b0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.987+0000 7f0b83067700 1 --2- 172.21.15.3:0/661561292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c06b070 0x7f0b7c1f2c70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.987+0000 7f0b83067700 1 --2- 172.21.15.3:0/661561292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b7c06f920 0x7f0b7c0c73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:50.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.987+0000 7f0b83067700 1 -- 172.21.15.3:0/661561292 >> 172.21.15.3:0/661561292 conn(0x7f0b7c0b1e10 msgr2=0x7f0b7c06a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:50.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.987+0000 7f0b83067700 1 -- 172.21.15.3:0/661561292 shutdown_connections 2024-05-08T21:22:50.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:50.987+0000 7f0b83067700 1 -- 172.21.15.3:0/661561292 wait complete. 2024-05-08T21:22:51.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-08T21:22:51.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.140+0000 7f88e0e36700 1 Processor -- start 2024-05-08T21:22:51.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.140+0000 7f88e0e36700 1 -- start start 2024-05-08T21:22:51.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.141+0000 7f88e0e36700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f88dc14b790 0x7f88dc0c43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:51.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.141+0000 7f88e0e36700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88dc0ae9d0 0x7f88dc16bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:51.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.141+0000 7f88e0e36700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88dc0c4da0 con 0x7f88dc0c4920 2024-05-08T21:22:51.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.141+0000 7f88e0e36700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88dc0b00b0 con 0x7f88dc14b790 2024-05-08T21:22:51.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.141+0000 7f88e0e36700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88dc0af010 con 0x7f88dc0ae9d0 2024-05-08T21:22:51.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.142+0000 7f88da59c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f88dc14b790 0x7f88dc0c43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:51.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.142+0000 7f88dad9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88dc0ae9d0 0x7f88dc16bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:51.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.142+0000 7f88d9d9b700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f88dc0c4920 0x7f88dc0ae2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:52372/0 (socket says 172.21.15.3:52372) 2024-05-08T21:22:51.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.142+0000 7f88da59c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f88dc14b790 0x7f88dc0c43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:49712/0 (socket says 172.21.15.3:49712) 2024-05-08T21:22:51.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.142+0000 7f88d9d9b700 1 -- 172.21.15.3:0/2578052985 learned_addr learned my addr 172.21.15.3:0/2578052985 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.142+0000 7f88da59c700 1 -- 172.21.15.3:0/2578052985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88dc0ae9d0 msgr2=0x7f88dc16bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.142+0000 7f88da59c700 1 --2- 172.21.15.3:0/2578052985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88dc0ae9d0 0x7f88dc16bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.142+0000 7f88da59c700 1 -- 172.21.15.3:0/2578052985 >> v1:172.21.15.186:6789/0 conn(0x7f88dc0c4920 legacy=0x7f88dc0ae2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.142+0000 7f88d959a700 1 -- 172.21.15.3:0/2578052985 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921057047 0 0) 0x7f88d0001110 con 0x7f88dc0c4920 2024-05-08T21:22:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.142+0000 7f88da59c700 1 -- 172.21.15.3:0/2578052985 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88dc0af2f0 con 0x7f88dc14b790 2024-05-08T21:22:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.142+0000 7f88dad9d700 1 --2- 172.21.15.3:0/2578052985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88dc0ae9d0 0x7f88dc16bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.142+0000 7f88da59c700 1 --2- 172.21.15.3:0/2578052985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88dc14b790 0x7f88dc0c43e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f88cc009840 tx=0x7f88cc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e2b1a6466f7c135 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.142+0000 7f88d959a700 1 -- 172.21.15.3:0/2578052985 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88cc011830 con 0x7f88dc14b790 2024-05-08T21:22:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.142+0000 7f88d959a700 1 -- 172.21.15.3:0/2578052985 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88cc011990 con 0x7f88dc14b790 2024-05-08T21:22:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.142+0000 7f88d959a700 1 -- 172.21.15.3:0/2578052985 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88cc011d20 con 0x7f88dc14b790 2024-05-08T21:22:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.143+0000 7f88e0e36700 1 -- 172.21.15.3:0/2578052985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88dc14b790 msgr2=0x7f88dc0c43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.143+0000 7f88e0e36700 1 --2- 172.21.15.3:0/2578052985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88dc14b790 0x7f88dc0c43e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f88cc009840 tx=0x7f88cc009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.143+0000 7f88e0e36700 1 -- 172.21.15.3:0/2578052985 shutdown_connections 2024-05-08T21:22:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.143+0000 7f88e0e36700 1 --2- 172.21.15.3:0/2578052985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88dc0ae9d0 0x7f88dc16bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.143+0000 7f88e0e36700 1 --2- 172.21.15.3:0/2578052985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88dc14b790 0x7f88dc0c43e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.143+0000 7f88e0e36700 1 -- 172.21.15.3:0/2578052985 >> 172.21.15.3:0/2578052985 conn(0x7f88dc08afe0 msgr2=0x7f88dc08b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.143+0000 7f88e0e36700 1 -- 172.21.15.3:0/2578052985 shutdown_connections 2024-05-08T21:22:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.143+0000 7f88e0e36700 1 -- 172.21.15.3:0/2578052985 wait complete. 2024-05-08T21:22:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.145+0000 7f88e0e36700 1 Processor -- start 2024-05-08T21:22:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.145+0000 7f88e0e36700 1 -- start start 2024-05-08T21:22:51.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.145+0000 7f88e0e36700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88dc0ae9d0 0x7f88dc164ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:51.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.145+0000 7f88e0e36700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88dc0c4920 0x7f88dc163750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:51.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.145+0000 7f88e0e36700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88dc14b790 0x7f88dc0c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:51.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.145+0000 7f88e0e36700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88dc0c6800 con 0x7f88dc0ae9d0 2024-05-08T21:22:51.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.145+0000 7f88e0e36700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88dc163e50 con 0x7f88dc0c4920 2024-05-08T21:22:51.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.145+0000 7f88e0e36700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88dc164130 con 0x7f88dc14b790 2024-05-08T21:22:51.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.146+0000 7f88da59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88dc0ae9d0 0x7f88dc164ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:51.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.146+0000 7f88da59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88dc0ae9d0 0x7f88dc164ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60108/0 (socket says 172.21.15.3:60108) 2024-05-08T21:22:51.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.146+0000 7f88da59c700 1 -- 172.21.15.3:0/1789197973 learned_addr learned my addr 172.21.15.3:0/1789197973 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:51.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.146+0000 7f88dad9d700 1 --2- 172.21.15.3:0/1789197973 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88dc14b790 0x7f88dc0c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:51.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.146+0000 7f88d9d9b700 1 --2- 172.21.15.3:0/1789197973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88dc0c4920 0x7f88dc163750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:51.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.146+0000 7f88da59c700 1 -- 172.21.15.3:0/1789197973 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88dc14b790 msgr2=0x7f88dc0c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:51.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.146+0000 7f88da59c700 1 --2- 172.21.15.3:0/1789197973 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88dc14b790 0x7f88dc0c62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:51.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.146+0000 7f88da59c700 1 -- 172.21.15.3:0/1789197973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88dc0c4920 msgr2=0x7f88dc163750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:51.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.146+0000 7f88da59c700 1 --2- 172.21.15.3:0/1789197973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88dc0c4920 0x7f88dc163750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:51.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.146+0000 7f88da59c700 1 -- 172.21.15.3:0/1789197973 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d4009a70 con 0x7f88dc0ae9d0 2024-05-08T21:22:51.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.146+0000 7f88da59c700 1 --2- 172.21.15.3:0/1789197973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88dc0ae9d0 0x7f88dc164ef0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f88cc009fd0 tx=0x7f88cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:51.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.146+0000 7f88caffd700 1 -- 172.21.15.3:0/1789197973 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88cc0045d0 con 0x7f88dc0ae9d0 2024-05-08T21:22:51.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.146+0000 7f88caffd700 1 -- 172.21.15.3:0/1789197973 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88cc01f070 con 0x7f88dc0ae9d0 2024-05-08T21:22:51.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.146+0000 7f88e0e36700 1 -- 172.21.15.3:0/1789197973 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88cc0094a0 con 0x7f88dc0ae9d0 2024-05-08T21:22:51.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.146+0000 7f88caffd700 1 -- 172.21.15.3:0/1789197973 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88cc0038a0 con 0x7f88dc0ae9d0 2024-05-08T21:22:51.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.146+0000 7f88e0e36700 1 -- 172.21.15.3:0/1789197973 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88dc1646f0 con 0x7f88dc0ae9d0 2024-05-08T21:22:51.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.147+0000 7f88e0e36700 1 -- 172.21.15.3:0/1789197973 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88dc161b00 con 0x7f88dc0ae9d0 2024-05-08T21:22:51.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.148+0000 7f88caffd700 1 -- 172.21.15.3:0/1789197973 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f88cc003a00 con 0x7f88dc0ae9d0 2024-05-08T21:22:51.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.148+0000 7f88caffd700 1 --2- 172.21.15.3:0/1789197973 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f88bc06ac90 0x7f88bc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:51.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.149+0000 7f88caffd700 1 -- 172.21.15.3:0/1789197973 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88cc021070 con 0x7f88dc0ae9d0 2024-05-08T21:22:51.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.149+0000 7f88d9d9b700 1 --2- 172.21.15.3:0/1789197973 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f88bc06ac90 0x7f88bc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:51.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.149+0000 7f88d9d9b700 1 --2- 172.21.15.3:0/1789197973 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f88bc06ac90 0x7f88bc06d150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f88dc06c770 tx=0x7f88d0007c20 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:51.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.154+0000 7f88caffd700 1 -- 172.21.15.3:0/1789197973 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f88cc055dc0 con 0x7f88dc0ae9d0 2024-05-08T21:22:51.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:51.420+0000 7f88e0e36700 1 -- 172.21.15.3:0/1789197973 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f88dc074bc0 con 0x7f88dc0ae9d0 2024-05-08T21:22:52.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.192+0000 7f88caffd700 1 -- 172.21.15.3:0/1789197973 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f88cc003cb0 con 0x7f88dc0ae9d0 2024-05-08T21:22:52.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.196+0000 7f88e0e36700 1 -- 172.21.15.3:0/1789197973 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f88bc06ac90 msgr2=0x7f88bc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:52.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.196+0000 7f88e0e36700 1 --2- 172.21.15.3:0/1789197973 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f88bc06ac90 0x7f88bc06d150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f88dc06c770 tx=0x7f88d0007c20 comp rx=0 tx=0).stop 2024-05-08T21:22:52.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.196+0000 7f88e0e36700 1 -- 172.21.15.3:0/1789197973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88dc0ae9d0 msgr2=0x7f88dc164ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:52.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.196+0000 7f88e0e36700 1 --2- 172.21.15.3:0/1789197973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88dc0ae9d0 0x7f88dc164ef0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f88cc009fd0 tx=0x7f88cc004750 comp rx=0 tx=0).stop 2024-05-08T21:22:52.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.196+0000 7f88e0e36700 1 -- 172.21.15.3:0/1789197973 shutdown_connections 2024-05-08T21:22:52.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.196+0000 7f88e0e36700 1 --2- 172.21.15.3:0/1789197973 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88dc14b790 0x7f88dc0c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:52.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.196+0000 7f88e0e36700 1 --2- 172.21.15.3:0/1789197973 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f88bc06ac90 0x7f88bc06d150 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:52.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.196+0000 7f88e0e36700 1 --2- 172.21.15.3:0/1789197973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88dc0ae9d0 0x7f88dc164ef0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:52.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.196+0000 7f88e0e36700 1 --2- 172.21.15.3:0/1789197973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88dc0c4920 0x7f88dc163750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:52.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.196+0000 7f88e0e36700 1 -- 172.21.15.3:0/1789197973 >> 172.21.15.3:0/1789197973 conn(0x7f88dc08afe0 msgr2=0x7f88dc15fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:52.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.196+0000 7f88e0e36700 1 -- 172.21.15.3:0/1789197973 shutdown_connections 2024-05-08T21:22:52.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.196+0000 7f88e0e36700 1 -- 172.21.15.3:0/1789197973 wait complete. 2024-05-08T21:22:52.213 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-08T21:22:52.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-08T21:22:52.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.354+0000 7fd20a99d700 1 Processor -- start 2024-05-08T21:22:52.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.355+0000 7fd20a99d700 1 -- start start 2024-05-08T21:22:52.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.355+0000 7fd20a99d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd2040d18d0 0x7fd204070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:52.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.355+0000 7fd20a99d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd204070a20 con 0x7fd204144510 2024-05-08T21:22:52.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.355+0000 7fd20a99d700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd204070d00 con 0x7fd2040b44d0 2024-05-08T21:22:52.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.355+0000 7fd20a99d700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd204070fe0 con 0x7fd2040d18d0 2024-05-08T21:22:52.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.356+0000 7fd20999b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd204144510 0x7fd2040d0a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59768/0 (socket says 172.21.15.3:59768) 2024-05-08T21:22:52.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.356+0000 7fd20999b700 1 -- 172.21.15.3:0/1075704711 learned_addr learned my addr 172.21.15.3:0/1075704711 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:52.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.356+0000 7fd20a19c700 1 --2- 172.21.15.3:0/1075704711 >> v2:172.21.15.186:3300/0 conn(0x7fd2040d18d0 0x7fd204070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:52.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.356+0000 7fd208999700 1 -- 172.21.15.3:0/1075704711 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259269209 0 0) 0x7fd204070a20 con 0x7fd204144510 2024-05-08T21:22:52.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.356+0000 7fd208999700 1 -- 172.21.15.3:0/1075704711 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1f00035b0 con 0x7fd204144510 2024-05-08T21:22:52.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.356+0000 7fd20a19c700 1 -- 172.21.15.3:0/1075704711 >> v1:172.21.15.186:6789/0 conn(0x7fd2040b44d0 legacy=0x7fd2040d11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:52.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.356+0000 7fd20a19c700 1 -- 172.21.15.3:0/1075704711 >> v1:172.21.15.3:6789/0 conn(0x7fd204144510 legacy=0x7fd2040d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:52.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.356+0000 7fd20a19c700 1 -- 172.21.15.3:0/1075704711 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2040712c0 con 0x7fd2040d18d0 2024-05-08T21:22:52.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.356+0000 7fd20a19c700 1 --2- 172.21.15.3:0/1075704711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2040d18d0 0x7fd204070440 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009d90 tx=0x7fd1fc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc6518ee30fe0ee1 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:52.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.357+0000 7fd208999700 1 -- 172.21.15.3:0/1075704711 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc00e070 con 0x7fd2040d18d0 2024-05-08T21:22:52.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.357+0000 7fd208999700 1 -- 172.21.15.3:0/1075704711 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1fc00bb20 con 0x7fd2040d18d0 2024-05-08T21:22:52.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.357+0000 7fd208999700 1 -- 172.21.15.3:0/1075704711 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc012430 con 0x7fd2040d18d0 2024-05-08T21:22:52.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.357+0000 7fd20a99d700 1 -- 172.21.15.3:0/1075704711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2040d18d0 msgr2=0x7fd204070440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:52.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.357+0000 7fd20a99d700 1 --2- 172.21.15.3:0/1075704711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2040d18d0 0x7fd204070440 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009d90 tx=0x7fd1fc00b330 comp rx=0 tx=0).stop 2024-05-08T21:22:52.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.357+0000 7fd20a99d700 1 -- 172.21.15.3:0/1075704711 shutdown_connections 2024-05-08T21:22:52.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.357+0000 7fd20a99d700 1 --2- 172.21.15.3:0/1075704711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2040d18d0 0x7fd204070440 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:52.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.357+0000 7fd20a99d700 1 -- 172.21.15.3:0/1075704711 >> 172.21.15.3:0/1075704711 conn(0x7fd20406ece0 msgr2=0x7fd20406f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:52.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.357+0000 7fd20a99d700 1 -- 172.21.15.3:0/1075704711 shutdown_connections 2024-05-08T21:22:52.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.357+0000 7fd20a99d700 1 -- 172.21.15.3:0/1075704711 wait complete. 2024-05-08T21:22:52.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.359+0000 7fd20a99d700 1 Processor -- start 2024-05-08T21:22:52.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.359+0000 7fd20a99d700 1 -- start start 2024-05-08T21:22:52.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.359+0000 7fd20a99d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2040b44d0 0x7fd20413d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:52.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.359+0000 7fd20a99d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2040d18d0 0x7fd2040b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:52.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.359+0000 7fd20a99d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd204144510 0x7fd2040b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:52.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.359+0000 7fd20a99d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2040b23b0 con 0x7fd204144510 2024-05-08T21:22:52.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.359+0000 7fd20a99d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2040b2840 con 0x7fd2040b44d0 2024-05-08T21:22:52.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.359+0000 7fd20a99d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2040b17a0 con 0x7fd2040d18d0 2024-05-08T21:22:52.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.359+0000 7fd20a19c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd204144510 0x7fd2040b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:52.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.359+0000 7fd20a19c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd204144510 0x7fd2040b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45468/0 (socket says 172.21.15.3:45468) 2024-05-08T21:22:52.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.359+0000 7fd20a19c700 1 -- 172.21.15.3:0/399038720 learned_addr learned my addr 172.21.15.3:0/399038720 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:52.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.360+0000 7fd20999b700 1 --2- 172.21.15.3:0/399038720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2040b44d0 0x7fd20413d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:52.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.360+0000 7fd20919a700 1 --2- 172.21.15.3:0/399038720 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2040d18d0 0x7fd2040b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:52.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.360+0000 7fd20a19c700 1 -- 172.21.15.3:0/399038720 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2040d18d0 msgr2=0x7fd2040b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:52.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.360+0000 7fd20a19c700 1 --2- 172.21.15.3:0/399038720 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2040d18d0 0x7fd2040b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:52.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.360+0000 7fd20a19c700 1 -- 172.21.15.3:0/399038720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2040b44d0 msgr2=0x7fd20413d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:52.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.360+0000 7fd20a19c700 1 --2- 172.21.15.3:0/399038720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2040b44d0 0x7fd20413d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:52.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.360+0000 7fd20a19c700 1 -- 172.21.15.3:0/399038720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1fc009a70 con 0x7fd204144510 2024-05-08T21:22:52.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.360+0000 7fd20a19c700 1 --2- 172.21.15.3:0/399038720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd204144510 0x7fd2040b31d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc014040 tx=0x7fd1fc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:52.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.360+0000 7fd1f67fc700 1 -- 172.21.15.3:0/399038720 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc012430 con 0x7fd204144510 2024-05-08T21:22:52.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.360+0000 7fd1f67fc700 1 -- 172.21.15.3:0/399038720 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1fc020070 con 0x7fd204144510 2024-05-08T21:22:52.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.360+0000 7fd20a99d700 1 -- 172.21.15.3:0/399038720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2040b1a80 con 0x7fd204144510 2024-05-08T21:22:52.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.360+0000 7fd1f67fc700 1 -- 172.21.15.3:0/399038720 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc004280 con 0x7fd204144510 2024-05-08T21:22:52.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.360+0000 7fd20a99d700 1 -- 172.21.15.3:0/399038720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2040b2040 con 0x7fd204144510 2024-05-08T21:22:52.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.361+0000 7fd20a99d700 1 -- 172.21.15.3:0/399038720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd20404ec60 con 0x7fd204144510 2024-05-08T21:22:52.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.362+0000 7fd1f67fc700 1 -- 172.21.15.3:0/399038720 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd1fc0043e0 con 0x7fd204144510 2024-05-08T21:22:52.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.362+0000 7fd1f67fc700 1 --2- 172.21.15.3:0/399038720 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1e806ac90 0x7fd1e806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:52.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.362+0000 7fd1f67fc700 1 -- 172.21.15.3:0/399038720 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1fc022070 con 0x7fd204144510 2024-05-08T21:22:52.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.363+0000 7fd20999b700 1 --2- 172.21.15.3:0/399038720 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1e806ac90 0x7fd1e806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:52.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.363+0000 7fd20999b700 1 --2- 172.21.15.3:0/399038720 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1e806ac90 0x7fd1e806d150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd200007fd0 tx=0x7fd200007dc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:52.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.368+0000 7fd1f67fc700 1 -- 172.21.15.3:0/399038720 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd1fc055fc0 con 0x7fd204144510 2024-05-08T21:22:52.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:52.640+0000 7fd20a99d700 1 -- 172.21.15.3:0/399038720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd20415ef10 con 0x7fd204144510 2024-05-08T21:22:52.706 INFO:tasks.ceph.mds.a.smithi003.stderr:2024-05-08T21:22:52.704+0000 7ff476db9700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-08T21:22:53.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.198+0000 7fd1f67fc700 1 -- 172.21.15.3:0/399038720 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd1fc0595e0 con 0x7fd204144510 2024-05-08T21:22:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.201+0000 7fd20a99d700 1 -- 172.21.15.3:0/399038720 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1e806ac90 msgr2=0x7fd1e806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.201+0000 7fd20a99d700 1 --2- 172.21.15.3:0/399038720 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1e806ac90 0x7fd1e806d150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd200007fd0 tx=0x7fd200007dc0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.201+0000 7fd20a99d700 1 -- 172.21.15.3:0/399038720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd204144510 msgr2=0x7fd2040b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.201+0000 7fd20a99d700 1 --2- 172.21.15.3:0/399038720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd204144510 0x7fd2040b31d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc014040 tx=0x7fd1fc004ad0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.201+0000 7fd20a99d700 1 -- 172.21.15.3:0/399038720 shutdown_connections 2024-05-08T21:22:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.201+0000 7fd20a99d700 1 --2- 172.21.15.3:0/399038720 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd1e806ac90 0x7fd1e806d150 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.201+0000 7fd20a99d700 1 --2- 172.21.15.3:0/399038720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd204144510 0x7fd2040b31d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.201+0000 7fd20a99d700 1 --2- 172.21.15.3:0/399038720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2040b44d0 0x7fd20413d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.201+0000 7fd20a99d700 1 --2- 172.21.15.3:0/399038720 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2040d18d0 0x7fd2040b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.201+0000 7fd20a99d700 1 -- 172.21.15.3:0/399038720 >> 172.21.15.3:0/399038720 conn(0x7fd20406ece0 msgr2=0x7fd20415f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.201+0000 7fd20a99d700 1 -- 172.21.15.3:0/399038720 shutdown_connections 2024-05-08T21:22:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.201+0000 7fd20a99d700 1 -- 172.21.15.3:0/399038720 wait complete. 2024-05-08T21:22:53.218 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-08T21:22:53.218 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:22:53.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.357+0000 7fd94a092700 1 Processor -- start 2024-05-08T21:22:53.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.357+0000 7fd94a092700 1 -- start start 2024-05-08T21:22:53.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.358+0000 7fd94a092700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd9440cc590 0x7fd94406f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:53.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.358+0000 7fd94a092700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9440cb160 con 0x7fd94415db00 2024-05-08T21:22:53.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.358+0000 7fd94a092700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94406ad20 con 0x7fd94406f7c0 2024-05-08T21:22:53.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.358+0000 7fd94a092700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd94406fc50 con 0x7fd9440cc590 2024-05-08T21:22:53.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.358+0000 7fd9437fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd94415db00 0x7fd9440caa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59782/0 (socket says 172.21.15.3:59782) 2024-05-08T21:22:53.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.358+0000 7fd9437fe700 1 -- 172.21.15.3:0/3366899575 learned_addr learned my addr 172.21.15.3:0/3366899575 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:53.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.358+0000 7fd942ffd700 1 --2- 172.21.15.3:0/3366899575 >> v2:172.21.15.186:3300/0 conn(0x7fd9440cc590 0x7fd94406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:53.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.358+0000 7fd9427fc700 1 -- 172.21.15.3:0/3366899575 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440607431 0 0) 0x7fd9440cb160 con 0x7fd94415db00 2024-05-08T21:22:53.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.358+0000 7fd9427fc700 1 -- 172.21.15.3:0/3366899575 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9300035b0 con 0x7fd94415db00 2024-05-08T21:22:53.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.359+0000 7fd9427fc700 1 -- 172.21.15.3:0/3366899575 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370069875 0 0) 0x7fd94406ad20 con 0x7fd94406f7c0 2024-05-08T21:22:53.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.359+0000 7fd9427fc700 1 -- 172.21.15.3:0/3366899575 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9440cb160 con 0x7fd94406f7c0 2024-05-08T21:22:53.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.359+0000 7fd9427fc700 1 -- 172.21.15.3:0/3366899575 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2522990554 0 0) 0x7fd9300035b0 con 0x7fd94415db00 2024-05-08T21:22:53.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.359+0000 7fd942ffd700 1 -- 172.21.15.3:0/3366899575 >> v1:172.21.15.186:6789/0 conn(0x7fd94406f7c0 legacy=0x7fd94406e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:53.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.359+0000 7fd942ffd700 1 -- 172.21.15.3:0/3366899575 >> v1:172.21.15.3:6789/0 conn(0x7fd94415db00 legacy=0x7fd9440caa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:53.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.359+0000 7fd942ffd700 1 -- 172.21.15.3:0/3366899575 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94406b000 con 0x7fd9440cc590 2024-05-08T21:22:53.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.359+0000 7fd942ffd700 1 --2- 172.21.15.3:0/3366899575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9440cc590 0x7fd94406f280 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd92c009840 tx=0x7fd92c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65cf87452037199f server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:53.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.359+0000 7fd9427fc700 1 -- 172.21.15.3:0/3366899575 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd934001670 con 0x7fd9440cc590 2024-05-08T21:22:53.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.359+0000 7fd9427fc700 1 -- 172.21.15.3:0/3366899575 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92c004030 con 0x7fd9440cc590 2024-05-08T21:22:53.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.359+0000 7fd9427fc700 1 -- 172.21.15.3:0/3366899575 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c0043c0 con 0x7fd9440cc590 2024-05-08T21:22:53.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.360+0000 7fd94a092700 1 -- 172.21.15.3:0/3366899575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9440cc590 msgr2=0x7fd94406f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:53.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.360+0000 7fd94a092700 1 --2- 172.21.15.3:0/3366899575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9440cc590 0x7fd94406f280 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd92c009840 tx=0x7fd92c009b50 comp rx=0 tx=0).stop 2024-05-08T21:22:53.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.360+0000 7fd94a092700 1 -- 172.21.15.3:0/3366899575 shutdown_connections 2024-05-08T21:22:53.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.360+0000 7fd94a092700 1 --2- 172.21.15.3:0/3366899575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9440cc590 0x7fd94406f280 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.360+0000 7fd94a092700 1 -- 172.21.15.3:0/3366899575 >> 172.21.15.3:0/3366899575 conn(0x7fd944099a50 msgr2=0x7fd9440d0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:53.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.360+0000 7fd94a092700 1 -- 172.21.15.3:0/3366899575 shutdown_connections 2024-05-08T21:22:53.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.360+0000 7fd94a092700 1 -- 172.21.15.3:0/3366899575 wait complete. 2024-05-08T21:22:53.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.361+0000 7fd94a092700 1 Processor -- start 2024-05-08T21:22:53.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.361+0000 7fd94a092700 1 -- start start 2024-05-08T21:22:53.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.362+0000 7fd94a092700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94406f7c0 0x7fd9440c5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:53.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.362+0000 7fd94a092700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9440cc590 0x7fd9440c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:53.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.362+0000 7fd94a092700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94415db00 0x7fd9440b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:53.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.362+0000 7fd94a092700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9441513d0 con 0x7fd94415db00 2024-05-08T21:22:53.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.362+0000 7fd94a092700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9440c7300 con 0x7fd94406f7c0 2024-05-08T21:22:53.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.362+0000 7fd94a092700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9440c5f90 con 0x7fd9440cc590 2024-05-08T21:22:53.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.362+0000 7fd943fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94415db00 0x7fd9440b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:53.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.362+0000 7fd943fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94415db00 0x7fd9440b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45482/0 (socket says 172.21.15.3:45482) 2024-05-08T21:22:53.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.362+0000 7fd943fff700 1 -- 172.21.15.3:0/325176643 learned_addr learned my addr 172.21.15.3:0/325176643 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:53.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.363+0000 7fd942ffd700 1 --2- 172.21.15.3:0/325176643 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9440cc590 0x7fd9440c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:53.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.363+0000 7fd9437fe700 1 --2- 172.21.15.3:0/325176643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94406f7c0 0x7fd9440c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:53.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.363+0000 7fd943fff700 1 -- 172.21.15.3:0/325176643 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9440cc590 msgr2=0x7fd9440c5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:53.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.363+0000 7fd943fff700 1 --2- 172.21.15.3:0/325176643 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9440cc590 0x7fd9440c5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.363+0000 7fd943fff700 1 -- 172.21.15.3:0/325176643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94406f7c0 msgr2=0x7fd9440c5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:53.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.363+0000 7fd943fff700 1 --2- 172.21.15.3:0/325176643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94406f7c0 0x7fd9440c5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.363+0000 7fd943fff700 1 -- 172.21.15.3:0/325176643 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92c0094a0 con 0x7fd94415db00 2024-05-08T21:22:53.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.363+0000 7fd943fff700 1 --2- 172.21.15.3:0/325176643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94415db00 0x7fd9440b04f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd93c00cc50 tx=0x7fd93c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:53.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.363+0000 7fd93bfff700 1 -- 172.21.15.3:0/325176643 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c00ada0 con 0x7fd94415db00 2024-05-08T21:22:53.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.363+0000 7fd93bfff700 1 -- 172.21.15.3:0/325176643 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93c00ce50 con 0x7fd94415db00 2024-05-08T21:22:53.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.363+0000 7fd94a092700 1 -- 172.21.15.3:0/325176643 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9440c6260 con 0x7fd94415db00 2024-05-08T21:22:53.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.363+0000 7fd93bfff700 1 -- 172.21.15.3:0/325176643 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c003c40 con 0x7fd94415db00 2024-05-08T21:22:53.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.363+0000 7fd94a092700 1 -- 172.21.15.3:0/325176643 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9440c6820 con 0x7fd94415db00 2024-05-08T21:22:53.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.364+0000 7fd94a092700 1 -- 172.21.15.3:0/325176643 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd944068000 con 0x7fd94415db00 2024-05-08T21:22:53.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.365+0000 7fd93bfff700 1 -- 172.21.15.3:0/325176643 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd93c0050d0 con 0x7fd94415db00 2024-05-08T21:22:53.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.365+0000 7fd93bfff700 1 --2- 172.21.15.3:0/325176643 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd92406ac90 0x7fd92406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:53.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.365+0000 7fd93bfff700 1 -- 172.21.15.3:0/325176643 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd93c0894a0 con 0x7fd94415db00 2024-05-08T21:22:53.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.365+0000 7fd9437fe700 1 --2- 172.21.15.3:0/325176643 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd92406ac90 0x7fd92406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:53.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.366+0000 7fd9437fe700 1 --2- 172.21.15.3:0/325176643 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd92406ac90 0x7fd92406d150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd9440c7660 tx=0x7fd934001490 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:53.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.370+0000 7fd93bfff700 1 -- 172.21.15.3:0/325176643 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd93c054680 con 0x7fd94415db00 2024-05-08T21:22:53.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.633+0000 7fd94a092700 1 -- 172.21.15.3:0/325176643 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd94406e220 con 0x7fd94415db00 2024-05-08T21:22:53.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.634+0000 7fd93bfff700 1 -- 172.21.15.3:0/325176643 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4734 (secure 0 0 0) 0x7fd93c057ca0 con 0x7fd94415db00 2024-05-08T21:22:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:53.635 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/3571929018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3571929018},{"type":"v1","addr":"172.21.15.186:6836","nonce":3571929018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4760,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3520732046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3520732046},{"type":"v1","addr":"172.21.15.186:6837","nonce":3520732046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4957,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2579650460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":2579650460},{"type":"v1","addr":"172.21.15.3:6835","nonce":2579650460}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:22:50.184245+0000","modified":"2024-05-08T21:22:53.192328+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.3:6834/3599481370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":3599481370},{"type":"v1","addr":"172.21.15.3:6834","nonce":3599481370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-05-08T21:22:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.636+0000 7fd94a092700 1 -- 172.21.15.3:0/325176643 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd92406ac90 msgr2=0x7fd92406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.636+0000 7fd94a092700 1 --2- 172.21.15.3:0/325176643 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd92406ac90 0x7fd92406d150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd9440c7660 tx=0x7fd934001490 comp rx=0 tx=0).stop 2024-05-08T21:22:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.636+0000 7fd94a092700 1 -- 172.21.15.3:0/325176643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94415db00 msgr2=0x7fd9440b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.636+0000 7fd94a092700 1 --2- 172.21.15.3:0/325176643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94415db00 0x7fd9440b04f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd93c00cc50 tx=0x7fd93c00a340 comp rx=0 tx=0).stop 2024-05-08T21:22:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.636+0000 7fd94a092700 1 -- 172.21.15.3:0/325176643 shutdown_connections 2024-05-08T21:22:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.636+0000 7fd94a092700 1 --2- 172.21.15.3:0/325176643 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd92406ac90 0x7fd92406d150 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.636+0000 7fd94a092700 1 --2- 172.21.15.3:0/325176643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94415db00 0x7fd9440b04f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.636+0000 7fd94a092700 1 --2- 172.21.15.3:0/325176643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94406f7c0 0x7fd9440c5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.636+0000 7fd94a092700 1 --2- 172.21.15.3:0/325176643 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9440cc590 0x7fd9440c5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.636+0000 7fd94a092700 1 -- 172.21.15.3:0/325176643 >> 172.21.15.3:0/325176643 conn(0x7fd944099a50 msgr2=0x7fd9440caa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.636+0000 7fd94a092700 1 -- 172.21.15.3:0/325176643 shutdown_connections 2024-05-08T21:22:53.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.636+0000 7fd94a092700 1 -- 172.21.15.3:0/325176643 wait complete. 2024-05-08T21:22:53.639 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 16 2024-05-08T21:22:53.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:53.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.786+0000 7f0fd98dd700 1 Processor -- start 2024-05-08T21:22:53.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.786+0000 7f0fd98dd700 1 -- start start 2024-05-08T21:22:53.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.787+0000 7f0fd98dd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0fd40d23b0 0x7f0fd40cff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:53.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.787+0000 7f0fd98dd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd40d0440 0x7f0fd41676b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:53.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.787+0000 7f0fd98dd700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd40d4020 con 0x7f0fd415d8b0 2024-05-08T21:22:53.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.787+0000 7f0fd98dd700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd40d08d0 con 0x7f0fd40d23b0 2024-05-08T21:22:53.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.787+0000 7f0fd98dd700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd415dc90 con 0x7f0fd40d0440 2024-05-08T21:22:53.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.787+0000 7f0fd27fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0fd40d23b0 0x7f0fd40cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:53.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.787+0000 7f0fd27fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0fd40d23b0 0x7f0fd40cff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45496/0 (socket says 172.21.15.3:45496) 2024-05-08T21:22:53.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.787+0000 7f0fd27fc700 1 -- 172.21.15.3:0/2361219787 learned_addr learned my addr 172.21.15.3:0/2361219787 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:53.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.788+0000 7f0fd37fe700 1 --2- 172.21.15.3:0/2361219787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd40d0440 0x7f0fd41676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:53.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.788+0000 7f0fd27fc700 1 -- 172.21.15.3:0/2361219787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd40d0440 msgr2=0x7f0fd41676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:53.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.788+0000 7f0fd27fc700 1 --2- 172.21.15.3:0/2361219787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd40d0440 0x7f0fd41676b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.788+0000 7f0fd27fc700 1 -- 172.21.15.3:0/2361219787 >> v1:172.21.15.186:6789/0 conn(0x7f0fd415d8b0 legacy=0x7f0fd40cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:53.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.788+0000 7f0fd27fc700 1 -- 172.21.15.3:0/2361219787 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd40d1310 con 0x7f0fd40d23b0 2024-05-08T21:22:53.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.788+0000 7f0fd27fc700 1 --2- 172.21.15.3:0/2361219787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fd40d23b0 0x7f0fd40cff00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8006fd0 tx=0x7f0fc8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9024d7b4850a172 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:53.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.788+0000 7f0fd1ffb700 1 -- 172.21.15.3:0/2361219787 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc8005e30 con 0x7f0fd40d23b0 2024-05-08T21:22:53.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.788+0000 7f0fd1ffb700 1 -- 172.21.15.3:0/2361219787 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc8004030 con 0x7f0fd40d23b0 2024-05-08T21:22:53.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.788+0000 7f0fd1ffb700 1 -- 172.21.15.3:0/2361219787 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc80043e0 con 0x7f0fd40d23b0 2024-05-08T21:22:53.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.789+0000 7f0fd98dd700 1 -- 172.21.15.3:0/2361219787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fd40d23b0 msgr2=0x7f0fd40cff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:53.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.789+0000 7f0fd98dd700 1 --2- 172.21.15.3:0/2361219787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fd40d23b0 0x7f0fd40cff00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8006fd0 tx=0x7f0fc8009bd0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.789+0000 7f0fd98dd700 1 -- 172.21.15.3:0/2361219787 shutdown_connections 2024-05-08T21:22:53.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.789+0000 7f0fd98dd700 1 --2- 172.21.15.3:0/2361219787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd40d0440 0x7f0fd41676b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.789+0000 7f0fd98dd700 1 --2- 172.21.15.3:0/2361219787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fd40d23b0 0x7f0fd40cff00 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.789+0000 7f0fd98dd700 1 -- 172.21.15.3:0/2361219787 >> 172.21.15.3:0/2361219787 conn(0x7f0fd408afe0 msgr2=0x7f0fd40d3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:53.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.789+0000 7f0fd98dd700 1 -- 172.21.15.3:0/2361219787 shutdown_connections 2024-05-08T21:22:53.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.789+0000 7f0fd98dd700 1 -- 172.21.15.3:0/2361219787 wait complete. 2024-05-08T21:22:53.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.790+0000 7f0fd98dd700 1 Processor -- start 2024-05-08T21:22:53.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.791+0000 7f0fd98dd700 1 -- start start 2024-05-08T21:22:53.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd98dd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fd40d0440 0x7f0fd4151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:53.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd98dd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd40d23b0 0x7f0fd40b7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:53.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd98dd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd415d8b0 0x7f0fd415bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:53.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd98dd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd40b6f10 con 0x7f0fd40d0440 2024-05-08T21:22:53.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd98dd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd40b71f0 con 0x7f0fd415d8b0 2024-05-08T21:22:53.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd98dd700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd40b74d0 con 0x7f0fd40d23b0 2024-05-08T21:22:53.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fd40d0440 0x7f0fd4151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:53.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fd40d0440 0x7f0fd4151640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45500/0 (socket says 172.21.15.3:45500) 2024-05-08T21:22:53.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd2ffd700 1 -- 172.21.15.3:0/3474122143 learned_addr learned my addr 172.21.15.3:0/3474122143 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:53.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd27fc700 1 --2- 172.21.15.3:0/3474122143 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd40d23b0 0x7f0fd40b7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:53.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd37fe700 1 --2- 172.21.15.3:0/3474122143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd415d8b0 0x7f0fd415bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:53.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd2ffd700 1 -- 172.21.15.3:0/3474122143 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd40d23b0 msgr2=0x7f0fd40b7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:53.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd2ffd700 1 --2- 172.21.15.3:0/3474122143 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd40d23b0 0x7f0fd40b7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd2ffd700 1 -- 172.21.15.3:0/3474122143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd415d8b0 msgr2=0x7f0fd415bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:53.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd2ffd700 1 --2- 172.21.15.3:0/3474122143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd415d8b0 0x7f0fd415bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:53.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd2ffd700 1 -- 172.21.15.3:0/3474122143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc8009500 con 0x7f0fd40d0440 2024-05-08T21:22:53.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd37fe700 1 --2- 172.21.15.3:0/3474122143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd415d8b0 0x7f0fd415bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:53.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd27fc700 1 --2- 172.21.15.3:0/3474122143 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd40d23b0 0x7f0fd40b7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:53.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.792+0000 7f0fd2ffd700 1 --2- 172.21.15.3:0/3474122143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fd40d0440 0x7f0fd4151640 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400c4e0 tx=0x7f0fc400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:53.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.793+0000 7f0fbbfff700 1 -- 172.21.15.3:0/3474122143 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc40052d0 con 0x7f0fd40d0440 2024-05-08T21:22:53.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.793+0000 7f0fbbfff700 1 -- 172.21.15.3:0/3474122143 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc4005430 con 0x7f0fd40d0440 2024-05-08T21:22:53.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.793+0000 7f0fd98dd700 1 -- 172.21.15.3:0/3474122143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd40b77b0 con 0x7f0fd40d0440 2024-05-08T21:22:53.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.793+0000 7f0fbbfff700 1 -- 172.21.15.3:0/3474122143 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc40038c0 con 0x7f0fd40d0440 2024-05-08T21:22:53.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.793+0000 7f0fd98dd700 1 -- 172.21.15.3:0/3474122143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd40b84d0 con 0x7f0fd40d0440 2024-05-08T21:22:53.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.794+0000 7f0fd98dd700 1 -- 172.21.15.3:0/3474122143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd40b7e00 con 0x7f0fd40d0440 2024-05-08T21:22:53.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.794+0000 7f0fbbfff700 1 -- 172.21.15.3:0/3474122143 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0fc4003a20 con 0x7f0fd40d0440 2024-05-08T21:22:53.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.795+0000 7f0fbbfff700 1 --2- 172.21.15.3:0/3474122143 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0fc006ac90 0x7f0fc006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:53.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.795+0000 7f0fbbfff700 1 -- 172.21.15.3:0/3474122143 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0fc40887f0 con 0x7f0fd40d0440 2024-05-08T21:22:53.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.795+0000 7f0fd27fc700 1 --2- 172.21.15.3:0/3474122143 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0fc006ac90 0x7f0fc006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:53.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.795+0000 7f0fd27fc700 1 --2- 172.21.15.3:0/3474122143 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0fc006ac90 0x7f0fc006d150 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8006fd0 tx=0x7f0fc8008070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:53.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:53.799+0000 7f0fbbfff700 1 -- 172.21.15.3:0/3474122143 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0fc40539d0 con 0x7f0fd40d0440 2024-05-08T21:22:54.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.018+0000 7f0fd98dd700 1 -- 172.21.15.3:0/3474122143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fd40b8660 con 0x7f0fd40d0440 2024-05-08T21:22:54.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.020+0000 7f0fbbfff700 1 -- 172.21.15.3:0/3474122143 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16053 (secure 0 0 0) 0x7f0fd40b8660 con 0x7f0fd40d0440 2024-05-08T21:22:54.021 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:54.021 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":34,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:22:51.186147+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:22:47.853786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-08T21:22:49.153759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:54.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.022+0000 7f0fd98dd700 1 -- 172.21.15.3:0/3474122143 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0fc006ac90 msgr2=0x7f0fc006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:54.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.022+0000 7f0fd98dd700 1 --2- 172.21.15.3:0/3474122143 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0fc006ac90 0x7f0fc006d150 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8006fd0 tx=0x7f0fc8008070 comp rx=0 tx=0).stop 2024-05-08T21:22:54.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.022+0000 7f0fd98dd700 1 -- 172.21.15.3:0/3474122143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fd40d0440 msgr2=0x7f0fd4151640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:54.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.022+0000 7f0fd98dd700 1 --2- 172.21.15.3:0/3474122143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fd40d0440 0x7f0fd4151640 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400c4e0 tx=0x7f0fc400c8a0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.022+0000 7f0fd98dd700 1 -- 172.21.15.3:0/3474122143 shutdown_connections 2024-05-08T21:22:54.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.022+0000 7f0fd98dd700 1 --2- 172.21.15.3:0/3474122143 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0fc006ac90 0x7f0fc006d150 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.022+0000 7f0fd98dd700 1 --2- 172.21.15.3:0/3474122143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd415d8b0 0x7f0fd415bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.022+0000 7f0fd98dd700 1 --2- 172.21.15.3:0/3474122143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fd40d0440 0x7f0fd4151640 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.022+0000 7f0fd98dd700 1 --2- 172.21.15.3:0/3474122143 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd40d23b0 0x7f0fd40b7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.022+0000 7f0fd98dd700 1 -- 172.21.15.3:0/3474122143 >> 172.21.15.3:0/3474122143 conn(0x7f0fd408afe0 msgr2=0x7f0fd406cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:54.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.022+0000 7f0fd98dd700 1 -- 172.21.15.3:0/3474122143 shutdown_connections 2024-05-08T21:22:54.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.022+0000 7f0fd98dd700 1 -- 172.21.15.3:0/3474122143 wait complete. 2024-05-08T21:22:54.038 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-08T21:22:54.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T21:22:54.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.172+0000 7fb6f36de700 1 Processor -- start 2024-05-08T21:22:54.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.172+0000 7fb6f36de700 1 -- start start 2024-05-08T21:22:54.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.173+0000 7fb6f36de700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb6ec15b810 0x7fb6ec0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:54.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.173+0000 7fb6f36de700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6ec0d0520 0x7fb6ec099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:54.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.173+0000 7fb6f36de700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb6ec099f90 0x7fb6ec0ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:54.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.173+0000 7fb6f36de700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6ec09a440 con 0x7fb6ec15b810 2024-05-08T21:22:54.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.173+0000 7fb6f36de700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6ec09aca0 con 0x7fb6ec099f90 2024-05-08T21:22:54.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.173+0000 7fb6f36de700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec0cd9e0 con 0x7fb6ec0d0520 2024-05-08T21:22:54.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.173+0000 7fb6f147a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb6ec15b810 0x7fb6ec0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:54.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.174+0000 7fb6f147a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb6ec15b810 0x7fb6ec0cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45510/0 (socket says 172.21.15.3:45510) 2024-05-08T21:22:54.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.174+0000 7fb6f147a700 1 -- 172.21.15.3:0/1178715941 learned_addr learned my addr 172.21.15.3:0/1178715941 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:54.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.174+0000 7fb6f0c79700 1 --2- 172.21.15.3:0/1178715941 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6ec0d0520 0x7fb6ec099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:54.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.174+0000 7fb6f1c7b700 1 --2- 172.21.15.3:0/1178715941 >> v2:172.21.15.186:3300/0 conn(0x7fb6ec099f90 0x7fb6ec0ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:54.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.174+0000 7fb6f147a700 1 -- 172.21.15.3:0/1178715941 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6ec0d0520 msgr2=0x7fb6ec099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:54.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.174+0000 7fb6f147a700 1 --2- 172.21.15.3:0/1178715941 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6ec0d0520 0x7fb6ec099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.174+0000 7fb6f147a700 1 -- 172.21.15.3:0/1178715941 >> v2:172.21.15.186:3300/0 conn(0x7fb6ec099f90 msgr2=0x7fb6ec0ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:54.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.174+0000 7fb6f147a700 1 --2- 172.21.15.3:0/1178715941 >> v2:172.21.15.186:3300/0 conn(0x7fb6ec099f90 0x7fb6ec0ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.174+0000 7fb6f147a700 1 -- 172.21.15.3:0/1178715941 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ec0cf220 con 0x7fb6ec15b810 2024-05-08T21:22:54.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.174+0000 7fb6f1c7b700 1 --2- 172.21.15.3:0/1178715941 >> v2:172.21.15.186:3300/0 conn(0x7fb6ec099f90 0x7fb6ec0ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:54.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.174+0000 7fb6f147a700 1 --2- 172.21.15.3:0/1178715941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6ec15b810 0x7fb6ec0cffe0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec0cbb00 tx=0x7fb6e8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f174af8701909bd5 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:54.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.174+0000 7fb6e3fff700 1 -- 172.21.15.3:0/1178715941 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e800b860 con 0x7fb6ec15b810 2024-05-08T21:22:54.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.174+0000 7fb6e3fff700 1 -- 172.21.15.3:0/1178715941 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e800b9c0 con 0x7fb6ec15b810 2024-05-08T21:22:54.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.174+0000 7fb6e3fff700 1 -- 172.21.15.3:0/1178715941 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e800bd50 con 0x7fb6ec15b810 2024-05-08T21:22:54.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.175+0000 7fb6f36de700 1 -- 172.21.15.3:0/1178715941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6ec15b810 msgr2=0x7fb6ec0cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:54.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.175+0000 7fb6f36de700 1 --2- 172.21.15.3:0/1178715941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6ec15b810 0x7fb6ec0cffe0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec0cbb00 tx=0x7fb6e8009c00 comp rx=0 tx=0).stop 2024-05-08T21:22:54.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.175+0000 7fb6f36de700 1 -- 172.21.15.3:0/1178715941 shutdown_connections 2024-05-08T21:22:54.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.175+0000 7fb6f36de700 1 --2- 172.21.15.3:0/1178715941 >> v2:172.21.15.186:3300/0 conn(0x7fb6ec099f90 0x7fb6ec0ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.175+0000 7fb6f36de700 1 --2- 172.21.15.3:0/1178715941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6ec15b810 0x7fb6ec0cffe0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.175+0000 7fb6f36de700 1 --2- 172.21.15.3:0/1178715941 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6ec0d0520 0x7fb6ec099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.175+0000 7fb6f36de700 1 -- 172.21.15.3:0/1178715941 >> 172.21.15.3:0/1178715941 conn(0x7fb6ec0b3010 msgr2=0x7fb6ec0b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:54.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.175+0000 7fb6f36de700 1 -- 172.21.15.3:0/1178715941 shutdown_connections 2024-05-08T21:22:54.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.175+0000 7fb6f36de700 1 -- 172.21.15.3:0/1178715941 wait complete. 2024-05-08T21:22:54.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.176+0000 7fb6f36de700 1 Processor -- start 2024-05-08T21:22:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.176+0000 7fb6f36de700 1 -- start start 2024-05-08T21:22:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.177+0000 7fb6f36de700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6ec099f90 0x7fb6ec0b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.177+0000 7fb6f36de700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6ec0d0520 0x7fb6ec0caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.177+0000 7fb6f36de700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6ec15b810 0x7fb6ec0c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.177+0000 7fb6f36de700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec0ca680 con 0x7fb6ec15b810 2024-05-08T21:22:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.177+0000 7fb6f36de700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec0cafe0 con 0x7fb6ec099f90 2024-05-08T21:22:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.177+0000 7fb6f36de700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec0c9a00 con 0x7fb6ec0d0520 2024-05-08T21:22:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.177+0000 7fb6f1c7b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6ec15b810 0x7fb6ec0c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.177+0000 7fb6f1c7b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6ec15b810 0x7fb6ec0c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45524/0 (socket says 172.21.15.3:45524) 2024-05-08T21:22:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.177+0000 7fb6f1c7b700 1 -- 172.21.15.3:0/2333218485 learned_addr learned my addr 172.21.15.3:0/2333218485 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.177+0000 7fb6f147a700 1 --2- 172.21.15.3:0/2333218485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6ec099f90 0x7fb6ec0b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.177+0000 7fb6f0c79700 1 --2- 172.21.15.3:0/2333218485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6ec0d0520 0x7fb6ec0caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.178+0000 7fb6f1c7b700 1 -- 172.21.15.3:0/2333218485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6ec0d0520 msgr2=0x7fb6ec0caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.178+0000 7fb6f1c7b700 1 --2- 172.21.15.3:0/2333218485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6ec0d0520 0x7fb6ec0caaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.178+0000 7fb6f1c7b700 1 -- 172.21.15.3:0/2333218485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6ec099f90 msgr2=0x7fb6ec0b69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.178+0000 7fb6f1c7b700 1 --2- 172.21.15.3:0/2333218485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6ec099f90 0x7fb6ec0b69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.178+0000 7fb6f1c7b700 1 -- 172.21.15.3:0/2333218485 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e8009550 con 0x7fb6ec15b810 2024-05-08T21:22:54.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.178+0000 7fb6f1c7b700 1 --2- 172.21.15.3:0/2333218485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6ec15b810 0x7fb6ec0c7350 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb6e400cd10 tx=0x7fb6e400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:54.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.178+0000 7fb6e1ffb700 1 -- 172.21.15.3:0/2333218485 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e400ce60 con 0x7fb6ec15b810 2024-05-08T21:22:54.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.178+0000 7fb6e1ffb700 1 -- 172.21.15.3:0/2333218485 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e400f480 con 0x7fb6ec15b810 2024-05-08T21:22:54.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.178+0000 7fb6f36de700 1 -- 172.21.15.3:0/2333218485 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ec0c9ce0 con 0x7fb6ec15b810 2024-05-08T21:22:54.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.178+0000 7fb6e1ffb700 1 -- 172.21.15.3:0/2333218485 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e400f810 con 0x7fb6ec15b810 2024-05-08T21:22:54.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.178+0000 7fb6f36de700 1 -- 172.21.15.3:0/2333218485 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ec0ca2a0 con 0x7fb6ec15b810 2024-05-08T21:22:54.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.179+0000 7fb6f36de700 1 -- 172.21.15.3:0/2333218485 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ec068000 con 0x7fb6ec15b810 2024-05-08T21:22:54.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.180+0000 7fb6e1ffb700 1 -- 172.21.15.3:0/2333218485 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb6e400b040 con 0x7fb6ec15b810 2024-05-08T21:22:54.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.180+0000 7fb6e1ffb700 1 --2- 172.21.15.3:0/2333218485 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb6d006ac90 0x7fb6d006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:54.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.181+0000 7fb6e1ffb700 1 -- 172.21.15.3:0/2333218485 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6e4089980 con 0x7fb6ec15b810 2024-05-08T21:22:54.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.181+0000 7fb6f147a700 1 --2- 172.21.15.3:0/2333218485 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb6d006ac90 0x7fb6d006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:54.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.181+0000 7fb6f147a700 1 --2- 172.21.15.3:0/2333218485 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb6d006ac90 0x7fb6d006d150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8000c00 tx=0x7fb6e8008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:54.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.185+0000 7fb6e1ffb700 1 -- 172.21.15.3:0/2333218485 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb6e4054b50 con 0x7fb6ec15b810 2024-05-08T21:22:54.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.398+0000 7fb6f36de700 1 -- 172.21.15.3:0/2333218485 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb6ec0cbef0 con 0x7fb6d006ac90 2024-05-08T21:22:54.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.408+0000 7fb6e1ffb700 1 -- 172.21.15.3:0/2333218485 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289235 (secure 0 0 0) 0x7fb6ec0cbef0 con 0x7fb6d006ac90 2024-05-08T21:22:54.413 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:54.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.411+0000 7fb6f36de700 1 -- 172.21.15.3:0/2333218485 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb6d006ac90 msgr2=0x7fb6d006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:54.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.411+0000 7fb6f36de700 1 --2- 172.21.15.3:0/2333218485 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb6d006ac90 0x7fb6d006d150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8000c00 tx=0x7fb6e8008030 comp rx=0 tx=0).stop 2024-05-08T21:22:54.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.411+0000 7fb6f36de700 1 -- 172.21.15.3:0/2333218485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6ec15b810 msgr2=0x7fb6ec0c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:54.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.411+0000 7fb6f36de700 1 --2- 172.21.15.3:0/2333218485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6ec15b810 0x7fb6ec0c7350 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb6e400cd10 tx=0x7fb6e400a560 comp rx=0 tx=0).stop 2024-05-08T21:22:54.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.411+0000 7fb6f36de700 1 -- 172.21.15.3:0/2333218485 shutdown_connections 2024-05-08T21:22:54.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.411+0000 7fb6f36de700 1 --2- 172.21.15.3:0/2333218485 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb6d006ac90 0x7fb6d006d150 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.411+0000 7fb6f36de700 1 --2- 172.21.15.3:0/2333218485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6ec15b810 0x7fb6ec0c7350 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.411+0000 7fb6f36de700 1 --2- 172.21.15.3:0/2333218485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6ec099f90 0x7fb6ec0b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.411+0000 7fb6f36de700 1 --2- 172.21.15.3:0/2333218485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6ec0d0520 0x7fb6ec0caaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.411+0000 7fb6f36de700 1 -- 172.21.15.3:0/2333218485 >> 172.21.15.3:0/2333218485 conn(0x7fb6ec0b3010 msgr2=0x7fb6ec0b27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:54.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.411+0000 7fb6f36de700 1 -- 172.21.15.3:0/2333218485 shutdown_connections 2024-05-08T21:22:54.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.411+0000 7fb6f36de700 1 -- 172.21.15.3:0/2333218485 wait complete. 2024-05-08T21:22:54.417 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-08T21:22:54.427 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-05-08T21:22:52.545355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220252,"kb_used_data":4068,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754468,"statfs":{"total":773094113280,"available":772868575232,"internally_reserved":0,"allocated":4165632,"data_stored":2327533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000789"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214117+0000","last_change":"2024-05-08T21:22:50.209349+0000","last_active":"2024-05-08T21:22:50.214117+0000","last_peered":"2024-05-08T21:22:50.214117+0000","last_clean":"2024-05-08T21:22:50.214117+0000","last_became_active":"2024-05-08T21:22:50.209092+0000","last_became_peered":"2024-05-08T21:22:50.209092+0000","last_unstale":"2024-05-08T21:22:50.214117+0000","last_undegraded":"2024-05-08T21:22:50.214117+0000","last_fullsized":"2024-05-08T21:22:50.214117+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:52:47.510546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213268+0000","last_change":"2024-05-08T21:22:50.205276+0000","last_active":"2024-05-08T21:22:50.213268+0000","last_peered":"2024-05-08T21:22:50.213268+0000","last_clean":"2024-05-08T21:22:50.213268+0000","last_became_active":"2024-05-08T21:22:50.205028+0000","last_became_peered":"2024-05-08T21:22:50.205028+0000","last_unstale":"2024-05-08T21:22:50.213268+0000","last_undegraded":"2024-05-08T21:22:50.213268+0000","last_fullsized":"2024-05-08T21:22:50.213268+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:24:34.674332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210756+0000","last_change":"2024-05-08T21:22:50.205185+0000","last_active":"2024-05-08T21:22:50.210756+0000","last_peered":"2024-05-08T21:22:50.210756+0000","last_clean":"2024-05-08T21:22:50.210756+0000","last_became_active":"2024-05-08T21:22:50.204826+0000","last_became_peered":"2024-05-08T21:22:50.204826+0000","last_unstale":"2024-05-08T21:22:50.210756+0000","last_undegraded":"2024-05-08T21:22:50.210756+0000","last_fullsized":"2024-05-08T21:22:50.210756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:05:25.952745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213792+0000","last_change":"2024-05-08T21:22:50.210536+0000","last_active":"2024-05-08T21:22:50.213792+0000","last_peered":"2024-05-08T21:22:50.213792+0000","last_clean":"2024-05-08T21:22:50.213792+0000","last_became_active":"2024-05-08T21:22:50.210248+0000","last_became_peered":"2024-05-08T21:22:50.210248+0000","last_unstale":"2024-05-08T21:22:50.213792+0000","last_undegraded":"2024-05-08T21:22:50.213792+0000","last_fullsized":"2024-05-08T21:22:50.213792+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:27:40.074356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.216126+0000","last_change":"2024-05-08T21:22:50.208946+0000","last_active":"2024-05-08T21:22:50.216126+0000","last_peered":"2024-05-08T21:22:50.216126+0000","last_clean":"2024-05-08T21:22:50.216126+0000","last_became_active":"2024-05-08T21:22:50.208626+0000","last_became_peered":"2024-05-08T21:22:50.208626+0000","last_unstale":"2024-05-08T21:22:50.216126+0000","last_undegraded":"2024-05-08T21:22:50.216126+0000","last_fullsized":"2024-05-08T21:22:50.216126+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:09.165560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214059+0000","last_change":"2024-05-08T21:22:50.207734+0000","last_active":"2024-05-08T21:22:50.214059+0000","last_peered":"2024-05-08T21:22:50.214059+0000","last_clean":"2024-05-08T21:22:50.214059+0000","last_became_active":"2024-05-08T21:22:50.207392+0000","last_became_peered":"2024-05-08T21:22:50.207392+0000","last_unstale":"2024-05-08T21:22:50.214059+0000","last_undegraded":"2024-05-08T21:22:50.214059+0000","last_fullsized":"2024-05-08T21:22:50.214059+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:23:48.163955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212627+0000","last_change":"2024-05-08T21:22:50.210352+0000","last_active":"2024-05-08T21:22:50.212627+0000","last_peered":"2024-05-08T21:22:50.212627+0000","last_clean":"2024-05-08T21:22:50.212627+0000","last_became_active":"2024-05-08T21:22:50.210060+0000","last_became_peered":"2024-05-08T21:22:50.210060+0000","last_unstale":"2024-05-08T21:22:50.212627+0000","last_undegraded":"2024-05-08T21:22:50.212627+0000","last_fullsized":"2024-05-08T21:22:50.212627+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:58:46.704338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.211040+0000","last_change":"2024-05-08T21:22:50.203181+0000","last_active":"2024-05-08T21:22:50.211040+0000","last_peered":"2024-05-08T21:22:50.211040+0000","last_clean":"2024-05-08T21:22:50.211040+0000","last_became_active":"2024-05-08T21:22:50.202914+0000","last_became_peered":"2024-05-08T21:22:50.202914+0000","last_unstale":"2024-05-08T21:22:50.211040+0000","last_undegraded":"2024-05-08T21:22:50.211040+0000","last_fullsized":"2024-05-08T21:22:50.211040+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:24:39.295215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212281+0000","last_change":"2024-05-08T21:22:50.207050+0000","last_active":"2024-05-08T21:22:50.212281+0000","last_peered":"2024-05-08T21:22:50.212281+0000","last_clean":"2024-05-08T21:22:50.212281+0000","last_became_active":"2024-05-08T21:22:50.206737+0000","last_became_peered":"2024-05-08T21:22:50.206737+0000","last_unstale":"2024-05-08T21:22:50.212281+0000","last_undegraded":"2024-05-08T21:22:50.212281+0000","last_fullsized":"2024-05-08T21:22:50.212281+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:52:47.510546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213836+0000","last_change":"2024-05-08T21:22:50.212260+0000","last_active":"2024-05-08T21:22:50.213836+0000","last_peered":"2024-05-08T21:22:50.213836+0000","last_clean":"2024-05-08T21:22:50.213836+0000","last_became_active":"2024-05-08T21:22:50.211928+0000","last_became_peered":"2024-05-08T21:22:50.211928+0000","last_unstale":"2024-05-08T21:22:50.213836+0000","last_undegraded":"2024-05-08T21:22:50.213836+0000","last_fullsized":"2024-05-08T21:22:50.213836+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:43:00.051225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212482+0000","last_change":"2024-05-08T21:22:50.205779+0000","last_active":"2024-05-08T21:22:50.212482+0000","last_peered":"2024-05-08T21:22:50.212482+0000","last_clean":"2024-05-08T21:22:50.212482+0000","last_became_active":"2024-05-08T21:22:50.205533+0000","last_became_peered":"2024-05-08T21:22:50.205533+0000","last_unstale":"2024-05-08T21:22:50.212482+0000","last_undegraded":"2024-05-08T21:22:50.212482+0000","last_fullsized":"2024-05-08T21:22:50.212482+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:24:34.674332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210296+0000","last_change":"2024-05-08T21:22:50.207359+0000","last_active":"2024-05-08T21:22:50.210296+0000","last_peered":"2024-05-08T21:22:50.210296+0000","last_clean":"2024-05-08T21:22:50.210296+0000","last_became_active":"2024-05-08T21:22:50.207045+0000","last_became_peered":"2024-05-08T21:22:50.207045+0000","last_unstale":"2024-05-08T21:22:50.210296+0000","last_undegraded":"2024-05-08T21:22:50.210296+0000","last_fullsized":"2024-05-08T21:22:50.210296+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:24:34.674332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.215394+0000","last_change":"2024-05-08T21:22:50.207671+0000","last_active":"2024-05-08T21:22:50.215394+0000","last_peered":"2024-05-08T21:22:50.215394+0000","last_clean":"2024-05-08T21:22:50.215394+0000","last_became_active":"2024-05-08T21:22:50.207119+0000","last_became_peered":"2024-05-08T21:22:50.207119+0000","last_unstale":"2024-05-08T21:22:50.215394+0000","last_undegraded":"2024-05-08T21:22:50.215394+0000","last_fullsized":"2024-05-08T21:22:50.215394+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:59:37.805505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.217164+0000","last_change":"2024-05-08T21:22:50.216066+0000","last_active":"2024-05-08T21:22:50.217164+0000","last_peered":"2024-05-08T21:22:50.217164+0000","last_clean":"2024-05-08T21:22:50.217164+0000","last_became_active":"2024-05-08T21:22:50.215796+0000","last_became_peered":"2024-05-08T21:22:50.215796+0000","last_unstale":"2024-05-08T21:22:50.217164+0000","last_undegraded":"2024-05-08T21:22:50.217164+0000","last_fullsized":"2024-05-08T21:22:50.217164+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:24:39.295215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214235+0000","last_change":"2024-05-08T21:22:50.211332+0000","last_active":"2024-05-08T21:22:50.214235+0000","last_peered":"2024-05-08T21:22:50.214235+0000","last_clean":"2024-05-08T21:22:50.214235+0000","last_became_active":"2024-05-08T21:22:50.210865+0000","last_became_peered":"2024-05-08T21:22:50.210865+0000","last_unstale":"2024-05-08T21:22:50.214235+0000","last_undegraded":"2024-05-08T21:22:50.214235+0000","last_fullsized":"2024-05-08T21:22:50.214235+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:24:39.295215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.216263+0000","last_change":"2024-05-08T21:22:50.215021+0000","last_active":"2024-05-08T21:22:50.216263+0000","last_peered":"2024-05-08T21:22:50.216263+0000","last_clean":"2024-05-08T21:22:50.216263+0000","last_became_active":"2024-05-08T21:22:50.214821+0000","last_became_peered":"2024-05-08T21:22:50.214821+0000","last_unstale":"2024-05-08T21:22:50.216263+0000","last_undegraded":"2024-05-08T21:22:50.216263+0000","last_fullsized":"2024-05-08T21:22:50.216263+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:14.396616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212692+0000","last_change":"2024-05-08T21:22:50.205302+0000","last_active":"2024-05-08T21:22:50.212692+0000","last_peered":"2024-05-08T21:22:50.212692+0000","last_clean":"2024-05-08T21:22:50.212692+0000","last_became_active":"2024-05-08T21:22:50.204962+0000","last_became_peered":"2024-05-08T21:22:50.204962+0000","last_unstale":"2024-05-08T21:22:50.212692+0000","last_undegraded":"2024-05-08T21:22:50.212692+0000","last_fullsized":"2024-05-08T21:22:50.212692+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:09:24.655227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214257+0000","last_change":"2024-05-08T21:22:50.208448+0000","last_active":"2024-05-08T21:22:50.214257+0000","last_peered":"2024-05-08T21:22:50.214257+0000","last_clean":"2024-05-08T21:22:50.214257+0000","last_became_active":"2024-05-08T21:22:50.208210+0000","last_became_peered":"2024-05-08T21:22:50.208210+0000","last_unstale":"2024-05-08T21:22:50.214257+0000","last_undegraded":"2024-05-08T21:22:50.214257+0000","last_fullsized":"2024-05-08T21:22:50.214257+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:38:13.012930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214015+0000","last_change":"2024-05-08T21:22:50.206974+0000","last_active":"2024-05-08T21:22:50.214015+0000","last_peered":"2024-05-08T21:22:50.214015+0000","last_clean":"2024-05-08T21:22:50.214015+0000","last_became_active":"2024-05-08T21:22:50.206688+0000","last_became_peered":"2024-05-08T21:22:50.206688+0000","last_unstale":"2024-05-08T21:22:50.214015+0000","last_undegraded":"2024-05-08T21:22:50.214015+0000","last_fullsized":"2024-05-08T21:22:50.214015+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:41:28.819967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.215337+0000","last_change":"2024-05-08T21:22:50.207398+0000","last_active":"2024-05-08T21:22:50.215337+0000","last_peered":"2024-05-08T21:22:50.215337+0000","last_clean":"2024-05-08T21:22:50.215337+0000","last_became_active":"2024-05-08T21:22:50.207040+0000","last_became_peered":"2024-05-08T21:22:50.207040+0000","last_unstale":"2024-05-08T21:22:50.215337+0000","last_undegraded":"2024-05-08T21:22:50.215337+0000","last_fullsized":"2024-05-08T21:22:50.215337+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:28.358029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.206511+0000","last_change":"2024-05-08T21:22:50.199407+0000","last_active":"2024-05-08T21:22:50.206511+0000","last_peered":"2024-05-08T21:22:50.206511+0000","last_clean":"2024-05-08T21:22:50.206511+0000","last_became_active":"2024-05-08T21:22:50.199133+0000","last_became_peered":"2024-05-08T21:22:50.199133+0000","last_unstale":"2024-05-08T21:22:50.206511+0000","last_undegraded":"2024-05-08T21:22:50.206511+0000","last_fullsized":"2024-05-08T21:22:50.206511+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:40.200175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212126+0000","last_change":"2024-05-08T21:22:50.207042+0000","last_active":"2024-05-08T21:22:50.212126+0000","last_peered":"2024-05-08T21:22:50.212126+0000","last_clean":"2024-05-08T21:22:50.212126+0000","last_became_active":"2024-05-08T21:22:50.206554+0000","last_became_peered":"2024-05-08T21:22:50.206554+0000","last_unstale":"2024-05-08T21:22:50.212126+0000","last_undegraded":"2024-05-08T21:22:50.212126+0000","last_fullsized":"2024-05-08T21:22:50.212126+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:15:17.946198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.204324+0000","last_change":"2024-05-08T21:22:50.200284+0000","last_active":"2024-05-08T21:22:50.204324+0000","last_peered":"2024-05-08T21:22:50.204324+0000","last_clean":"2024-05-08T21:22:50.204324+0000","last_became_active":"2024-05-08T21:22:50.200004+0000","last_became_peered":"2024-05-08T21:22:50.200004+0000","last_unstale":"2024-05-08T21:22:50.204324+0000","last_undegraded":"2024-05-08T21:22:50.204324+0000","last_fullsized":"2024-05-08T21:22:50.204324+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:28.358029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.211103+0000","last_change":"2024-05-08T21:22:50.208138+0000","last_active":"2024-05-08T21:22:50.211103+0000","last_peered":"2024-05-08T21:22:50.211103+0000","last_clean":"2024-05-08T21:22:50.211103+0000","last_became_active":"2024-05-08T21:22:50.207813+0000","last_became_peered":"2024-05-08T21:22:50.207813+0000","last_unstale":"2024-05-08T21:22:50.211103+0000","last_undegraded":"2024-05-08T21:22:50.211103+0000","last_fullsized":"2024-05-08T21:22:50.211103+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:07:17.470994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210704+0000","last_change":"2024-05-08T21:22:50.202937+0000","last_active":"2024-05-08T21:22:50.210704+0000","last_peered":"2024-05-08T21:22:50.210704+0000","last_clean":"2024-05-08T21:22:50.210704+0000","last_became_active":"2024-05-08T21:22:50.202633+0000","last_became_peered":"2024-05-08T21:22:50.202633+0000","last_unstale":"2024-05-08T21:22:50.210704+0000","last_undegraded":"2024-05-08T21:22:50.210704+0000","last_fullsized":"2024-05-08T21:22:50.210704+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:48:46.138832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213247+0000","last_change":"2024-05-08T21:22:50.204581+0000","last_active":"2024-05-08T21:22:50.213247+0000","last_peered":"2024-05-08T21:22:50.213247+0000","last_clean":"2024-05-08T21:22:50.213247+0000","last_became_active":"2024-05-08T21:22:50.204228+0000","last_became_peered":"2024-05-08T21:22:50.204228+0000","last_unstale":"2024-05-08T21:22:50.213247+0000","last_undegraded":"2024-05-08T21:22:50.213247+0000","last_fullsized":"2024-05-08T21:22:50.213247+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:43:00.051225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213651+0000","last_change":"2024-05-08T21:22:50.210823+0000","last_active":"2024-05-08T21:22:50.213651+0000","last_peered":"2024-05-08T21:22:50.213651+0000","last_clean":"2024-05-08T21:22:50.213651+0000","last_became_active":"2024-05-08T21:22:50.210517+0000","last_became_peered":"2024-05-08T21:22:50.210517+0000","last_unstale":"2024-05-08T21:22:50.213651+0000","last_undegraded":"2024-05-08T21:22:50.213651+0000","last_fullsized":"2024-05-08T21:22:50.213651+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:09:24.655227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.208862+0000","last_change":"2024-05-08T21:22:50.201860+0000","last_active":"2024-05-08T21:22:50.208862+0000","last_peered":"2024-05-08T21:22:50.208862+0000","last_clean":"2024-05-08T21:22:50.208862+0000","last_became_active":"2024-05-08T21:22:50.201204+0000","last_became_peered":"2024-05-08T21:22:50.201204+0000","last_unstale":"2024-05-08T21:22:50.208862+0000","last_undegraded":"2024-05-08T21:22:50.208862+0000","last_fullsized":"2024-05-08T21:22:50.208862+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:23:48.163955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212049+0000","last_change":"2024-05-08T21:22:50.206666+0000","last_active":"2024-05-08T21:22:50.212049+0000","last_peered":"2024-05-08T21:22:50.212049+0000","last_clean":"2024-05-08T21:22:50.212049+0000","last_became_active":"2024-05-08T21:22:50.206283+0000","last_became_peered":"2024-05-08T21:22:50.206283+0000","last_unstale":"2024-05-08T21:22:50.212049+0000","last_undegraded":"2024-05-08T21:22:50.212049+0000","last_fullsized":"2024-05-08T21:22:50.212049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:53:39.386458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212345+0000","last_change":"2024-05-08T21:22:50.205319+0000","last_active":"2024-05-08T21:22:50.212345+0000","last_peered":"2024-05-08T21:22:50.212345+0000","last_clean":"2024-05-08T21:22:50.212345+0000","last_became_active":"2024-05-08T21:22:50.205014+0000","last_became_peered":"2024-05-08T21:22:50.205014+0000","last_unstale":"2024-05-08T21:22:50.212345+0000","last_undegraded":"2024-05-08T21:22:50.212345+0000","last_fullsized":"2024-05-08T21:22:50.212345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:10:48.376692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214174+0000","last_change":"2024-05-08T21:22:50.210271+0000","last_active":"2024-05-08T21:22:50.214174+0000","last_peered":"2024-05-08T21:22:50.214174+0000","last_clean":"2024-05-08T21:22:50.214174+0000","last_became_active":"2024-05-08T21:22:50.209998+0000","last_became_peered":"2024-05-08T21:22:50.209998+0000","last_unstale":"2024-05-08T21:22:50.214174+0000","last_undegraded":"2024-05-08T21:22:50.214174+0000","last_fullsized":"2024-05-08T21:22:50.214174+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:09.165560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213641+0000","last_change":"2024-05-08T21:22:50.211741+0000","last_active":"2024-05-08T21:22:50.213641+0000","last_peered":"2024-05-08T21:22:50.213641+0000","last_clean":"2024-05-08T21:22:50.213641+0000","last_became_active":"2024-05-08T21:22:50.211430+0000","last_became_peered":"2024-05-08T21:22:50.211430+0000","last_unstale":"2024-05-08T21:22:50.213641+0000","last_undegraded":"2024-05-08T21:22:50.213641+0000","last_fullsized":"2024-05-08T21:22:50.213641+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:23:48.163955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.204975+0000","last_change":"2024-05-08T21:22:50.201116+0000","last_active":"2024-05-08T21:22:50.204975+0000","last_peered":"2024-05-08T21:22:50.204975+0000","last_clean":"2024-05-08T21:22:50.204975+0000","last_became_active":"2024-05-08T21:22:50.200817+0000","last_became_peered":"2024-05-08T21:22:50.200817+0000","last_unstale":"2024-05-08T21:22:50.204975+0000","last_undegraded":"2024-05-08T21:22:50.204975+0000","last_fullsized":"2024-05-08T21:22:50.204975+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:09.165560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210819+0000","last_change":"2024-05-08T21:22:49.740529+0000","last_active":"2024-05-08T21:22:50.210819+0000","last_peered":"2024-05-08T21:22:50.210819+0000","last_clean":"2024-05-08T21:22:50.210819+0000","last_became_active":"2024-05-08T21:22:49.740247+0000","last_became_peered":"2024-05-08T21:22:49.740247+0000","last_unstale":"2024-05-08T21:22:50.210819+0000","last_undegraded":"2024-05-08T21:22:50.210819+0000","last_fullsized":"2024-05-08T21:22:50.210819+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:53:44.252529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.211713+0000","last_change":"2024-05-08T21:22:50.208077+0000","last_active":"2024-05-08T21:22:50.211713+0000","last_peered":"2024-05-08T21:22:50.211713+0000","last_clean":"2024-05-08T21:22:50.211713+0000","last_became_active":"2024-05-08T21:22:50.207729+0000","last_became_peered":"2024-05-08T21:22:50.207729+0000","last_unstale":"2024-05-08T21:22:50.211713+0000","last_undegraded":"2024-05-08T21:22:50.211713+0000","last_fullsized":"2024-05-08T21:22:50.211713+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:40.200175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.200537+0000","last_change":"2024-05-08T21:22:49.743083+0000","last_active":"2024-05-08T21:22:50.200537+0000","last_peered":"2024-05-08T21:22:50.200537+0000","last_clean":"2024-05-08T21:22:50.200537+0000","last_became_active":"2024-05-08T21:22:49.742790+0000","last_became_peered":"2024-05-08T21:22:49.742790+0000","last_unstale":"2024-05-08T21:22:50.200537+0000","last_undegraded":"2024-05-08T21:22:50.200537+0000","last_fullsized":"2024-05-08T21:22:50.200537+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:06:42.743465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212465+0000","last_change":"2024-05-08T21:22:50.207141+0000","last_active":"2024-05-08T21:22:50.212465+0000","last_peered":"2024-05-08T21:22:50.212465+0000","last_clean":"2024-05-08T21:22:50.212465+0000","last_became_active":"2024-05-08T21:22:50.206838+0000","last_became_peered":"2024-05-08T21:22:50.206838+0000","last_unstale":"2024-05-08T21:22:50.212465+0000","last_undegraded":"2024-05-08T21:22:50.212465+0000","last_fullsized":"2024-05-08T21:22:50.212465+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:16:58.048181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.206671+0000","last_change":"2024-05-08T21:22:49.747636+0000","last_active":"2024-05-08T21:22:50.206671+0000","last_peered":"2024-05-08T21:22:50.206671+0000","last_clean":"2024-05-08T21:22:50.206671+0000","last_became_active":"2024-05-08T21:22:49.747345+0000","last_became_peered":"2024-05-08T21:22:49.747345+0000","last_unstale":"2024-05-08T21:22:50.206671+0000","last_undegraded":"2024-05-08T21:22:50.206671+0000","last_fullsized":"2024-05-08T21:22:50.206671+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:29:03.389411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212991+0000","last_change":"2024-05-08T21:22:50.209316+0000","last_active":"2024-05-08T21:22:50.212991+0000","last_peered":"2024-05-08T21:22:50.212991+0000","last_clean":"2024-05-08T21:22:50.212991+0000","last_became_active":"2024-05-08T21:22:50.209044+0000","last_became_peered":"2024-05-08T21:22:50.209044+0000","last_unstale":"2024-05-08T21:22:50.212991+0000","last_undegraded":"2024-05-08T21:22:50.212991+0000","last_fullsized":"2024-05-08T21:22:50.212991+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:25:52.293163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.211278+0000","last_change":"2024-05-08T21:22:49.744152+0000","last_active":"2024-05-08T21:22:50.211278+0000","last_peered":"2024-05-08T21:22:50.211278+0000","last_clean":"2024-05-08T21:22:50.211278+0000","last_became_active":"2024-05-08T21:22:49.743867+0000","last_became_peered":"2024-05-08T21:22:49.743867+0000","last_unstale":"2024-05-08T21:22:50.211278+0000","last_undegraded":"2024-05-08T21:22:50.211278+0000","last_fullsized":"2024-05-08T21:22:50.211278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:55:00.385106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.209452+0000","last_change":"2024-05-08T21:22:50.204033+0000","last_active":"2024-05-08T21:22:50.209452+0000","last_peered":"2024-05-08T21:22:50.209452+0000","last_clean":"2024-05-08T21:22:50.209452+0000","last_became_active":"2024-05-08T21:22:50.203766+0000","last_became_peered":"2024-05-08T21:22:50.203766+0000","last_unstale":"2024-05-08T21:22:50.209452+0000","last_undegraded":"2024-05-08T21:22:50.209452+0000","last_fullsized":"2024-05-08T21:22:50.209452+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:48:46.138832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.209470+0000","last_change":"2024-05-08T21:22:49.740399+0000","last_active":"2024-05-08T21:22:50.209470+0000","last_peered":"2024-05-08T21:22:50.209470+0000","last_clean":"2024-05-08T21:22:50.209470+0000","last_became_active":"2024-05-08T21:22:49.740125+0000","last_became_peered":"2024-05-08T21:22:49.740125+0000","last_unstale":"2024-05-08T21:22:50.209470+0000","last_undegraded":"2024-05-08T21:22:50.209470+0000","last_fullsized":"2024-05-08T21:22:50.209470+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:01:42.646117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210459+0000","last_change":"2024-05-08T21:22:50.205784+0000","last_active":"2024-05-08T21:22:50.210459+0000","last_peered":"2024-05-08T21:22:50.210459+0000","last_clean":"2024-05-08T21:22:50.210459+0000","last_became_active":"2024-05-08T21:22:50.205525+0000","last_became_peered":"2024-05-08T21:22:50.205525+0000","last_unstale":"2024-05-08T21:22:50.210459+0000","last_undegraded":"2024-05-08T21:22:50.210459+0000","last_fullsized":"2024-05-08T21:22:50.210459+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:23.732223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.207041+0000","last_change":"2024-05-08T21:22:49.739604+0000","last_active":"2024-05-08T21:22:50.207041+0000","last_peered":"2024-05-08T21:22:50.207041+0000","last_clean":"2024-05-08T21:22:50.207041+0000","last_became_active":"2024-05-08T21:22:49.739343+0000","last_became_peered":"2024-05-08T21:22:49.739343+0000","last_unstale":"2024-05-08T21:22:50.207041+0000","last_undegraded":"2024-05-08T21:22:50.207041+0000","last_fullsized":"2024-05-08T21:22:50.207041+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:55:43.121728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212081+0000","last_change":"2024-05-08T21:22:50.206820+0000","last_active":"2024-05-08T21:22:50.212081+0000","last_peered":"2024-05-08T21:22:50.212081+0000","last_clean":"2024-05-08T21:22:50.212081+0000","last_became_active":"2024-05-08T21:22:50.206477+0000","last_became_peered":"2024-05-08T21:22:50.206477+0000","last_unstale":"2024-05-08T21:22:50.212081+0000","last_undegraded":"2024-05-08T21:22:50.212081+0000","last_fullsized":"2024-05-08T21:22:50.212081+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:28.358029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.194728+0000","last_change":"2024-05-08T21:22:49.737447+0000","last_active":"2024-05-08T21:22:50.194728+0000","last_peered":"2024-05-08T21:22:50.194728+0000","last_clean":"2024-05-08T21:22:50.194728+0000","last_became_active":"2024-05-08T21:22:49.736975+0000","last_became_peered":"2024-05-08T21:22:49.736975+0000","last_unstale":"2024-05-08T21:22:50.194728+0000","last_undegraded":"2024-05-08T21:22:50.194728+0000","last_fullsized":"2024-05-08T21:22:50.194728+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:39.192702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210202+0000","last_change":"2024-05-08T21:22:50.205263+0000","last_active":"2024-05-08T21:22:50.210202+0000","last_peered":"2024-05-08T21:22:50.210202+0000","last_clean":"2024-05-08T21:22:50.210202+0000","last_became_active":"2024-05-08T21:22:50.204963+0000","last_became_peered":"2024-05-08T21:22:50.204963+0000","last_unstale":"2024-05-08T21:22:50.210202+0000","last_undegraded":"2024-05-08T21:22:50.210202+0000","last_fullsized":"2024-05-08T21:22:50.210202+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:44:18.055232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.191099+0000","last_change":"2024-05-08T21:22:49.733981+0000","last_active":"2024-05-08T21:22:50.191099+0000","last_peered":"2024-05-08T21:22:50.191099+0000","last_clean":"2024-05-08T21:22:50.191099+0000","last_became_active":"2024-05-08T21:22:49.733714+0000","last_became_peered":"2024-05-08T21:22:49.733714+0000","last_unstale":"2024-05-08T21:22:50.191099+0000","last_undegraded":"2024-05-08T21:22:50.191099+0000","last_fullsized":"2024-05-08T21:22:50.191099+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:47.931452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.211942+0000","last_change":"2024-05-08T21:22:50.209646+0000","last_active":"2024-05-08T21:22:50.211942+0000","last_peered":"2024-05-08T21:22:50.211942+0000","last_clean":"2024-05-08T21:22:50.211942+0000","last_became_active":"2024-05-08T21:22:50.209335+0000","last_became_peered":"2024-05-08T21:22:50.209335+0000","last_unstale":"2024-05-08T21:22:50.211942+0000","last_undegraded":"2024-05-08T21:22:50.211942+0000","last_fullsized":"2024-05-08T21:22:50.211942+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:21:31.919852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.192386+0000","last_change":"2024-05-08T21:22:49.738039+0000","last_active":"2024-05-08T21:22:50.192386+0000","last_peered":"2024-05-08T21:22:50.192386+0000","last_clean":"2024-05-08T21:22:50.192386+0000","last_became_active":"2024-05-08T21:22:49.737785+0000","last_became_peered":"2024-05-08T21:22:49.737785+0000","last_unstale":"2024-05-08T21:22:50.192386+0000","last_undegraded":"2024-05-08T21:22:50.192386+0000","last_fullsized":"2024-05-08T21:22:50.192386+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:27.350557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214175+0000","last_change":"2024-05-08T21:22:50.206695+0000","last_active":"2024-05-08T21:22:50.214175+0000","last_peered":"2024-05-08T21:22:50.214175+0000","last_clean":"2024-05-08T21:22:50.214175+0000","last_became_active":"2024-05-08T21:22:50.206398+0000","last_became_peered":"2024-05-08T21:22:50.206398+0000","last_unstale":"2024-05-08T21:22:50.214175+0000","last_undegraded":"2024-05-08T21:22:50.214175+0000","last_fullsized":"2024-05-08T21:22:50.214175+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:07:17.470994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.188839+0000","last_change":"2024-05-08T21:22:49.740160+0000","last_active":"2024-05-08T21:22:50.188839+0000","last_peered":"2024-05-08T21:22:50.188839+0000","last_clean":"2024-05-08T21:22:50.188839+0000","last_became_active":"2024-05-08T21:22:49.739818+0000","last_became_peered":"2024-05-08T21:22:49.739818+0000","last_unstale":"2024-05-08T21:22:50.188839+0000","last_undegraded":"2024-05-08T21:22:50.188839+0000","last_fullsized":"2024-05-08T21:22:50.188839+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:39:42.856227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213426+0000","last_change":"2024-05-08T21:22:50.204717+0000","last_active":"2024-05-08T21:22:50.213426+0000","last_peered":"2024-05-08T21:22:50.213426+0000","last_clean":"2024-05-08T21:22:50.213426+0000","last_became_active":"2024-05-08T21:22:50.204404+0000","last_became_peered":"2024-05-08T21:22:50.204404+0000","last_unstale":"2024-05-08T21:22:50.213426+0000","last_undegraded":"2024-05-08T21:22:50.213426+0000","last_fullsized":"2024-05-08T21:22:50.213426+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:27:40.074356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.205088+0000","last_change":"2024-05-08T21:22:49.736860+0000","last_active":"2024-05-08T21:22:50.205088+0000","last_peered":"2024-05-08T21:22:50.205088+0000","last_clean":"2024-05-08T21:22:50.205088+0000","last_became_active":"2024-05-08T21:22:49.736497+0000","last_became_peered":"2024-05-08T21:22:49.736497+0000","last_unstale":"2024-05-08T21:22:50.205088+0000","last_undegraded":"2024-05-08T21:22:50.205088+0000","last_fullsized":"2024-05-08T21:22:50.205088+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:15:16.938725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210778+0000","last_change":"2024-05-08T21:22:50.205233+0000","last_active":"2024-05-08T21:22:50.210778+0000","last_peered":"2024-05-08T21:22:50.210778+0000","last_clean":"2024-05-08T21:22:50.210778+0000","last_became_active":"2024-05-08T21:22:50.204889+0000","last_became_peered":"2024-05-08T21:22:50.204889+0000","last_unstale":"2024-05-08T21:22:50.210778+0000","last_undegraded":"2024-05-08T21:22:50.210778+0000","last_fullsized":"2024-05-08T21:22:50.210778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:38:13.012930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.217253+0000","last_change":"2024-05-08T21:22:49.743647+0000","last_active":"2024-05-08T21:22:50.217253+0000","last_peered":"2024-05-08T21:22:50.217253+0000","last_clean":"2024-05-08T21:22:50.217253+0000","last_became_active":"2024-05-08T21:22:49.743348+0000","last_became_peered":"2024-05-08T21:22:49.743348+0000","last_unstale":"2024-05-08T21:22:50.217253+0000","last_undegraded":"2024-05-08T21:22:50.217253+0000","last_fullsized":"2024-05-08T21:22:50.217253+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:06:09.036243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210162+0000","last_change":"2024-05-08T21:22:50.202647+0000","last_active":"2024-05-08T21:22:50.210162+0000","last_peered":"2024-05-08T21:22:50.210162+0000","last_clean":"2024-05-08T21:22:50.210162+0000","last_became_active":"2024-05-08T21:22:50.202144+0000","last_became_peered":"2024-05-08T21:22:50.202144+0000","last_unstale":"2024-05-08T21:22:50.210162+0000","last_undegraded":"2024-05-08T21:22:50.210162+0000","last_fullsized":"2024-05-08T21:22:50.210162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:10:48.376692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212455+0000","last_change":"2024-05-08T21:22:49.735484+0000","last_active":"2024-05-08T21:22:50.212455+0000","last_peered":"2024-05-08T21:22:50.212455+0000","last_clean":"2024-05-08T21:22:50.212455+0000","last_became_active":"2024-05-08T21:22:49.735232+0000","last_became_peered":"2024-05-08T21:22:49.735232+0000","last_unstale":"2024-05-08T21:22:50.212455+0000","last_undegraded":"2024-05-08T21:22:50.212455+0000","last_fullsized":"2024-05-08T21:22:50.212455+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:13:03.055943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212615+0000","last_change":"2024-05-08T21:22:50.204754+0000","last_active":"2024-05-08T21:22:50.212615+0000","last_peered":"2024-05-08T21:22:50.212615+0000","last_clean":"2024-05-08T21:22:50.212615+0000","last_became_active":"2024-05-08T21:22:50.204473+0000","last_became_peered":"2024-05-08T21:22:50.204473+0000","last_unstale":"2024-05-08T21:22:50.212615+0000","last_undegraded":"2024-05-08T21:22:50.212615+0000","last_fullsized":"2024-05-08T21:22:50.212615+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:14.396616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.199440+0000","last_change":"2024-05-08T21:22:49.741850+0000","last_active":"2024-05-08T21:22:50.199440+0000","last_peered":"2024-05-08T21:22:50.199440+0000","last_clean":"2024-05-08T21:22:50.199440+0000","last_became_active":"2024-05-08T21:22:49.741527+0000","last_became_peered":"2024-05-08T21:22:49.741527+0000","last_unstale":"2024-05-08T21:22:50.199440+0000","last_undegraded":"2024-05-08T21:22:50.199440+0000","last_fullsized":"2024-05-08T21:22:50.199440+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:29:03.389411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213628+0000","last_change":"2024-05-08T21:22:50.211613+0000","last_active":"2024-05-08T21:22:50.213628+0000","last_peered":"2024-05-08T21:22:50.213628+0000","last_clean":"2024-05-08T21:22:50.213628+0000","last_became_active":"2024-05-08T21:22:50.211335+0000","last_became_peered":"2024-05-08T21:22:50.211335+0000","last_unstale":"2024-05-08T21:22:50.213628+0000","last_undegraded":"2024-05-08T21:22:50.213628+0000","last_fullsized":"2024-05-08T21:22:50.213628+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:43:00.051225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.194827+0000","last_change":"2024-05-08T21:22:49.737257+0000","last_active":"2024-05-08T21:22:50.194827+0000","last_peered":"2024-05-08T21:22:50.194827+0000","last_clean":"2024-05-08T21:22:50.194827+0000","last_became_active":"2024-05-08T21:22:49.736591+0000","last_became_peered":"2024-05-08T21:22:49.736591+0000","last_unstale":"2024-05-08T21:22:50.194827+0000","last_undegraded":"2024-05-08T21:22:50.194827+0000","last_fullsized":"2024-05-08T21:22:50.194827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:08.158087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214092+0000","last_change":"2024-05-08T21:22:50.209206+0000","last_active":"2024-05-08T21:22:50.214092+0000","last_peered":"2024-05-08T21:22:50.214092+0000","last_clean":"2024-05-08T21:22:50.214092+0000","last_became_active":"2024-05-08T21:22:50.208950+0000","last_became_peered":"2024-05-08T21:22:50.208950+0000","last_unstale":"2024-05-08T21:22:50.214092+0000","last_undegraded":"2024-05-08T21:22:50.214092+0000","last_fullsized":"2024-05-08T21:22:50.214092+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:15:17.946198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.215733+0000","last_change":"2024-05-08T21:22:49.743861+0000","last_active":"2024-05-08T21:22:50.215733+0000","last_peered":"2024-05-08T21:22:50.215733+0000","last_clean":"2024-05-08T21:22:50.215733+0000","last_became_active":"2024-05-08T21:22:49.743575+0000","last_became_peered":"2024-05-08T21:22:49.743575+0000","last_unstale":"2024-05-08T21:22:50.215733+0000","last_undegraded":"2024-05-08T21:22:50.215733+0000","last_fullsized":"2024-05-08T21:22:50.215733+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:51.977551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.207908+0000","last_change":"2024-05-08T21:22:50.203855+0000","last_active":"2024-05-08T21:22:50.207908+0000","last_peered":"2024-05-08T21:22:50.207908+0000","last_clean":"2024-05-08T21:22:50.207908+0000","last_became_active":"2024-05-08T21:22:50.203557+0000","last_became_peered":"2024-05-08T21:22:50.203557+0000","last_unstale":"2024-05-08T21:22:50.207908+0000","last_undegraded":"2024-05-08T21:22:50.207908+0000","last_fullsized":"2024-05-08T21:22:50.207908+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:09.165560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213662+0000","last_change":"2024-05-08T21:22:49.742208+0000","last_active":"2024-05-08T21:22:50.213662+0000","last_peered":"2024-05-08T21:22:50.213662+0000","last_clean":"2024-05-08T21:22:50.213662+0000","last_became_active":"2024-05-08T21:22:49.741894+0000","last_became_peered":"2024-05-08T21:22:49.741894+0000","last_unstale":"2024-05-08T21:22:50.213662+0000","last_undegraded":"2024-05-08T21:22:50.213662+0000","last_fullsized":"2024-05-08T21:22:50.213662+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:59:36.798032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210434+0000","last_change":"2024-05-08T21:22:50.208102+0000","last_active":"2024-05-08T21:22:50.210434+0000","last_peered":"2024-05-08T21:22:50.210434+0000","last_clean":"2024-05-08T21:22:50.210434+0000","last_became_active":"2024-05-08T21:22:50.207765+0000","last_became_peered":"2024-05-08T21:22:50.207765+0000","last_unstale":"2024-05-08T21:22:50.210434+0000","last_undegraded":"2024-05-08T21:22:50.210434+0000","last_fullsized":"2024-05-08T21:22:50.210434+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:14.396616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212404+0000","last_change":"2024-05-08T21:22:49.734619+0000","last_active":"2024-05-08T21:22:50.212404+0000","last_peered":"2024-05-08T21:22:50.212404+0000","last_clean":"2024-05-08T21:22:50.212404+0000","last_became_active":"2024-05-08T21:22:49.734354+0000","last_became_peered":"2024-05-08T21:22:49.734354+0000","last_unstale":"2024-05-08T21:22:50.212404+0000","last_undegraded":"2024-05-08T21:22:50.212404+0000","last_fullsized":"2024-05-08T21:22:50.212404+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:23:35.220950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213811+0000","last_change":"2024-05-08T21:22:50.210640+0000","last_active":"2024-05-08T21:22:50.213811+0000","last_peered":"2024-05-08T21:22:50.213811+0000","last_clean":"2024-05-08T21:22:50.213811+0000","last_became_active":"2024-05-08T21:22:50.210283+0000","last_became_peered":"2024-05-08T21:22:50.210283+0000","last_unstale":"2024-05-08T21:22:50.213811+0000","last_undegraded":"2024-05-08T21:22:50.213811+0000","last_fullsized":"2024-05-08T21:22:50.213811+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:58:46.704338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.215511+0000","last_change":"2024-05-08T21:22:49.737099+0000","last_active":"2024-05-08T21:22:50.215511+0000","last_peered":"2024-05-08T21:22:50.215511+0000","last_clean":"2024-05-08T21:22:50.215511+0000","last_became_active":"2024-05-08T21:22:49.736839+0000","last_became_peered":"2024-05-08T21:22:49.736839+0000","last_unstale":"2024-05-08T21:22:50.215511+0000","last_undegraded":"2024-05-08T21:22:50.215511+0000","last_fullsized":"2024-05-08T21:22:50.215511+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:05:26.886351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.215303+0000","last_change":"2024-05-08T21:22:50.206786+0000","last_active":"2024-05-08T21:22:50.215303+0000","last_peered":"2024-05-08T21:22:50.215303+0000","last_clean":"2024-05-08T21:22:50.215303+0000","last_became_active":"2024-05-08T21:22:50.206484+0000","last_became_peered":"2024-05-08T21:22:50.206484+0000","last_unstale":"2024-05-08T21:22:50.215303+0000","last_undegraded":"2024-05-08T21:22:50.215303+0000","last_fullsized":"2024-05-08T21:22:50.215303+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:36:44.630301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.191296+0000","last_change":"2024-05-08T21:22:49.740491+0000","last_active":"2024-05-08T21:22:50.191296+0000","last_peered":"2024-05-08T21:22:50.191296+0000","last_clean":"2024-05-08T21:22:50.191296+0000","last_became_active":"2024-05-08T21:22:49.740166+0000","last_became_peered":"2024-05-08T21:22:49.740166+0000","last_unstale":"2024-05-08T21:22:50.191296+0000","last_undegraded":"2024-05-08T21:22:50.191296+0000","last_fullsized":"2024-05-08T21:22:50.191296+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:27:16.794172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.216049+0000","last_change":"2024-05-08T21:22:50.207827+0000","last_active":"2024-05-08T21:22:50.216049+0000","last_peered":"2024-05-08T21:22:50.216049+0000","last_clean":"2024-05-08T21:22:50.216049+0000","last_became_active":"2024-05-08T21:22:50.207560+0000","last_became_peered":"2024-05-08T21:22:50.207560+0000","last_unstale":"2024-05-08T21:22:50.216049+0000","last_undegraded":"2024-05-08T21:22:50.216049+0000","last_fullsized":"2024-05-08T21:22:50.216049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:29:04.396884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.202910+0000","last_change":"2024-05-08T21:22:49.743647+0000","last_active":"2024-05-08T21:22:50.202910+0000","last_peered":"2024-05-08T21:22:50.202910+0000","last_clean":"2024-05-08T21:22:50.202910+0000","last_became_active":"2024-05-08T21:22:49.743349+0000","last_became_peered":"2024-05-08T21:22:49.743349+0000","last_unstale":"2024-05-08T21:22:50.202910+0000","last_undegraded":"2024-05-08T21:22:50.202910+0000","last_fullsized":"2024-05-08T21:22:50.202910+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:53:44.252529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.215372+0000","last_change":"2024-05-08T21:22:50.205819+0000","last_active":"2024-05-08T21:22:50.215372+0000","last_peered":"2024-05-08T21:22:50.215372+0000","last_clean":"2024-05-08T21:22:50.215372+0000","last_became_active":"2024-05-08T21:22:50.205239+0000","last_became_peered":"2024-05-08T21:22:50.205239+0000","last_unstale":"2024-05-08T21:22:50.215372+0000","last_undegraded":"2024-05-08T21:22:50.215372+0000","last_fullsized":"2024-05-08T21:22:50.215372+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:52.985024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.200371+0000","last_change":"2024-05-08T21:22:49.743818+0000","last_active":"2024-05-08T21:22:50.200371+0000","last_peered":"2024-05-08T21:22:50.200371+0000","last_clean":"2024-05-08T21:22:50.200371+0000","last_became_active":"2024-05-08T21:22:49.743469+0000","last_became_peered":"2024-05-08T21:22:49.743469+0000","last_unstale":"2024-05-08T21:22:50.200371+0000","last_undegraded":"2024-05-08T21:22:50.200371+0000","last_fullsized":"2024-05-08T21:22:50.200371+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:06:42.743465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.215141+0000","last_change":"2024-05-08T21:22:50.203236+0000","last_active":"2024-05-08T21:22:50.215141+0000","last_peered":"2024-05-08T21:22:50.215141+0000","last_clean":"2024-05-08T21:22:50.215141+0000","last_became_active":"2024-05-08T21:22:50.201914+0000","last_became_peered":"2024-05-08T21:22:50.201914+0000","last_unstale":"2024-05-08T21:22:50.215141+0000","last_undegraded":"2024-05-08T21:22:50.215141+0000","last_fullsized":"2024-05-08T21:22:50.215141+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:06:10.043716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.191299+0000","last_change":"2024-05-08T21:22:49.739798+0000","last_active":"2024-05-08T21:22:50.191299+0000","last_peered":"2024-05-08T21:22:50.191299+0000","last_clean":"2024-05-08T21:22:50.191299+0000","last_became_active":"2024-05-08T21:22:49.739520+0000","last_became_peered":"2024-05-08T21:22:49.739520+0000","last_unstale":"2024-05-08T21:22:50.191299+0000","last_undegraded":"2024-05-08T21:22:50.191299+0000","last_fullsized":"2024-05-08T21:22:50.191299+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:47:50.170209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.209189+0000","last_change":"2024-05-08T21:22:50.204561+0000","last_active":"2024-05-08T21:22:50.209189+0000","last_peered":"2024-05-08T21:22:50.209189+0000","last_clean":"2024-05-08T21:22:50.209189+0000","last_became_active":"2024-05-08T21:22:50.204296+0000","last_became_peered":"2024-05-08T21:22:50.204296+0000","last_unstale":"2024-05-08T21:22:50.209189+0000","last_undegraded":"2024-05-08T21:22:50.209189+0000","last_fullsized":"2024-05-08T21:22:50.209189+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:25:52.293163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.195794+0000","last_change":"2024-05-08T21:22:49.737100+0000","last_active":"2024-05-08T21:22:50.195794+0000","last_peered":"2024-05-08T21:22:50.195794+0000","last_clean":"2024-05-08T21:22:50.195794+0000","last_became_active":"2024-05-08T21:22:49.736715+0000","last_became_peered":"2024-05-08T21:22:49.736715+0000","last_unstale":"2024-05-08T21:22:50.195794+0000","last_undegraded":"2024-05-08T21:22:50.195794+0000","last_fullsized":"2024-05-08T21:22:50.195794+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:52:46.503073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.215282+0000","last_change":"2024-05-08T21:22:50.206910+0000","last_active":"2024-05-08T21:22:50.215282+0000","last_peered":"2024-05-08T21:22:50.215282+0000","last_clean":"2024-05-08T21:22:50.215282+0000","last_became_active":"2024-05-08T21:22:50.206623+0000","last_became_peered":"2024-05-08T21:22:50.206623+0000","last_unstale":"2024-05-08T21:22:50.215282+0000","last_undegraded":"2024-05-08T21:22:50.215282+0000","last_fullsized":"2024-05-08T21:22:50.215282+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:24:25.163652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.211304+0000","last_change":"2024-05-08T21:22:49.743344+0000","last_active":"2024-05-08T21:22:50.211304+0000","last_peered":"2024-05-08T21:22:50.211304+0000","last_clean":"2024-05-08T21:22:50.211304+0000","last_became_active":"2024-05-08T21:22:49.742413+0000","last_became_peered":"2024-05-08T21:22:49.742413+0000","last_unstale":"2024-05-08T21:22:50.211304+0000","last_undegraded":"2024-05-08T21:22:50.211304+0000","last_fullsized":"2024-05-08T21:22:50.211304+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:06:42.743465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210416+0000","last_change":"2024-05-08T21:22:50.205911+0000","last_active":"2024-05-08T21:22:50.210416+0000","last_peered":"2024-05-08T21:22:50.210416+0000","last_clean":"2024-05-08T21:22:50.210416+0000","last_became_active":"2024-05-08T21:22:50.205609+0000","last_became_peered":"2024-05-08T21:22:50.205609+0000","last_unstale":"2024-05-08T21:22:50.210416+0000","last_undegraded":"2024-05-08T21:22:50.210416+0000","last_fullsized":"2024-05-08T21:22:50.210416+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:41:28.819967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.188953+0000","last_change":"2024-05-08T21:22:49.745047+0000","last_active":"2024-05-08T21:22:50.188953+0000","last_peered":"2024-05-08T21:22:50.188953+0000","last_clean":"2024-05-08T21:22:50.188953+0000","last_became_active":"2024-05-08T21:22:49.744747+0000","last_became_peered":"2024-05-08T21:22:49.744747+0000","last_unstale":"2024-05-08T21:22:50.188953+0000","last_undegraded":"2024-05-08T21:22:50.188953+0000","last_fullsized":"2024-05-08T21:22:50.188953+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:05:26.886351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.207987+0000","last_change":"2024-05-08T21:22:50.203176+0000","last_active":"2024-05-08T21:22:50.207987+0000","last_peered":"2024-05-08T21:22:50.207987+0000","last_clean":"2024-05-08T21:22:50.207987+0000","last_became_active":"2024-05-08T21:22:50.202886+0000","last_became_peered":"2024-05-08T21:22:50.202886+0000","last_unstale":"2024-05-08T21:22:50.207987+0000","last_undegraded":"2024-05-08T21:22:50.207987+0000","last_fullsized":"2024-05-08T21:22:50.207987+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:40.200175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.199735+0000","last_change":"2024-05-08T21:22:49.742028+0000","last_active":"2024-05-08T21:22:50.199735+0000","last_peered":"2024-05-08T21:22:50.199735+0000","last_clean":"2024-05-08T21:22:50.199735+0000","last_became_active":"2024-05-08T21:22:49.741706+0000","last_became_peered":"2024-05-08T21:22:49.741706+0000","last_unstale":"2024-05-08T21:22:50.199735+0000","last_undegraded":"2024-05-08T21:22:50.199735+0000","last_fullsized":"2024-05-08T21:22:50.199735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:06:09.036243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214167+0000","last_change":"2024-05-08T21:22:50.210183+0000","last_active":"2024-05-08T21:22:50.214167+0000","last_peered":"2024-05-08T21:22:50.214167+0000","last_clean":"2024-05-08T21:22:50.214167+0000","last_became_active":"2024-05-08T21:22:50.209860+0000","last_became_peered":"2024-05-08T21:22:50.209860+0000","last_unstale":"2024-05-08T21:22:50.214167+0000","last_undegraded":"2024-05-08T21:22:50.214167+0000","last_fullsized":"2024-05-08T21:22:50.214167+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:28.358029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.188638+0000","last_change":"2024-05-08T21:22:49.740650+0000","last_active":"2024-05-08T21:22:50.188638+0000","last_peered":"2024-05-08T21:22:50.188638+0000","last_clean":"2024-05-08T21:22:50.188638+0000","last_became_active":"2024-05-08T21:22:49.740373+0000","last_became_peered":"2024-05-08T21:22:49.740373+0000","last_unstale":"2024-05-08T21:22:50.188638+0000","last_undegraded":"2024-05-08T21:22:50.188638+0000","last_fullsized":"2024-05-08T21:22:50.188638+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:36:43.622828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212952+0000","last_change":"2024-05-08T21:22:50.208073+0000","last_active":"2024-05-08T21:22:50.212952+0000","last_peered":"2024-05-08T21:22:50.212952+0000","last_clean":"2024-05-08T21:22:50.212952+0000","last_became_active":"2024-05-08T21:22:50.207506+0000","last_became_peered":"2024-05-08T21:22:50.207506+0000","last_unstale":"2024-05-08T21:22:50.212952+0000","last_undegraded":"2024-05-08T21:22:50.212952+0000","last_fullsized":"2024-05-08T21:22:50.212952+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:15:17.946198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.199222+0000","last_change":"2024-05-08T21:22:49.742392+0000","last_active":"2024-05-08T21:22:50.199222+0000","last_peered":"2024-05-08T21:22:50.199222+0000","last_clean":"2024-05-08T21:22:50.199222+0000","last_became_active":"2024-05-08T21:22:49.742093+0000","last_became_peered":"2024-05-08T21:22:49.742093+0000","last_unstale":"2024-05-08T21:22:50.199222+0000","last_undegraded":"2024-05-08T21:22:50.199222+0000","last_fullsized":"2024-05-08T21:22:50.199222+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:51.977551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.216104+0000","last_change":"2024-05-08T21:22:50.214641+0000","last_active":"2024-05-08T21:22:50.216104+0000","last_peered":"2024-05-08T21:22:50.216104+0000","last_clean":"2024-05-08T21:22:50.216104+0000","last_became_active":"2024-05-08T21:22:50.213619+0000","last_became_peered":"2024-05-08T21:22:50.213619+0000","last_unstale":"2024-05-08T21:22:50.216104+0000","last_undegraded":"2024-05-08T21:22:50.216104+0000","last_fullsized":"2024-05-08T21:22:50.216104+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:21:31.919852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.191412+0000","last_change":"2024-05-08T21:22:49.738120+0000","last_active":"2024-05-08T21:22:50.191412+0000","last_peered":"2024-05-08T21:22:50.191412+0000","last_clean":"2024-05-08T21:22:50.191412+0000","last_became_active":"2024-05-08T21:22:49.737805+0000","last_became_peered":"2024-05-08T21:22:49.737805+0000","last_unstale":"2024-05-08T21:22:50.191412+0000","last_undegraded":"2024-05-08T21:22:50.191412+0000","last_fullsized":"2024-05-08T21:22:50.191412+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:55:00.385106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212011+0000","last_change":"2024-05-08T21:22:50.205043+0000","last_active":"2024-05-08T21:22:50.212011+0000","last_peered":"2024-05-08T21:22:50.212011+0000","last_clean":"2024-05-08T21:22:50.212011+0000","last_became_active":"2024-05-08T21:22:50.204630+0000","last_became_peered":"2024-05-08T21:22:50.204630+0000","last_unstale":"2024-05-08T21:22:50.212011+0000","last_undegraded":"2024-05-08T21:22:50.212011+0000","last_fullsized":"2024-05-08T21:22:50.212011+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:26:57.146675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.205320+0000","last_change":"2024-05-08T21:22:49.736783+0000","last_active":"2024-05-08T21:22:50.205320+0000","last_peered":"2024-05-08T21:22:50.205320+0000","last_clean":"2024-05-08T21:22:50.205320+0000","last_became_active":"2024-05-08T21:22:49.736479+0000","last_became_peered":"2024-05-08T21:22:49.736479+0000","last_unstale":"2024-05-08T21:22:50.205320+0000","last_undegraded":"2024-05-08T21:22:50.205320+0000","last_fullsized":"2024-05-08T21:22:50.205320+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:24:24.156179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.206121+0000","last_change":"2024-05-08T21:22:50.202594+0000","last_active":"2024-05-08T21:22:50.206121+0000","last_peered":"2024-05-08T21:22:50.206121+0000","last_clean":"2024-05-08T21:22:50.206121+0000","last_became_active":"2024-05-08T21:22:50.202278+0000","last_became_peered":"2024-05-08T21:22:50.202278+0000","last_unstale":"2024-05-08T21:22:50.206121+0000","last_undegraded":"2024-05-08T21:22:50.206121+0000","last_fullsized":"2024-05-08T21:22:50.206121+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:44:18.055232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.198296+0000","last_change":"2024-05-08T21:22:49.739311+0000","last_active":"2024-05-08T21:22:50.198296+0000","last_peered":"2024-05-08T21:22:50.198296+0000","last_clean":"2024-05-08T21:22:50.198296+0000","last_became_active":"2024-05-08T21:22:49.739006+0000","last_became_peered":"2024-05-08T21:22:49.739006+0000","last_unstale":"2024-05-08T21:22:50.198296+0000","last_undegraded":"2024-05-08T21:22:50.198296+0000","last_fullsized":"2024-05-08T21:22:50.198296+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:39:42.856227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":176,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-08T21:22:49.723088+0000","last_change":"2024-05-08T21:21:42.333097+0000","last_active":"2024-05-08T21:22:49.723088+0000","last_peered":"2024-05-08T21:22:49.723088+0000","last_clean":"2024-05-08T21:22:49.723088+0000","last_became_active":"2024-05-08T21:21:42.332033+0000","last_became_peered":"2024-05-08T21:21:42.332033+0000","last_unstale":"2024-05-08T21:22:49.723088+0000","last_undegraded":"2024-05-08T21:22:49.723088+0000","last_fullsized":"2024-05-08T21:22:49.723088+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_clean_scrub_stamp":"2024-05-08T21:21:40.862622+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:39:24.772551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":970752,"data_stored":736053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.55800000000000005}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.627},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.44700000000000001}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.59099999999999997}]},{"osd":3,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.499}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.42899999999999999}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.39700000000000002}]},{"osd":6,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.52100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.53000000000000003}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.41099999999999998}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.375},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.437}]},{"osd":3,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":1.716}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.497},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.48699999999999999}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.57399999999999995}]},{"osd":7,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.46999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":147181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.26700000000000002}]},{"osd":2,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.33100000000000002}]},{"osd":3,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.432}]},{"osd":4,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.45100000000000001}]},{"osd":5,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.54900000000000004}]},{"osd":6,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.49399999999999999}]},{"osd":7,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.63}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":966656,"data_stored":735963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.29899999999999999}]},{"osd":2,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.34899999999999998}]},{"osd":3,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.318}]},{"osd":4,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.63400000000000001}]},{"osd":5,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.56299999999999994}]},{"osd":6,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.61199999999999999}]},{"osd":7,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.57299999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":376832,"data_stored":145665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.435},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.39000000000000001}]},{"osd":1,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32300000000000001}]},{"osd":3,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.378},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.47399999999999998}]},{"osd":4,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.65400000000000003}]},{"osd":5,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.66700000000000004}]},{"osd":6,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.622},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.69499999999999995}]},{"osd":7,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.79200000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":138820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":2.7919999999999998,"5min":2.7919999999999998,"15min":2.7919999999999998},"last":0.68700000000000006}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":4.2370000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.435}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":2.9369999999999998,"5min":2.9369999999999998,"15min":2.9369999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":4.2969999999999997}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.8380000000000001,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":2.7490000000000001}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.9060000000000001,"5min":2.9060000000000001,"15min":2.9060000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":5.1699999999999999}]},{"osd":6,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":2.8730000000000002,"5min":2.8730000000000002,"15min":2.8730000000000002},"last":0.55100000000000005}]},{"osd":7,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":5.2030000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":1.1399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":138820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.51100000000000001}]},{"osd":1,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.47099999999999997}]},{"osd":2,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.48699999999999999}]},{"osd":5,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.63}]},{"osd":6,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60099999999999998}]},{"osd":7,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.65400000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":138820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.33500000000000002}]},{"osd":1,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.52500000000000002}]},{"osd":2,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.55400000000000005}]},{"osd":3,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.441}]},{"osd":4,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.42499999999999999}]},{"osd":6,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.47899999999999998}]},{"osd":7,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.49199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T21:22:54.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T21:22:54.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.604+0000 7ff1dea8d700 1 Processor -- start 2024-05-08T21:22:54.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.605+0000 7ff1dea8d700 1 -- start start 2024-05-08T21:22:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.605+0000 7ff1dea8d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff1d80d0eb0 0x7ff1d81688c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.605+0000 7ff1dea8d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d80cd980 con 0x7ff1d80cf840 2024-05-08T21:22:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.605+0000 7ff1dea8d700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d80ccdb0 con 0x7ff1d806a960 2024-05-08T21:22:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.605+0000 7ff1dea8d700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1d80cd050 con 0x7ff1d80d0eb0 2024-05-08T21:22:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1dc829700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff1d80cf840 0x7ff1d80af440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59790/0 (socket says 172.21.15.3:59790) 2024-05-08T21:22:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1dc829700 1 -- 172.21.15.3:0/1851862705 learned_addr learned my addr 172.21.15.3:0/1851862705 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1dd02a700 1 --2- 172.21.15.3:0/1851862705 >> v2:172.21.15.186:3300/0 conn(0x7ff1d80d0eb0 0x7ff1d81688c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1d77fe700 1 -- 172.21.15.3:0/1851862705 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466737390 0 0) 0x7ff1d80cd980 con 0x7ff1d80cf840 2024-05-08T21:22:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1d77fe700 1 -- 172.21.15.3:0/1851862705 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c00035b0 con 0x7ff1d80cf840 2024-05-08T21:22:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1d77fe700 1 -- 172.21.15.3:0/1851862705 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3173716533 0 0) 0x7ff1c00035b0 con 0x7ff1d80cf840 2024-05-08T21:22:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1d77fe700 1 -- 172.21.15.3:0/1851862705 >> v2:172.21.15.186:3300/0 conn(0x7ff1d80d0eb0 msgr2=0x7ff1d81688c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1d77fe700 1 --2- 172.21.15.3:0/1851862705 >> v2:172.21.15.186:3300/0 conn(0x7ff1d80d0eb0 0x7ff1d81688c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1d77fe700 1 -- 172.21.15.3:0/1851862705 >> v1:172.21.15.186:6789/0 conn(0x7ff1d806a960 legacy=0x7ff1d815d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1d77fe700 1 -- 172.21.15.3:0/1851862705 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d80cd330 con 0x7ff1d80cf840 2024-05-08T21:22:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1d77fe700 1 -- 172.21.15.3:0/1851862705 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7ff1c8001670 con 0x7ff1d80cf840 2024-05-08T21:22:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1d77fe700 1 -- 172.21.15.3:0/1851862705 >> v1:172.21.15.3:6789/0 conn(0x7ff1d80cf840 legacy=0x7ff1d80af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1d77fe700 1 --2- 172.21.15.3:0/1851862705 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c0003a00 0x7ff1c0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1d77fe700 1 --2- 172.21.15.3:0/1851862705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1c0004400 0x7ff1c00048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1d77fe700 1 --2- 172.21.15.3:0/1851862705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c00050d0 0x7ff1c000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1d77fe700 1 -- 172.21.15.3:0/1851862705 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d80cd050 con 0x7ff1c0004400 2024-05-08T21:22:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1d77fe700 1 -- 172.21.15.3:0/1851862705 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c000c030 con 0x7ff1c00050d0 2024-05-08T21:22:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.606+0000 7ff1d77fe700 1 -- 172.21.15.3:0/1851862705 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c000c1f0 con 0x7ff1c0003a00 2024-05-08T21:22:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.607+0000 7ff1d7fff700 1 --2- 172.21.15.3:0/1851862705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1c0004400 0x7ff1c00048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.607+0000 7ff1dc829700 1 --2- 172.21.15.3:0/1851862705 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c0003a00 0x7ff1c0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.607+0000 7ff1dd02a700 1 --2- 172.21.15.3:0/1851862705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c00050d0 0x7ff1c000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.607+0000 7ff1d7fff700 1 -- 172.21.15.3:0/1851862705 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c0003a00 msgr2=0x7ff1c0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.607+0000 7ff1d7fff700 1 --2- 172.21.15.3:0/1851862705 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c0003a00 0x7ff1c0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.607+0000 7ff1d7fff700 1 -- 172.21.15.3:0/1851862705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c00050d0 msgr2=0x7ff1c000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:54.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.607+0000 7ff1d7fff700 1 --2- 172.21.15.3:0/1851862705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c00050d0 0x7ff1c000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.607+0000 7ff1dd02a700 1 -- 172.21.15.3:0/1851862705 reap_dead start 2024-05-08T21:22:54.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.607+0000 7ff1d7fff700 1 -- 172.21.15.3:0/1851862705 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c00035b0 con 0x7ff1c0004400 2024-05-08T21:22:54.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.607+0000 7ff1d7fff700 1 --2- 172.21.15.3:0/1851862705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1c0004400 0x7ff1c00048d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0018b0 tx=0x7ff1c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd86e6fa71345738 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:54.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.607+0000 7ff1d77fe700 1 -- 172.21.15.3:0/1851862705 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc001370 con 0x7ff1c0004400 2024-05-08T21:22:54.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.607+0000 7ff1d77fe700 1 -- 172.21.15.3:0/1851862705 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1cc0052d0 con 0x7ff1c0004400 2024-05-08T21:22:54.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.607+0000 7ff1d77fe700 1 -- 172.21.15.3:0/1851862705 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc003680 con 0x7ff1c0004400 2024-05-08T21:22:54.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.608+0000 7ff1dea8d700 1 -- 172.21.15.3:0/1851862705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1c0004400 msgr2=0x7ff1c00048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:54.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.608+0000 7ff1dea8d700 1 --2- 172.21.15.3:0/1851862705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1c0004400 0x7ff1c00048d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0018b0 tx=0x7ff1c0005880 comp rx=0 tx=0).stop 2024-05-08T21:22:54.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.608+0000 7ff1dea8d700 1 -- 172.21.15.3:0/1851862705 shutdown_connections 2024-05-08T21:22:54.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.608+0000 7ff1dea8d700 1 --2- 172.21.15.3:0/1851862705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1c0004400 0x7ff1c00048d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.608+0000 7ff1dea8d700 1 -- 172.21.15.3:0/1851862705 >> 172.21.15.3:0/1851862705 conn(0x7ff1d80d0320 msgr2=0x7ff1d80d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:54.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.608+0000 7ff1dea8d700 1 -- 172.21.15.3:0/1851862705 shutdown_connections 2024-05-08T21:22:54.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.608+0000 7ff1dea8d700 1 -- 172.21.15.3:0/1851862705 wait complete. 2024-05-08T21:22:54.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.609+0000 7ff1dea8d700 1 Processor -- start 2024-05-08T21:22:54.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.609+0000 7ff1dea8d700 1 -- start start 2024-05-08T21:22:54.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.610+0000 7ff1dea8d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c0004400 0x7ff1d80b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:54.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.610+0000 7ff1dea8d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1d81f6db0 0x7ff1d80aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:54.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.610+0000 7ff1dea8d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1d81f7230 0x7ff1d80aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:54.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.610+0000 7ff1dea8d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d81550b0 con 0x7ff1d81f7230 2024-05-08T21:22:54.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.610+0000 7ff1dea8d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d81f65f0 con 0x7ff1d81f6db0 2024-05-08T21:22:54.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.610+0000 7ff1dea8d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d81f38c0 con 0x7ff1c0004400 2024-05-08T21:22:54.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.610+0000 7ff1dd02a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1d81f7230 0x7ff1d80aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:54.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.610+0000 7ff1dd02a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1d81f7230 0x7ff1d80aef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45530/0 (socket says 172.21.15.3:45530) 2024-05-08T21:22:54.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.610+0000 7ff1dd02a700 1 -- 172.21.15.3:0/631395122 learned_addr learned my addr 172.21.15.3:0/631395122 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:54.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.611+0000 7ff1d7fff700 1 --2- 172.21.15.3:0/631395122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1d81f6db0 0x7ff1d80aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:54.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.611+0000 7ff1dc829700 1 --2- 172.21.15.3:0/631395122 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c0004400 0x7ff1d80b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:54.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.611+0000 7ff1dd02a700 1 -- 172.21.15.3:0/631395122 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c0004400 msgr2=0x7ff1d80b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:54.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.611+0000 7ff1dd02a700 1 --2- 172.21.15.3:0/631395122 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c0004400 0x7ff1d80b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.611+0000 7ff1dd02a700 1 -- 172.21.15.3:0/631395122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1d81f6db0 msgr2=0x7ff1d80aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.611+0000 7ff1dd02a700 1 --2- 172.21.15.3:0/631395122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1d81f6db0 0x7ff1d80aea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.611+0000 7ff1dd02a700 1 -- 172.21.15.3:0/631395122 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d81f3ba0 con 0x7ff1d81f7230 2024-05-08T21:22:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.611+0000 7ff1dd02a700 1 --2- 172.21.15.3:0/631395122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1d81f7230 0x7ff1d80aef60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80ced30 tx=0x7ff1d0011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.611+0000 7ff1d57fa700 1 -- 172.21.15.3:0/631395122 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d000e040 con 0x7ff1d81f7230 2024-05-08T21:22:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.611+0000 7ff1d57fa700 1 -- 172.21.15.3:0/631395122 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d000f040 con 0x7ff1d81f7230 2024-05-08T21:22:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.611+0000 7ff1dea8d700 1 -- 172.21.15.3:0/631395122 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d81f3e80 con 0x7ff1d81f7230 2024-05-08T21:22:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.611+0000 7ff1d57fa700 1 -- 172.21.15.3:0/631395122 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d000fdb0 con 0x7ff1d81f7230 2024-05-08T21:22:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.611+0000 7ff1dea8d700 1 -- 172.21.15.3:0/631395122 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d81e83c0 con 0x7ff1d81f7230 2024-05-08T21:22:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.612+0000 7ff1dea8d700 1 -- 172.21.15.3:0/631395122 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d81f4010 con 0x7ff1d81f7230 2024-05-08T21:22:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.613+0000 7ff1d57fa700 1 -- 172.21.15.3:0/631395122 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff1d0022460 con 0x7ff1d81f7230 2024-05-08T21:22:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.613+0000 7ff1d57fa700 1 --2- 172.21.15.3:0/631395122 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff1c406ac90 0x7ff1c406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.613+0000 7ff1d57fa700 1 -- 172.21.15.3:0/631395122 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1d0089a30 con 0x7ff1d81f7230 2024-05-08T21:22:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.613+0000 7ff1dc829700 1 --2- 172.21.15.3:0/631395122 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff1c406ac90 0x7ff1c406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:54.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.614+0000 7ff1dc829700 1 --2- 172.21.15.3:0/631395122 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff1c406ac90 0x7ff1c406d150 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8000c00 tx=0x7ff1c8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:54.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.618+0000 7ff1d57fa700 1 -- 172.21.15.3:0/631395122 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff1d0054c10 con 0x7ff1d81f7230 2024-05-08T21:22:54.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.832+0000 7ff1dea8d700 1 -- 172.21.15.3:0/631395122 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff1d8068000 con 0x7ff1c406ac90 2024-05-08T21:22:54.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.841+0000 7ff1d57fa700 1 -- 172.21.15.3:0/631395122 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289235 (secure 0 0 0) 0x7ff1d8068000 con 0x7ff1c406ac90 2024-05-08T21:22:54.846 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:54.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.845+0000 7ff1dea8d700 1 -- 172.21.15.3:0/631395122 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff1c406ac90 msgr2=0x7ff1c406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:54.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.845+0000 7ff1dea8d700 1 --2- 172.21.15.3:0/631395122 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff1c406ac90 0x7ff1c406d150 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8000c00 tx=0x7ff1c8009040 comp rx=0 tx=0).stop 2024-05-08T21:22:54.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.845+0000 7ff1dea8d700 1 -- 172.21.15.3:0/631395122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1d81f7230 msgr2=0x7ff1d80aef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:54.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.845+0000 7ff1dea8d700 1 --2- 172.21.15.3:0/631395122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1d81f7230 0x7ff1d80aef60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80ced30 tx=0x7ff1d0011f80 comp rx=0 tx=0).stop 2024-05-08T21:22:54.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.845+0000 7ff1dea8d700 1 -- 172.21.15.3:0/631395122 shutdown_connections 2024-05-08T21:22:54.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.845+0000 7ff1dea8d700 1 --2- 172.21.15.3:0/631395122 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c0004400 0x7ff1d80b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.845+0000 7ff1dea8d700 1 --2- 172.21.15.3:0/631395122 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff1c406ac90 0x7ff1c406d150 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.845+0000 7ff1dea8d700 1 --2- 172.21.15.3:0/631395122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1d81f7230 0x7ff1d80aef60 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.845+0000 7ff1dea8d700 1 --2- 172.21.15.3:0/631395122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1d81f6db0 0x7ff1d80aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:54.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.845+0000 7ff1dea8d700 1 -- 172.21.15.3:0/631395122 >> 172.21.15.3:0/631395122 conn(0x7ff1d80d0320 msgr2=0x7ff1d815c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:54.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.845+0000 7ff1dea8d700 1 -- 172.21.15.3:0/631395122 shutdown_connections 2024-05-08T21:22:54.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:54.845+0000 7ff1dea8d700 1 -- 172.21.15.3:0/631395122 wait complete. 2024-05-08T21:22:54.849 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-08T21:22:54.860 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2024-05-08T21:22:54.545774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220252,"kb_used_data":4068,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754468,"statfs":{"total":773094113280,"available":772868575232,"internally_reserved":0,"allocated":4165632,"data_stored":2327533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.826643"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214117+0000","last_change":"2024-05-08T21:22:50.209349+0000","last_active":"2024-05-08T21:22:50.214117+0000","last_peered":"2024-05-08T21:22:50.214117+0000","last_clean":"2024-05-08T21:22:50.214117+0000","last_became_active":"2024-05-08T21:22:50.209092+0000","last_became_peered":"2024-05-08T21:22:50.209092+0000","last_unstale":"2024-05-08T21:22:50.214117+0000","last_undegraded":"2024-05-08T21:22:50.214117+0000","last_fullsized":"2024-05-08T21:22:50.214117+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:52:47.510546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213268+0000","last_change":"2024-05-08T21:22:50.205276+0000","last_active":"2024-05-08T21:22:50.213268+0000","last_peered":"2024-05-08T21:22:50.213268+0000","last_clean":"2024-05-08T21:22:50.213268+0000","last_became_active":"2024-05-08T21:22:50.205028+0000","last_became_peered":"2024-05-08T21:22:50.205028+0000","last_unstale":"2024-05-08T21:22:50.213268+0000","last_undegraded":"2024-05-08T21:22:50.213268+0000","last_fullsized":"2024-05-08T21:22:50.213268+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:24:34.674332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210756+0000","last_change":"2024-05-08T21:22:50.205185+0000","last_active":"2024-05-08T21:22:50.210756+0000","last_peered":"2024-05-08T21:22:50.210756+0000","last_clean":"2024-05-08T21:22:50.210756+0000","last_became_active":"2024-05-08T21:22:50.204826+0000","last_became_peered":"2024-05-08T21:22:50.204826+0000","last_unstale":"2024-05-08T21:22:50.210756+0000","last_undegraded":"2024-05-08T21:22:50.210756+0000","last_fullsized":"2024-05-08T21:22:50.210756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:05:25.952745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213792+0000","last_change":"2024-05-08T21:22:50.210536+0000","last_active":"2024-05-08T21:22:50.213792+0000","last_peered":"2024-05-08T21:22:50.213792+0000","last_clean":"2024-05-08T21:22:50.213792+0000","last_became_active":"2024-05-08T21:22:50.210248+0000","last_became_peered":"2024-05-08T21:22:50.210248+0000","last_unstale":"2024-05-08T21:22:50.213792+0000","last_undegraded":"2024-05-08T21:22:50.213792+0000","last_fullsized":"2024-05-08T21:22:50.213792+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:27:40.074356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.216126+0000","last_change":"2024-05-08T21:22:50.208946+0000","last_active":"2024-05-08T21:22:50.216126+0000","last_peered":"2024-05-08T21:22:50.216126+0000","last_clean":"2024-05-08T21:22:50.216126+0000","last_became_active":"2024-05-08T21:22:50.208626+0000","last_became_peered":"2024-05-08T21:22:50.208626+0000","last_unstale":"2024-05-08T21:22:50.216126+0000","last_undegraded":"2024-05-08T21:22:50.216126+0000","last_fullsized":"2024-05-08T21:22:50.216126+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:09.165560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214059+0000","last_change":"2024-05-08T21:22:50.207734+0000","last_active":"2024-05-08T21:22:50.214059+0000","last_peered":"2024-05-08T21:22:50.214059+0000","last_clean":"2024-05-08T21:22:50.214059+0000","last_became_active":"2024-05-08T21:22:50.207392+0000","last_became_peered":"2024-05-08T21:22:50.207392+0000","last_unstale":"2024-05-08T21:22:50.214059+0000","last_undegraded":"2024-05-08T21:22:50.214059+0000","last_fullsized":"2024-05-08T21:22:50.214059+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:23:48.163955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212627+0000","last_change":"2024-05-08T21:22:50.210352+0000","last_active":"2024-05-08T21:22:50.212627+0000","last_peered":"2024-05-08T21:22:50.212627+0000","last_clean":"2024-05-08T21:22:50.212627+0000","last_became_active":"2024-05-08T21:22:50.210060+0000","last_became_peered":"2024-05-08T21:22:50.210060+0000","last_unstale":"2024-05-08T21:22:50.212627+0000","last_undegraded":"2024-05-08T21:22:50.212627+0000","last_fullsized":"2024-05-08T21:22:50.212627+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:58:46.704338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.211040+0000","last_change":"2024-05-08T21:22:50.203181+0000","last_active":"2024-05-08T21:22:50.211040+0000","last_peered":"2024-05-08T21:22:50.211040+0000","last_clean":"2024-05-08T21:22:50.211040+0000","last_became_active":"2024-05-08T21:22:50.202914+0000","last_became_peered":"2024-05-08T21:22:50.202914+0000","last_unstale":"2024-05-08T21:22:50.211040+0000","last_undegraded":"2024-05-08T21:22:50.211040+0000","last_fullsized":"2024-05-08T21:22:50.211040+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:24:39.295215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212281+0000","last_change":"2024-05-08T21:22:50.207050+0000","last_active":"2024-05-08T21:22:50.212281+0000","last_peered":"2024-05-08T21:22:50.212281+0000","last_clean":"2024-05-08T21:22:50.212281+0000","last_became_active":"2024-05-08T21:22:50.206737+0000","last_became_peered":"2024-05-08T21:22:50.206737+0000","last_unstale":"2024-05-08T21:22:50.212281+0000","last_undegraded":"2024-05-08T21:22:50.212281+0000","last_fullsized":"2024-05-08T21:22:50.212281+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:52:47.510546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213836+0000","last_change":"2024-05-08T21:22:50.212260+0000","last_active":"2024-05-08T21:22:50.213836+0000","last_peered":"2024-05-08T21:22:50.213836+0000","last_clean":"2024-05-08T21:22:50.213836+0000","last_became_active":"2024-05-08T21:22:50.211928+0000","last_became_peered":"2024-05-08T21:22:50.211928+0000","last_unstale":"2024-05-08T21:22:50.213836+0000","last_undegraded":"2024-05-08T21:22:50.213836+0000","last_fullsized":"2024-05-08T21:22:50.213836+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:43:00.051225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212482+0000","last_change":"2024-05-08T21:22:50.205779+0000","last_active":"2024-05-08T21:22:50.212482+0000","last_peered":"2024-05-08T21:22:50.212482+0000","last_clean":"2024-05-08T21:22:50.212482+0000","last_became_active":"2024-05-08T21:22:50.205533+0000","last_became_peered":"2024-05-08T21:22:50.205533+0000","last_unstale":"2024-05-08T21:22:50.212482+0000","last_undegraded":"2024-05-08T21:22:50.212482+0000","last_fullsized":"2024-05-08T21:22:50.212482+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:24:34.674332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210296+0000","last_change":"2024-05-08T21:22:50.207359+0000","last_active":"2024-05-08T21:22:50.210296+0000","last_peered":"2024-05-08T21:22:50.210296+0000","last_clean":"2024-05-08T21:22:50.210296+0000","last_became_active":"2024-05-08T21:22:50.207045+0000","last_became_peered":"2024-05-08T21:22:50.207045+0000","last_unstale":"2024-05-08T21:22:50.210296+0000","last_undegraded":"2024-05-08T21:22:50.210296+0000","last_fullsized":"2024-05-08T21:22:50.210296+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:24:34.674332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.215394+0000","last_change":"2024-05-08T21:22:50.207671+0000","last_active":"2024-05-08T21:22:50.215394+0000","last_peered":"2024-05-08T21:22:50.215394+0000","last_clean":"2024-05-08T21:22:50.215394+0000","last_became_active":"2024-05-08T21:22:50.207119+0000","last_became_peered":"2024-05-08T21:22:50.207119+0000","last_unstale":"2024-05-08T21:22:50.215394+0000","last_undegraded":"2024-05-08T21:22:50.215394+0000","last_fullsized":"2024-05-08T21:22:50.215394+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:59:37.805505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.217164+0000","last_change":"2024-05-08T21:22:50.216066+0000","last_active":"2024-05-08T21:22:50.217164+0000","last_peered":"2024-05-08T21:22:50.217164+0000","last_clean":"2024-05-08T21:22:50.217164+0000","last_became_active":"2024-05-08T21:22:50.215796+0000","last_became_peered":"2024-05-08T21:22:50.215796+0000","last_unstale":"2024-05-08T21:22:50.217164+0000","last_undegraded":"2024-05-08T21:22:50.217164+0000","last_fullsized":"2024-05-08T21:22:50.217164+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:24:39.295215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214235+0000","last_change":"2024-05-08T21:22:50.211332+0000","last_active":"2024-05-08T21:22:50.214235+0000","last_peered":"2024-05-08T21:22:50.214235+0000","last_clean":"2024-05-08T21:22:50.214235+0000","last_became_active":"2024-05-08T21:22:50.210865+0000","last_became_peered":"2024-05-08T21:22:50.210865+0000","last_unstale":"2024-05-08T21:22:50.214235+0000","last_undegraded":"2024-05-08T21:22:50.214235+0000","last_fullsized":"2024-05-08T21:22:50.214235+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:24:39.295215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.216263+0000","last_change":"2024-05-08T21:22:50.215021+0000","last_active":"2024-05-08T21:22:50.216263+0000","last_peered":"2024-05-08T21:22:50.216263+0000","last_clean":"2024-05-08T21:22:50.216263+0000","last_became_active":"2024-05-08T21:22:50.214821+0000","last_became_peered":"2024-05-08T21:22:50.214821+0000","last_unstale":"2024-05-08T21:22:50.216263+0000","last_undegraded":"2024-05-08T21:22:50.216263+0000","last_fullsized":"2024-05-08T21:22:50.216263+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:14.396616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212692+0000","last_change":"2024-05-08T21:22:50.205302+0000","last_active":"2024-05-08T21:22:50.212692+0000","last_peered":"2024-05-08T21:22:50.212692+0000","last_clean":"2024-05-08T21:22:50.212692+0000","last_became_active":"2024-05-08T21:22:50.204962+0000","last_became_peered":"2024-05-08T21:22:50.204962+0000","last_unstale":"2024-05-08T21:22:50.212692+0000","last_undegraded":"2024-05-08T21:22:50.212692+0000","last_fullsized":"2024-05-08T21:22:50.212692+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:09:24.655227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214257+0000","last_change":"2024-05-08T21:22:50.208448+0000","last_active":"2024-05-08T21:22:50.214257+0000","last_peered":"2024-05-08T21:22:50.214257+0000","last_clean":"2024-05-08T21:22:50.214257+0000","last_became_active":"2024-05-08T21:22:50.208210+0000","last_became_peered":"2024-05-08T21:22:50.208210+0000","last_unstale":"2024-05-08T21:22:50.214257+0000","last_undegraded":"2024-05-08T21:22:50.214257+0000","last_fullsized":"2024-05-08T21:22:50.214257+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:38:13.012930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214015+0000","last_change":"2024-05-08T21:22:50.206974+0000","last_active":"2024-05-08T21:22:50.214015+0000","last_peered":"2024-05-08T21:22:50.214015+0000","last_clean":"2024-05-08T21:22:50.214015+0000","last_became_active":"2024-05-08T21:22:50.206688+0000","last_became_peered":"2024-05-08T21:22:50.206688+0000","last_unstale":"2024-05-08T21:22:50.214015+0000","last_undegraded":"2024-05-08T21:22:50.214015+0000","last_fullsized":"2024-05-08T21:22:50.214015+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:41:28.819967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.215337+0000","last_change":"2024-05-08T21:22:50.207398+0000","last_active":"2024-05-08T21:22:50.215337+0000","last_peered":"2024-05-08T21:22:50.215337+0000","last_clean":"2024-05-08T21:22:50.215337+0000","last_became_active":"2024-05-08T21:22:50.207040+0000","last_became_peered":"2024-05-08T21:22:50.207040+0000","last_unstale":"2024-05-08T21:22:50.215337+0000","last_undegraded":"2024-05-08T21:22:50.215337+0000","last_fullsized":"2024-05-08T21:22:50.215337+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:28.358029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.206511+0000","last_change":"2024-05-08T21:22:50.199407+0000","last_active":"2024-05-08T21:22:50.206511+0000","last_peered":"2024-05-08T21:22:50.206511+0000","last_clean":"2024-05-08T21:22:50.206511+0000","last_became_active":"2024-05-08T21:22:50.199133+0000","last_became_peered":"2024-05-08T21:22:50.199133+0000","last_unstale":"2024-05-08T21:22:50.206511+0000","last_undegraded":"2024-05-08T21:22:50.206511+0000","last_fullsized":"2024-05-08T21:22:50.206511+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:40.200175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212126+0000","last_change":"2024-05-08T21:22:50.207042+0000","last_active":"2024-05-08T21:22:50.212126+0000","last_peered":"2024-05-08T21:22:50.212126+0000","last_clean":"2024-05-08T21:22:50.212126+0000","last_became_active":"2024-05-08T21:22:50.206554+0000","last_became_peered":"2024-05-08T21:22:50.206554+0000","last_unstale":"2024-05-08T21:22:50.212126+0000","last_undegraded":"2024-05-08T21:22:50.212126+0000","last_fullsized":"2024-05-08T21:22:50.212126+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:15:17.946198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.204324+0000","last_change":"2024-05-08T21:22:50.200284+0000","last_active":"2024-05-08T21:22:50.204324+0000","last_peered":"2024-05-08T21:22:50.204324+0000","last_clean":"2024-05-08T21:22:50.204324+0000","last_became_active":"2024-05-08T21:22:50.200004+0000","last_became_peered":"2024-05-08T21:22:50.200004+0000","last_unstale":"2024-05-08T21:22:50.204324+0000","last_undegraded":"2024-05-08T21:22:50.204324+0000","last_fullsized":"2024-05-08T21:22:50.204324+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:28.358029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.211103+0000","last_change":"2024-05-08T21:22:50.208138+0000","last_active":"2024-05-08T21:22:50.211103+0000","last_peered":"2024-05-08T21:22:50.211103+0000","last_clean":"2024-05-08T21:22:50.211103+0000","last_became_active":"2024-05-08T21:22:50.207813+0000","last_became_peered":"2024-05-08T21:22:50.207813+0000","last_unstale":"2024-05-08T21:22:50.211103+0000","last_undegraded":"2024-05-08T21:22:50.211103+0000","last_fullsized":"2024-05-08T21:22:50.211103+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:07:17.470994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210704+0000","last_change":"2024-05-08T21:22:50.202937+0000","last_active":"2024-05-08T21:22:50.210704+0000","last_peered":"2024-05-08T21:22:50.210704+0000","last_clean":"2024-05-08T21:22:50.210704+0000","last_became_active":"2024-05-08T21:22:50.202633+0000","last_became_peered":"2024-05-08T21:22:50.202633+0000","last_unstale":"2024-05-08T21:22:50.210704+0000","last_undegraded":"2024-05-08T21:22:50.210704+0000","last_fullsized":"2024-05-08T21:22:50.210704+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:48:46.138832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213247+0000","last_change":"2024-05-08T21:22:50.204581+0000","last_active":"2024-05-08T21:22:50.213247+0000","last_peered":"2024-05-08T21:22:50.213247+0000","last_clean":"2024-05-08T21:22:50.213247+0000","last_became_active":"2024-05-08T21:22:50.204228+0000","last_became_peered":"2024-05-08T21:22:50.204228+0000","last_unstale":"2024-05-08T21:22:50.213247+0000","last_undegraded":"2024-05-08T21:22:50.213247+0000","last_fullsized":"2024-05-08T21:22:50.213247+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:43:00.051225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213651+0000","last_change":"2024-05-08T21:22:50.210823+0000","last_active":"2024-05-08T21:22:50.213651+0000","last_peered":"2024-05-08T21:22:50.213651+0000","last_clean":"2024-05-08T21:22:50.213651+0000","last_became_active":"2024-05-08T21:22:50.210517+0000","last_became_peered":"2024-05-08T21:22:50.210517+0000","last_unstale":"2024-05-08T21:22:50.213651+0000","last_undegraded":"2024-05-08T21:22:50.213651+0000","last_fullsized":"2024-05-08T21:22:50.213651+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:09:24.655227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.208862+0000","last_change":"2024-05-08T21:22:50.201860+0000","last_active":"2024-05-08T21:22:50.208862+0000","last_peered":"2024-05-08T21:22:50.208862+0000","last_clean":"2024-05-08T21:22:50.208862+0000","last_became_active":"2024-05-08T21:22:50.201204+0000","last_became_peered":"2024-05-08T21:22:50.201204+0000","last_unstale":"2024-05-08T21:22:50.208862+0000","last_undegraded":"2024-05-08T21:22:50.208862+0000","last_fullsized":"2024-05-08T21:22:50.208862+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:23:48.163955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212049+0000","last_change":"2024-05-08T21:22:50.206666+0000","last_active":"2024-05-08T21:22:50.212049+0000","last_peered":"2024-05-08T21:22:50.212049+0000","last_clean":"2024-05-08T21:22:50.212049+0000","last_became_active":"2024-05-08T21:22:50.206283+0000","last_became_peered":"2024-05-08T21:22:50.206283+0000","last_unstale":"2024-05-08T21:22:50.212049+0000","last_undegraded":"2024-05-08T21:22:50.212049+0000","last_fullsized":"2024-05-08T21:22:50.212049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:53:39.386458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212345+0000","last_change":"2024-05-08T21:22:50.205319+0000","last_active":"2024-05-08T21:22:50.212345+0000","last_peered":"2024-05-08T21:22:50.212345+0000","last_clean":"2024-05-08T21:22:50.212345+0000","last_became_active":"2024-05-08T21:22:50.205014+0000","last_became_peered":"2024-05-08T21:22:50.205014+0000","last_unstale":"2024-05-08T21:22:50.212345+0000","last_undegraded":"2024-05-08T21:22:50.212345+0000","last_fullsized":"2024-05-08T21:22:50.212345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:10:48.376692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214174+0000","last_change":"2024-05-08T21:22:50.210271+0000","last_active":"2024-05-08T21:22:50.214174+0000","last_peered":"2024-05-08T21:22:50.214174+0000","last_clean":"2024-05-08T21:22:50.214174+0000","last_became_active":"2024-05-08T21:22:50.209998+0000","last_became_peered":"2024-05-08T21:22:50.209998+0000","last_unstale":"2024-05-08T21:22:50.214174+0000","last_undegraded":"2024-05-08T21:22:50.214174+0000","last_fullsized":"2024-05-08T21:22:50.214174+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:09.165560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213641+0000","last_change":"2024-05-08T21:22:50.211741+0000","last_active":"2024-05-08T21:22:50.213641+0000","last_peered":"2024-05-08T21:22:50.213641+0000","last_clean":"2024-05-08T21:22:50.213641+0000","last_became_active":"2024-05-08T21:22:50.211430+0000","last_became_peered":"2024-05-08T21:22:50.211430+0000","last_unstale":"2024-05-08T21:22:50.213641+0000","last_undegraded":"2024-05-08T21:22:50.213641+0000","last_fullsized":"2024-05-08T21:22:50.213641+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:23:48.163955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.204975+0000","last_change":"2024-05-08T21:22:50.201116+0000","last_active":"2024-05-08T21:22:50.204975+0000","last_peered":"2024-05-08T21:22:50.204975+0000","last_clean":"2024-05-08T21:22:50.204975+0000","last_became_active":"2024-05-08T21:22:50.200817+0000","last_became_peered":"2024-05-08T21:22:50.200817+0000","last_unstale":"2024-05-08T21:22:50.204975+0000","last_undegraded":"2024-05-08T21:22:50.204975+0000","last_fullsized":"2024-05-08T21:22:50.204975+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:09.165560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210819+0000","last_change":"2024-05-08T21:22:49.740529+0000","last_active":"2024-05-08T21:22:50.210819+0000","last_peered":"2024-05-08T21:22:50.210819+0000","last_clean":"2024-05-08T21:22:50.210819+0000","last_became_active":"2024-05-08T21:22:49.740247+0000","last_became_peered":"2024-05-08T21:22:49.740247+0000","last_unstale":"2024-05-08T21:22:50.210819+0000","last_undegraded":"2024-05-08T21:22:50.210819+0000","last_fullsized":"2024-05-08T21:22:50.210819+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:53:44.252529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.211713+0000","last_change":"2024-05-08T21:22:50.208077+0000","last_active":"2024-05-08T21:22:50.211713+0000","last_peered":"2024-05-08T21:22:50.211713+0000","last_clean":"2024-05-08T21:22:50.211713+0000","last_became_active":"2024-05-08T21:22:50.207729+0000","last_became_peered":"2024-05-08T21:22:50.207729+0000","last_unstale":"2024-05-08T21:22:50.211713+0000","last_undegraded":"2024-05-08T21:22:50.211713+0000","last_fullsized":"2024-05-08T21:22:50.211713+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:40.200175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.200537+0000","last_change":"2024-05-08T21:22:49.743083+0000","last_active":"2024-05-08T21:22:50.200537+0000","last_peered":"2024-05-08T21:22:50.200537+0000","last_clean":"2024-05-08T21:22:50.200537+0000","last_became_active":"2024-05-08T21:22:49.742790+0000","last_became_peered":"2024-05-08T21:22:49.742790+0000","last_unstale":"2024-05-08T21:22:50.200537+0000","last_undegraded":"2024-05-08T21:22:50.200537+0000","last_fullsized":"2024-05-08T21:22:50.200537+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:06:42.743465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212465+0000","last_change":"2024-05-08T21:22:50.207141+0000","last_active":"2024-05-08T21:22:50.212465+0000","last_peered":"2024-05-08T21:22:50.212465+0000","last_clean":"2024-05-08T21:22:50.212465+0000","last_became_active":"2024-05-08T21:22:50.206838+0000","last_became_peered":"2024-05-08T21:22:50.206838+0000","last_unstale":"2024-05-08T21:22:50.212465+0000","last_undegraded":"2024-05-08T21:22:50.212465+0000","last_fullsized":"2024-05-08T21:22:50.212465+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:16:58.048181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.206671+0000","last_change":"2024-05-08T21:22:49.747636+0000","last_active":"2024-05-08T21:22:50.206671+0000","last_peered":"2024-05-08T21:22:50.206671+0000","last_clean":"2024-05-08T21:22:50.206671+0000","last_became_active":"2024-05-08T21:22:49.747345+0000","last_became_peered":"2024-05-08T21:22:49.747345+0000","last_unstale":"2024-05-08T21:22:50.206671+0000","last_undegraded":"2024-05-08T21:22:50.206671+0000","last_fullsized":"2024-05-08T21:22:50.206671+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:29:03.389411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212991+0000","last_change":"2024-05-08T21:22:50.209316+0000","last_active":"2024-05-08T21:22:50.212991+0000","last_peered":"2024-05-08T21:22:50.212991+0000","last_clean":"2024-05-08T21:22:50.212991+0000","last_became_active":"2024-05-08T21:22:50.209044+0000","last_became_peered":"2024-05-08T21:22:50.209044+0000","last_unstale":"2024-05-08T21:22:50.212991+0000","last_undegraded":"2024-05-08T21:22:50.212991+0000","last_fullsized":"2024-05-08T21:22:50.212991+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:25:52.293163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.211278+0000","last_change":"2024-05-08T21:22:49.744152+0000","last_active":"2024-05-08T21:22:50.211278+0000","last_peered":"2024-05-08T21:22:50.211278+0000","last_clean":"2024-05-08T21:22:50.211278+0000","last_became_active":"2024-05-08T21:22:49.743867+0000","last_became_peered":"2024-05-08T21:22:49.743867+0000","last_unstale":"2024-05-08T21:22:50.211278+0000","last_undegraded":"2024-05-08T21:22:50.211278+0000","last_fullsized":"2024-05-08T21:22:50.211278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:55:00.385106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.209452+0000","last_change":"2024-05-08T21:22:50.204033+0000","last_active":"2024-05-08T21:22:50.209452+0000","last_peered":"2024-05-08T21:22:50.209452+0000","last_clean":"2024-05-08T21:22:50.209452+0000","last_became_active":"2024-05-08T21:22:50.203766+0000","last_became_peered":"2024-05-08T21:22:50.203766+0000","last_unstale":"2024-05-08T21:22:50.209452+0000","last_undegraded":"2024-05-08T21:22:50.209452+0000","last_fullsized":"2024-05-08T21:22:50.209452+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:48:46.138832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.209470+0000","last_change":"2024-05-08T21:22:49.740399+0000","last_active":"2024-05-08T21:22:50.209470+0000","last_peered":"2024-05-08T21:22:50.209470+0000","last_clean":"2024-05-08T21:22:50.209470+0000","last_became_active":"2024-05-08T21:22:49.740125+0000","last_became_peered":"2024-05-08T21:22:49.740125+0000","last_unstale":"2024-05-08T21:22:50.209470+0000","last_undegraded":"2024-05-08T21:22:50.209470+0000","last_fullsized":"2024-05-08T21:22:50.209470+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:01:42.646117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210459+0000","last_change":"2024-05-08T21:22:50.205784+0000","last_active":"2024-05-08T21:22:50.210459+0000","last_peered":"2024-05-08T21:22:50.210459+0000","last_clean":"2024-05-08T21:22:50.210459+0000","last_became_active":"2024-05-08T21:22:50.205525+0000","last_became_peered":"2024-05-08T21:22:50.205525+0000","last_unstale":"2024-05-08T21:22:50.210459+0000","last_undegraded":"2024-05-08T21:22:50.210459+0000","last_fullsized":"2024-05-08T21:22:50.210459+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:23.732223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.207041+0000","last_change":"2024-05-08T21:22:49.739604+0000","last_active":"2024-05-08T21:22:50.207041+0000","last_peered":"2024-05-08T21:22:50.207041+0000","last_clean":"2024-05-08T21:22:50.207041+0000","last_became_active":"2024-05-08T21:22:49.739343+0000","last_became_peered":"2024-05-08T21:22:49.739343+0000","last_unstale":"2024-05-08T21:22:50.207041+0000","last_undegraded":"2024-05-08T21:22:50.207041+0000","last_fullsized":"2024-05-08T21:22:50.207041+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:55:43.121728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212081+0000","last_change":"2024-05-08T21:22:50.206820+0000","last_active":"2024-05-08T21:22:50.212081+0000","last_peered":"2024-05-08T21:22:50.212081+0000","last_clean":"2024-05-08T21:22:50.212081+0000","last_became_active":"2024-05-08T21:22:50.206477+0000","last_became_peered":"2024-05-08T21:22:50.206477+0000","last_unstale":"2024-05-08T21:22:50.212081+0000","last_undegraded":"2024-05-08T21:22:50.212081+0000","last_fullsized":"2024-05-08T21:22:50.212081+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:28.358029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.194728+0000","last_change":"2024-05-08T21:22:49.737447+0000","last_active":"2024-05-08T21:22:50.194728+0000","last_peered":"2024-05-08T21:22:50.194728+0000","last_clean":"2024-05-08T21:22:50.194728+0000","last_became_active":"2024-05-08T21:22:49.736975+0000","last_became_peered":"2024-05-08T21:22:49.736975+0000","last_unstale":"2024-05-08T21:22:50.194728+0000","last_undegraded":"2024-05-08T21:22:50.194728+0000","last_fullsized":"2024-05-08T21:22:50.194728+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:39.192702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210202+0000","last_change":"2024-05-08T21:22:50.205263+0000","last_active":"2024-05-08T21:22:50.210202+0000","last_peered":"2024-05-08T21:22:50.210202+0000","last_clean":"2024-05-08T21:22:50.210202+0000","last_became_active":"2024-05-08T21:22:50.204963+0000","last_became_peered":"2024-05-08T21:22:50.204963+0000","last_unstale":"2024-05-08T21:22:50.210202+0000","last_undegraded":"2024-05-08T21:22:50.210202+0000","last_fullsized":"2024-05-08T21:22:50.210202+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:44:18.055232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.191099+0000","last_change":"2024-05-08T21:22:49.733981+0000","last_active":"2024-05-08T21:22:50.191099+0000","last_peered":"2024-05-08T21:22:50.191099+0000","last_clean":"2024-05-08T21:22:50.191099+0000","last_became_active":"2024-05-08T21:22:49.733714+0000","last_became_peered":"2024-05-08T21:22:49.733714+0000","last_unstale":"2024-05-08T21:22:50.191099+0000","last_undegraded":"2024-05-08T21:22:50.191099+0000","last_fullsized":"2024-05-08T21:22:50.191099+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:47.931452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.211942+0000","last_change":"2024-05-08T21:22:50.209646+0000","last_active":"2024-05-08T21:22:50.211942+0000","last_peered":"2024-05-08T21:22:50.211942+0000","last_clean":"2024-05-08T21:22:50.211942+0000","last_became_active":"2024-05-08T21:22:50.209335+0000","last_became_peered":"2024-05-08T21:22:50.209335+0000","last_unstale":"2024-05-08T21:22:50.211942+0000","last_undegraded":"2024-05-08T21:22:50.211942+0000","last_fullsized":"2024-05-08T21:22:50.211942+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:21:31.919852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.192386+0000","last_change":"2024-05-08T21:22:49.738039+0000","last_active":"2024-05-08T21:22:50.192386+0000","last_peered":"2024-05-08T21:22:50.192386+0000","last_clean":"2024-05-08T21:22:50.192386+0000","last_became_active":"2024-05-08T21:22:49.737785+0000","last_became_peered":"2024-05-08T21:22:49.737785+0000","last_unstale":"2024-05-08T21:22:50.192386+0000","last_undegraded":"2024-05-08T21:22:50.192386+0000","last_fullsized":"2024-05-08T21:22:50.192386+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:27.350557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214175+0000","last_change":"2024-05-08T21:22:50.206695+0000","last_active":"2024-05-08T21:22:50.214175+0000","last_peered":"2024-05-08T21:22:50.214175+0000","last_clean":"2024-05-08T21:22:50.214175+0000","last_became_active":"2024-05-08T21:22:50.206398+0000","last_became_peered":"2024-05-08T21:22:50.206398+0000","last_unstale":"2024-05-08T21:22:50.214175+0000","last_undegraded":"2024-05-08T21:22:50.214175+0000","last_fullsized":"2024-05-08T21:22:50.214175+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:07:17.470994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.188839+0000","last_change":"2024-05-08T21:22:49.740160+0000","last_active":"2024-05-08T21:22:50.188839+0000","last_peered":"2024-05-08T21:22:50.188839+0000","last_clean":"2024-05-08T21:22:50.188839+0000","last_became_active":"2024-05-08T21:22:49.739818+0000","last_became_peered":"2024-05-08T21:22:49.739818+0000","last_unstale":"2024-05-08T21:22:50.188839+0000","last_undegraded":"2024-05-08T21:22:50.188839+0000","last_fullsized":"2024-05-08T21:22:50.188839+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:39:42.856227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213426+0000","last_change":"2024-05-08T21:22:50.204717+0000","last_active":"2024-05-08T21:22:50.213426+0000","last_peered":"2024-05-08T21:22:50.213426+0000","last_clean":"2024-05-08T21:22:50.213426+0000","last_became_active":"2024-05-08T21:22:50.204404+0000","last_became_peered":"2024-05-08T21:22:50.204404+0000","last_unstale":"2024-05-08T21:22:50.213426+0000","last_undegraded":"2024-05-08T21:22:50.213426+0000","last_fullsized":"2024-05-08T21:22:50.213426+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:27:40.074356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.205088+0000","last_change":"2024-05-08T21:22:49.736860+0000","last_active":"2024-05-08T21:22:50.205088+0000","last_peered":"2024-05-08T21:22:50.205088+0000","last_clean":"2024-05-08T21:22:50.205088+0000","last_became_active":"2024-05-08T21:22:49.736497+0000","last_became_peered":"2024-05-08T21:22:49.736497+0000","last_unstale":"2024-05-08T21:22:50.205088+0000","last_undegraded":"2024-05-08T21:22:50.205088+0000","last_fullsized":"2024-05-08T21:22:50.205088+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:15:16.938725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210778+0000","last_change":"2024-05-08T21:22:50.205233+0000","last_active":"2024-05-08T21:22:50.210778+0000","last_peered":"2024-05-08T21:22:50.210778+0000","last_clean":"2024-05-08T21:22:50.210778+0000","last_became_active":"2024-05-08T21:22:50.204889+0000","last_became_peered":"2024-05-08T21:22:50.204889+0000","last_unstale":"2024-05-08T21:22:50.210778+0000","last_undegraded":"2024-05-08T21:22:50.210778+0000","last_fullsized":"2024-05-08T21:22:50.210778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:38:13.012930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.217253+0000","last_change":"2024-05-08T21:22:49.743647+0000","last_active":"2024-05-08T21:22:50.217253+0000","last_peered":"2024-05-08T21:22:50.217253+0000","last_clean":"2024-05-08T21:22:50.217253+0000","last_became_active":"2024-05-08T21:22:49.743348+0000","last_became_peered":"2024-05-08T21:22:49.743348+0000","last_unstale":"2024-05-08T21:22:50.217253+0000","last_undegraded":"2024-05-08T21:22:50.217253+0000","last_fullsized":"2024-05-08T21:22:50.217253+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:06:09.036243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210162+0000","last_change":"2024-05-08T21:22:50.202647+0000","last_active":"2024-05-08T21:22:50.210162+0000","last_peered":"2024-05-08T21:22:50.210162+0000","last_clean":"2024-05-08T21:22:50.210162+0000","last_became_active":"2024-05-08T21:22:50.202144+0000","last_became_peered":"2024-05-08T21:22:50.202144+0000","last_unstale":"2024-05-08T21:22:50.210162+0000","last_undegraded":"2024-05-08T21:22:50.210162+0000","last_fullsized":"2024-05-08T21:22:50.210162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:10:48.376692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212455+0000","last_change":"2024-05-08T21:22:49.735484+0000","last_active":"2024-05-08T21:22:50.212455+0000","last_peered":"2024-05-08T21:22:50.212455+0000","last_clean":"2024-05-08T21:22:50.212455+0000","last_became_active":"2024-05-08T21:22:49.735232+0000","last_became_peered":"2024-05-08T21:22:49.735232+0000","last_unstale":"2024-05-08T21:22:50.212455+0000","last_undegraded":"2024-05-08T21:22:50.212455+0000","last_fullsized":"2024-05-08T21:22:50.212455+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:13:03.055943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212615+0000","last_change":"2024-05-08T21:22:50.204754+0000","last_active":"2024-05-08T21:22:50.212615+0000","last_peered":"2024-05-08T21:22:50.212615+0000","last_clean":"2024-05-08T21:22:50.212615+0000","last_became_active":"2024-05-08T21:22:50.204473+0000","last_became_peered":"2024-05-08T21:22:50.204473+0000","last_unstale":"2024-05-08T21:22:50.212615+0000","last_undegraded":"2024-05-08T21:22:50.212615+0000","last_fullsized":"2024-05-08T21:22:50.212615+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:14.396616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.199440+0000","last_change":"2024-05-08T21:22:49.741850+0000","last_active":"2024-05-08T21:22:50.199440+0000","last_peered":"2024-05-08T21:22:50.199440+0000","last_clean":"2024-05-08T21:22:50.199440+0000","last_became_active":"2024-05-08T21:22:49.741527+0000","last_became_peered":"2024-05-08T21:22:49.741527+0000","last_unstale":"2024-05-08T21:22:50.199440+0000","last_undegraded":"2024-05-08T21:22:50.199440+0000","last_fullsized":"2024-05-08T21:22:50.199440+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:29:03.389411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213628+0000","last_change":"2024-05-08T21:22:50.211613+0000","last_active":"2024-05-08T21:22:50.213628+0000","last_peered":"2024-05-08T21:22:50.213628+0000","last_clean":"2024-05-08T21:22:50.213628+0000","last_became_active":"2024-05-08T21:22:50.211335+0000","last_became_peered":"2024-05-08T21:22:50.211335+0000","last_unstale":"2024-05-08T21:22:50.213628+0000","last_undegraded":"2024-05-08T21:22:50.213628+0000","last_fullsized":"2024-05-08T21:22:50.213628+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:43:00.051225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.194827+0000","last_change":"2024-05-08T21:22:49.737257+0000","last_active":"2024-05-08T21:22:50.194827+0000","last_peered":"2024-05-08T21:22:50.194827+0000","last_clean":"2024-05-08T21:22:50.194827+0000","last_became_active":"2024-05-08T21:22:49.736591+0000","last_became_peered":"2024-05-08T21:22:49.736591+0000","last_unstale":"2024-05-08T21:22:50.194827+0000","last_undegraded":"2024-05-08T21:22:50.194827+0000","last_fullsized":"2024-05-08T21:22:50.194827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:08.158087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214092+0000","last_change":"2024-05-08T21:22:50.209206+0000","last_active":"2024-05-08T21:22:50.214092+0000","last_peered":"2024-05-08T21:22:50.214092+0000","last_clean":"2024-05-08T21:22:50.214092+0000","last_became_active":"2024-05-08T21:22:50.208950+0000","last_became_peered":"2024-05-08T21:22:50.208950+0000","last_unstale":"2024-05-08T21:22:50.214092+0000","last_undegraded":"2024-05-08T21:22:50.214092+0000","last_fullsized":"2024-05-08T21:22:50.214092+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:15:17.946198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.215733+0000","last_change":"2024-05-08T21:22:49.743861+0000","last_active":"2024-05-08T21:22:50.215733+0000","last_peered":"2024-05-08T21:22:50.215733+0000","last_clean":"2024-05-08T21:22:50.215733+0000","last_became_active":"2024-05-08T21:22:49.743575+0000","last_became_peered":"2024-05-08T21:22:49.743575+0000","last_unstale":"2024-05-08T21:22:50.215733+0000","last_undegraded":"2024-05-08T21:22:50.215733+0000","last_fullsized":"2024-05-08T21:22:50.215733+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:51.977551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.207908+0000","last_change":"2024-05-08T21:22:50.203855+0000","last_active":"2024-05-08T21:22:50.207908+0000","last_peered":"2024-05-08T21:22:50.207908+0000","last_clean":"2024-05-08T21:22:50.207908+0000","last_became_active":"2024-05-08T21:22:50.203557+0000","last_became_peered":"2024-05-08T21:22:50.203557+0000","last_unstale":"2024-05-08T21:22:50.207908+0000","last_undegraded":"2024-05-08T21:22:50.207908+0000","last_fullsized":"2024-05-08T21:22:50.207908+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:09.165560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213662+0000","last_change":"2024-05-08T21:22:49.742208+0000","last_active":"2024-05-08T21:22:50.213662+0000","last_peered":"2024-05-08T21:22:50.213662+0000","last_clean":"2024-05-08T21:22:50.213662+0000","last_became_active":"2024-05-08T21:22:49.741894+0000","last_became_peered":"2024-05-08T21:22:49.741894+0000","last_unstale":"2024-05-08T21:22:50.213662+0000","last_undegraded":"2024-05-08T21:22:50.213662+0000","last_fullsized":"2024-05-08T21:22:50.213662+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:59:36.798032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210434+0000","last_change":"2024-05-08T21:22:50.208102+0000","last_active":"2024-05-08T21:22:50.210434+0000","last_peered":"2024-05-08T21:22:50.210434+0000","last_clean":"2024-05-08T21:22:50.210434+0000","last_became_active":"2024-05-08T21:22:50.207765+0000","last_became_peered":"2024-05-08T21:22:50.207765+0000","last_unstale":"2024-05-08T21:22:50.210434+0000","last_undegraded":"2024-05-08T21:22:50.210434+0000","last_fullsized":"2024-05-08T21:22:50.210434+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:14.396616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212404+0000","last_change":"2024-05-08T21:22:49.734619+0000","last_active":"2024-05-08T21:22:50.212404+0000","last_peered":"2024-05-08T21:22:50.212404+0000","last_clean":"2024-05-08T21:22:50.212404+0000","last_became_active":"2024-05-08T21:22:49.734354+0000","last_became_peered":"2024-05-08T21:22:49.734354+0000","last_unstale":"2024-05-08T21:22:50.212404+0000","last_undegraded":"2024-05-08T21:22:50.212404+0000","last_fullsized":"2024-05-08T21:22:50.212404+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:23:35.220950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.213811+0000","last_change":"2024-05-08T21:22:50.210640+0000","last_active":"2024-05-08T21:22:50.213811+0000","last_peered":"2024-05-08T21:22:50.213811+0000","last_clean":"2024-05-08T21:22:50.213811+0000","last_became_active":"2024-05-08T21:22:50.210283+0000","last_became_peered":"2024-05-08T21:22:50.210283+0000","last_unstale":"2024-05-08T21:22:50.213811+0000","last_undegraded":"2024-05-08T21:22:50.213811+0000","last_fullsized":"2024-05-08T21:22:50.213811+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:58:46.704338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.215511+0000","last_change":"2024-05-08T21:22:49.737099+0000","last_active":"2024-05-08T21:22:50.215511+0000","last_peered":"2024-05-08T21:22:50.215511+0000","last_clean":"2024-05-08T21:22:50.215511+0000","last_became_active":"2024-05-08T21:22:49.736839+0000","last_became_peered":"2024-05-08T21:22:49.736839+0000","last_unstale":"2024-05-08T21:22:50.215511+0000","last_undegraded":"2024-05-08T21:22:50.215511+0000","last_fullsized":"2024-05-08T21:22:50.215511+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:05:26.886351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.215303+0000","last_change":"2024-05-08T21:22:50.206786+0000","last_active":"2024-05-08T21:22:50.215303+0000","last_peered":"2024-05-08T21:22:50.215303+0000","last_clean":"2024-05-08T21:22:50.215303+0000","last_became_active":"2024-05-08T21:22:50.206484+0000","last_became_peered":"2024-05-08T21:22:50.206484+0000","last_unstale":"2024-05-08T21:22:50.215303+0000","last_undegraded":"2024-05-08T21:22:50.215303+0000","last_fullsized":"2024-05-08T21:22:50.215303+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:36:44.630301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.191296+0000","last_change":"2024-05-08T21:22:49.740491+0000","last_active":"2024-05-08T21:22:50.191296+0000","last_peered":"2024-05-08T21:22:50.191296+0000","last_clean":"2024-05-08T21:22:50.191296+0000","last_became_active":"2024-05-08T21:22:49.740166+0000","last_became_peered":"2024-05-08T21:22:49.740166+0000","last_unstale":"2024-05-08T21:22:50.191296+0000","last_undegraded":"2024-05-08T21:22:50.191296+0000","last_fullsized":"2024-05-08T21:22:50.191296+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:27:16.794172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.216049+0000","last_change":"2024-05-08T21:22:50.207827+0000","last_active":"2024-05-08T21:22:50.216049+0000","last_peered":"2024-05-08T21:22:50.216049+0000","last_clean":"2024-05-08T21:22:50.216049+0000","last_became_active":"2024-05-08T21:22:50.207560+0000","last_became_peered":"2024-05-08T21:22:50.207560+0000","last_unstale":"2024-05-08T21:22:50.216049+0000","last_undegraded":"2024-05-08T21:22:50.216049+0000","last_fullsized":"2024-05-08T21:22:50.216049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:29:04.396884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.202910+0000","last_change":"2024-05-08T21:22:49.743647+0000","last_active":"2024-05-08T21:22:50.202910+0000","last_peered":"2024-05-08T21:22:50.202910+0000","last_clean":"2024-05-08T21:22:50.202910+0000","last_became_active":"2024-05-08T21:22:49.743349+0000","last_became_peered":"2024-05-08T21:22:49.743349+0000","last_unstale":"2024-05-08T21:22:50.202910+0000","last_undegraded":"2024-05-08T21:22:50.202910+0000","last_fullsized":"2024-05-08T21:22:50.202910+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:53:44.252529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.215372+0000","last_change":"2024-05-08T21:22:50.205819+0000","last_active":"2024-05-08T21:22:50.215372+0000","last_peered":"2024-05-08T21:22:50.215372+0000","last_clean":"2024-05-08T21:22:50.215372+0000","last_became_active":"2024-05-08T21:22:50.205239+0000","last_became_peered":"2024-05-08T21:22:50.205239+0000","last_unstale":"2024-05-08T21:22:50.215372+0000","last_undegraded":"2024-05-08T21:22:50.215372+0000","last_fullsized":"2024-05-08T21:22:50.215372+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:52.985024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.200371+0000","last_change":"2024-05-08T21:22:49.743818+0000","last_active":"2024-05-08T21:22:50.200371+0000","last_peered":"2024-05-08T21:22:50.200371+0000","last_clean":"2024-05-08T21:22:50.200371+0000","last_became_active":"2024-05-08T21:22:49.743469+0000","last_became_peered":"2024-05-08T21:22:49.743469+0000","last_unstale":"2024-05-08T21:22:50.200371+0000","last_undegraded":"2024-05-08T21:22:50.200371+0000","last_fullsized":"2024-05-08T21:22:50.200371+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:06:42.743465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.215141+0000","last_change":"2024-05-08T21:22:50.203236+0000","last_active":"2024-05-08T21:22:50.215141+0000","last_peered":"2024-05-08T21:22:50.215141+0000","last_clean":"2024-05-08T21:22:50.215141+0000","last_became_active":"2024-05-08T21:22:50.201914+0000","last_became_peered":"2024-05-08T21:22:50.201914+0000","last_unstale":"2024-05-08T21:22:50.215141+0000","last_undegraded":"2024-05-08T21:22:50.215141+0000","last_fullsized":"2024-05-08T21:22:50.215141+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:06:10.043716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.191299+0000","last_change":"2024-05-08T21:22:49.739798+0000","last_active":"2024-05-08T21:22:50.191299+0000","last_peered":"2024-05-08T21:22:50.191299+0000","last_clean":"2024-05-08T21:22:50.191299+0000","last_became_active":"2024-05-08T21:22:49.739520+0000","last_became_peered":"2024-05-08T21:22:49.739520+0000","last_unstale":"2024-05-08T21:22:50.191299+0000","last_undegraded":"2024-05-08T21:22:50.191299+0000","last_fullsized":"2024-05-08T21:22:50.191299+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:47:50.170209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.209189+0000","last_change":"2024-05-08T21:22:50.204561+0000","last_active":"2024-05-08T21:22:50.209189+0000","last_peered":"2024-05-08T21:22:50.209189+0000","last_clean":"2024-05-08T21:22:50.209189+0000","last_became_active":"2024-05-08T21:22:50.204296+0000","last_became_peered":"2024-05-08T21:22:50.204296+0000","last_unstale":"2024-05-08T21:22:50.209189+0000","last_undegraded":"2024-05-08T21:22:50.209189+0000","last_fullsized":"2024-05-08T21:22:50.209189+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:25:52.293163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.195794+0000","last_change":"2024-05-08T21:22:49.737100+0000","last_active":"2024-05-08T21:22:50.195794+0000","last_peered":"2024-05-08T21:22:50.195794+0000","last_clean":"2024-05-08T21:22:50.195794+0000","last_became_active":"2024-05-08T21:22:49.736715+0000","last_became_peered":"2024-05-08T21:22:49.736715+0000","last_unstale":"2024-05-08T21:22:50.195794+0000","last_undegraded":"2024-05-08T21:22:50.195794+0000","last_fullsized":"2024-05-08T21:22:50.195794+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:52:46.503073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.215282+0000","last_change":"2024-05-08T21:22:50.206910+0000","last_active":"2024-05-08T21:22:50.215282+0000","last_peered":"2024-05-08T21:22:50.215282+0000","last_clean":"2024-05-08T21:22:50.215282+0000","last_became_active":"2024-05-08T21:22:50.206623+0000","last_became_peered":"2024-05-08T21:22:50.206623+0000","last_unstale":"2024-05-08T21:22:50.215282+0000","last_undegraded":"2024-05-08T21:22:50.215282+0000","last_fullsized":"2024-05-08T21:22:50.215282+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:24:25.163652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.211304+0000","last_change":"2024-05-08T21:22:49.743344+0000","last_active":"2024-05-08T21:22:50.211304+0000","last_peered":"2024-05-08T21:22:50.211304+0000","last_clean":"2024-05-08T21:22:50.211304+0000","last_became_active":"2024-05-08T21:22:49.742413+0000","last_became_peered":"2024-05-08T21:22:49.742413+0000","last_unstale":"2024-05-08T21:22:50.211304+0000","last_undegraded":"2024-05-08T21:22:50.211304+0000","last_fullsized":"2024-05-08T21:22:50.211304+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:06:42.743465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.210416+0000","last_change":"2024-05-08T21:22:50.205911+0000","last_active":"2024-05-08T21:22:50.210416+0000","last_peered":"2024-05-08T21:22:50.210416+0000","last_clean":"2024-05-08T21:22:50.210416+0000","last_became_active":"2024-05-08T21:22:50.205609+0000","last_became_peered":"2024-05-08T21:22:50.205609+0000","last_unstale":"2024-05-08T21:22:50.210416+0000","last_undegraded":"2024-05-08T21:22:50.210416+0000","last_fullsized":"2024-05-08T21:22:50.210416+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:41:28.819967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.188953+0000","last_change":"2024-05-08T21:22:49.745047+0000","last_active":"2024-05-08T21:22:50.188953+0000","last_peered":"2024-05-08T21:22:50.188953+0000","last_clean":"2024-05-08T21:22:50.188953+0000","last_became_active":"2024-05-08T21:22:49.744747+0000","last_became_peered":"2024-05-08T21:22:49.744747+0000","last_unstale":"2024-05-08T21:22:50.188953+0000","last_undegraded":"2024-05-08T21:22:50.188953+0000","last_fullsized":"2024-05-08T21:22:50.188953+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:05:26.886351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.207987+0000","last_change":"2024-05-08T21:22:50.203176+0000","last_active":"2024-05-08T21:22:50.207987+0000","last_peered":"2024-05-08T21:22:50.207987+0000","last_clean":"2024-05-08T21:22:50.207987+0000","last_became_active":"2024-05-08T21:22:50.202886+0000","last_became_peered":"2024-05-08T21:22:50.202886+0000","last_unstale":"2024-05-08T21:22:50.207987+0000","last_undegraded":"2024-05-08T21:22:50.207987+0000","last_fullsized":"2024-05-08T21:22:50.207987+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:40.200175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.199735+0000","last_change":"2024-05-08T21:22:49.742028+0000","last_active":"2024-05-08T21:22:50.199735+0000","last_peered":"2024-05-08T21:22:50.199735+0000","last_clean":"2024-05-08T21:22:50.199735+0000","last_became_active":"2024-05-08T21:22:49.741706+0000","last_became_peered":"2024-05-08T21:22:49.741706+0000","last_unstale":"2024-05-08T21:22:50.199735+0000","last_undegraded":"2024-05-08T21:22:50.199735+0000","last_fullsized":"2024-05-08T21:22:50.199735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:06:09.036243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.214167+0000","last_change":"2024-05-08T21:22:50.210183+0000","last_active":"2024-05-08T21:22:50.214167+0000","last_peered":"2024-05-08T21:22:50.214167+0000","last_clean":"2024-05-08T21:22:50.214167+0000","last_became_active":"2024-05-08T21:22:50.209860+0000","last_became_peered":"2024-05-08T21:22:50.209860+0000","last_unstale":"2024-05-08T21:22:50.214167+0000","last_undegraded":"2024-05-08T21:22:50.214167+0000","last_fullsized":"2024-05-08T21:22:50.214167+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:28.358029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.188638+0000","last_change":"2024-05-08T21:22:49.740650+0000","last_active":"2024-05-08T21:22:50.188638+0000","last_peered":"2024-05-08T21:22:50.188638+0000","last_clean":"2024-05-08T21:22:50.188638+0000","last_became_active":"2024-05-08T21:22:49.740373+0000","last_became_peered":"2024-05-08T21:22:49.740373+0000","last_unstale":"2024-05-08T21:22:50.188638+0000","last_undegraded":"2024-05-08T21:22:50.188638+0000","last_fullsized":"2024-05-08T21:22:50.188638+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:36:43.622828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212952+0000","last_change":"2024-05-08T21:22:50.208073+0000","last_active":"2024-05-08T21:22:50.212952+0000","last_peered":"2024-05-08T21:22:50.212952+0000","last_clean":"2024-05-08T21:22:50.212952+0000","last_became_active":"2024-05-08T21:22:50.207506+0000","last_became_peered":"2024-05-08T21:22:50.207506+0000","last_unstale":"2024-05-08T21:22:50.212952+0000","last_undegraded":"2024-05-08T21:22:50.212952+0000","last_fullsized":"2024-05-08T21:22:50.212952+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:15:17.946198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.199222+0000","last_change":"2024-05-08T21:22:49.742392+0000","last_active":"2024-05-08T21:22:50.199222+0000","last_peered":"2024-05-08T21:22:50.199222+0000","last_clean":"2024-05-08T21:22:50.199222+0000","last_became_active":"2024-05-08T21:22:49.742093+0000","last_became_peered":"2024-05-08T21:22:49.742093+0000","last_unstale":"2024-05-08T21:22:50.199222+0000","last_undegraded":"2024-05-08T21:22:50.199222+0000","last_fullsized":"2024-05-08T21:22:50.199222+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:51.977551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.216104+0000","last_change":"2024-05-08T21:22:50.214641+0000","last_active":"2024-05-08T21:22:50.216104+0000","last_peered":"2024-05-08T21:22:50.216104+0000","last_clean":"2024-05-08T21:22:50.216104+0000","last_became_active":"2024-05-08T21:22:50.213619+0000","last_became_peered":"2024-05-08T21:22:50.213619+0000","last_unstale":"2024-05-08T21:22:50.216104+0000","last_undegraded":"2024-05-08T21:22:50.216104+0000","last_fullsized":"2024-05-08T21:22:50.216104+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:21:31.919852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.191412+0000","last_change":"2024-05-08T21:22:49.738120+0000","last_active":"2024-05-08T21:22:50.191412+0000","last_peered":"2024-05-08T21:22:50.191412+0000","last_clean":"2024-05-08T21:22:50.191412+0000","last_became_active":"2024-05-08T21:22:49.737805+0000","last_became_peered":"2024-05-08T21:22:49.737805+0000","last_unstale":"2024-05-08T21:22:50.191412+0000","last_undegraded":"2024-05-08T21:22:50.191412+0000","last_fullsized":"2024-05-08T21:22:50.191412+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:55:00.385106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.212011+0000","last_change":"2024-05-08T21:22:50.205043+0000","last_active":"2024-05-08T21:22:50.212011+0000","last_peered":"2024-05-08T21:22:50.212011+0000","last_clean":"2024-05-08T21:22:50.212011+0000","last_became_active":"2024-05-08T21:22:50.204630+0000","last_became_peered":"2024-05-08T21:22:50.204630+0000","last_unstale":"2024-05-08T21:22:50.212011+0000","last_undegraded":"2024-05-08T21:22:50.212011+0000","last_fullsized":"2024-05-08T21:22:50.212011+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:26:57.146675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.205320+0000","last_change":"2024-05-08T21:22:49.736783+0000","last_active":"2024-05-08T21:22:50.205320+0000","last_peered":"2024-05-08T21:22:50.205320+0000","last_clean":"2024-05-08T21:22:50.205320+0000","last_became_active":"2024-05-08T21:22:49.736479+0000","last_became_peered":"2024-05-08T21:22:49.736479+0000","last_unstale":"2024-05-08T21:22:50.205320+0000","last_undegraded":"2024-05-08T21:22:50.205320+0000","last_fullsized":"2024-05-08T21:22:50.205320+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:24:24.156179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.206121+0000","last_change":"2024-05-08T21:22:50.202594+0000","last_active":"2024-05-08T21:22:50.206121+0000","last_peered":"2024-05-08T21:22:50.206121+0000","last_clean":"2024-05-08T21:22:50.206121+0000","last_became_active":"2024-05-08T21:22:50.202278+0000","last_became_peered":"2024-05-08T21:22:50.202278+0000","last_unstale":"2024-05-08T21:22:50.206121+0000","last_undegraded":"2024-05-08T21:22:50.206121+0000","last_fullsized":"2024-05-08T21:22:50.206121+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:44:18.055232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-08T21:22:50.198296+0000","last_change":"2024-05-08T21:22:49.739311+0000","last_active":"2024-05-08T21:22:50.198296+0000","last_peered":"2024-05-08T21:22:50.198296+0000","last_clean":"2024-05-08T21:22:50.198296+0000","last_became_active":"2024-05-08T21:22:49.739006+0000","last_became_peered":"2024-05-08T21:22:49.739006+0000","last_unstale":"2024-05-08T21:22:50.198296+0000","last_undegraded":"2024-05-08T21:22:50.198296+0000","last_fullsized":"2024-05-08T21:22:50.198296+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:39:42.856227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":176,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-08T21:22:49.723088+0000","last_change":"2024-05-08T21:21:42.333097+0000","last_active":"2024-05-08T21:22:49.723088+0000","last_peered":"2024-05-08T21:22:49.723088+0000","last_clean":"2024-05-08T21:22:49.723088+0000","last_became_active":"2024-05-08T21:21:42.332033+0000","last_became_peered":"2024-05-08T21:21:42.332033+0000","last_unstale":"2024-05-08T21:22:49.723088+0000","last_undegraded":"2024-05-08T21:22:49.723088+0000","last_fullsized":"2024-05-08T21:22:49.723088+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_clean_scrub_stamp":"2024-05-08T21:21:40.862622+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:39:24.772551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":970752,"data_stored":736053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.55800000000000005}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.627},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.44700000000000001}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.59099999999999997}]},{"osd":3,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.499}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.42899999999999999}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.39700000000000002}]},{"osd":6,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.52100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.53000000000000003}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.41099999999999998}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.375},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.437}]},{"osd":3,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":1.716}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.497},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.48699999999999999}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.57399999999999995}]},{"osd":7,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.46999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":147181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.26700000000000002}]},{"osd":2,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.33100000000000002}]},{"osd":3,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.432}]},{"osd":4,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.45100000000000001}]},{"osd":5,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.54900000000000004}]},{"osd":6,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.49399999999999999}]},{"osd":7,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.63}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":966656,"data_stored":735963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.29899999999999999}]},{"osd":2,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.34899999999999998}]},{"osd":3,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.318}]},{"osd":4,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.63400000000000001}]},{"osd":5,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.56299999999999994}]},{"osd":6,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.61199999999999999}]},{"osd":7,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.57299999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":376832,"data_stored":145665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.435},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.39000000000000001}]},{"osd":1,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32300000000000001}]},{"osd":3,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.378},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.47399999999999998}]},{"osd":4,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.65400000000000003}]},{"osd":5,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.66700000000000004}]},{"osd":6,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.622},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.69499999999999995}]},{"osd":7,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.79200000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":138820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":2.7919999999999998,"5min":2.7919999999999998,"15min":2.7919999999999998},"last":0.68700000000000006}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":4.2370000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.435}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":2.9369999999999998,"5min":2.9369999999999998,"15min":2.9369999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":4.2969999999999997}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.8380000000000001,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":2.7490000000000001}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.9060000000000001,"5min":2.9060000000000001,"15min":2.9060000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":5.1699999999999999}]},{"osd":6,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":2.8730000000000002,"5min":2.8730000000000002,"15min":2.8730000000000002},"last":0.55100000000000005}]},{"osd":7,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":5.2030000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":1.1399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":138820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.51100000000000001}]},{"osd":1,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.47099999999999997}]},{"osd":2,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.48699999999999999}]},{"osd":5,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.63}]},{"osd":6,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60099999999999998}]},{"osd":7,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.65400000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":138820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.33500000000000002}]},{"osd":1,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.52500000000000002}]},{"osd":2,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.55400000000000005}]},{"osd":3,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.441}]},{"osd":4,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.42499999999999999}]},{"osd":6,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.47899999999999998}]},{"osd":7,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.49199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T21:22:54.865 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-08T21:22:54.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-08T21:22:55.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.045+0000 7f6144b44700 1 Processor -- start 2024-05-08T21:22:55.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.045+0000 7f6144b44700 1 -- start start 2024-05-08T21:22:55.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.046+0000 7f6144b44700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61400b0d80 0x7f61400b3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:55.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.046+0000 7f6144b44700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6140145040 con 0x7f614006b070 2024-05-08T21:22:55.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.046+0000 7f6144b44700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61400b38c0 con 0x7f61400b4d50 2024-05-08T21:22:55.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.046+0000 7f6144b44700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61400b16e0 con 0x7f61400b0d80 2024-05-08T21:22:55.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.046+0000 7f613ed9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f614006b070 0x7f61401454d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59792/0 (socket says 172.21.15.3:59792) 2024-05-08T21:22:55.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.046+0000 7f613ed9d700 1 -- 172.21.15.3:0/916117332 learned_addr learned my addr 172.21.15.3:0/916117332 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:55.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.046+0000 7f613e59c700 1 --2- 172.21.15.3:0/916117332 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61400b0d80 0x7f61400b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:55.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.046+0000 7f613d59a700 1 -- 172.21.15.3:0/916117332 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120082404 0 0) 0x7f6140145040 con 0x7f614006b070 2024-05-08T21:22:55.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.046+0000 7f613d59a700 1 -- 172.21.15.3:0/916117332 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61280035b0 con 0x7f614006b070 2024-05-08T21:22:55.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.047+0000 7f613d59a700 1 -- 172.21.15.3:0/916117332 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3661728614 0 0) 0x7f61400b38c0 con 0x7f61400b4d50 2024-05-08T21:22:55.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.047+0000 7f613e59c700 1 -- 172.21.15.3:0/916117332 >> v1:172.21.15.186:6789/0 conn(0x7f61400b4d50 legacy=0x7f614006a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:55.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.047+0000 7f613e59c700 1 -- 172.21.15.3:0/916117332 >> v1:172.21.15.3:6789/0 conn(0x7f614006b070 legacy=0x7f61401454d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:55.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.047+0000 7f613e59c700 1 -- 172.21.15.3:0/916117332 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61400b19c0 con 0x7f61400b0d80 2024-05-08T21:22:55.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.047+0000 7f613e59c700 1 --2- 172.21.15.3:0/916117332 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61400b0d80 0x7f61400b3380 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6130009870 tx=0x7f6130009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73debd6d79538940 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.047+0000 7f613d59a700 1 -- 172.21.15.3:0/916117332 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130011830 con 0x7f61400b0d80 2024-05-08T21:22:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.047+0000 7f613d59a700 1 -- 172.21.15.3:0/916117332 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6130011990 con 0x7f61400b0d80 2024-05-08T21:22:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.047+0000 7f613d59a700 1 -- 172.21.15.3:0/916117332 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130011d20 con 0x7f61400b0d80 2024-05-08T21:22:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.048+0000 7f6144b44700 1 -- 172.21.15.3:0/916117332 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61400b0d80 msgr2=0x7f61400b3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.048+0000 7f6144b44700 1 --2- 172.21.15.3:0/916117332 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61400b0d80 0x7f61400b3380 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6130009870 tx=0x7f6130009b80 comp rx=0 tx=0).stop 2024-05-08T21:22:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.048+0000 7f6144b44700 1 -- 172.21.15.3:0/916117332 shutdown_connections 2024-05-08T21:22:55.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.048+0000 7f6144b44700 1 --2- 172.21.15.3:0/916117332 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61400b0d80 0x7f61400b3380 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:55.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.048+0000 7f6144b44700 1 -- 172.21.15.3:0/916117332 >> 172.21.15.3:0/916117332 conn(0x7f6140099a50 msgr2=0x7f61400b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:55.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.048+0000 7f6144b44700 1 -- 172.21.15.3:0/916117332 shutdown_connections 2024-05-08T21:22:55.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.048+0000 7f6144b44700 1 -- 172.21.15.3:0/916117332 wait complete. 2024-05-08T21:22:55.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.050+0000 7f6144b44700 1 Processor -- start 2024-05-08T21:22:55.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.050+0000 7f6144b44700 1 -- start start 2024-05-08T21:22:55.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f6144b44700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614006b070 0x7f61400af1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:55.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f6144b44700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61400b0d80 0x7f61401f33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:55.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f6144b44700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61400b4d50 0x7f61400adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:55.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f6144b44700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61400ae7e0 con 0x7f614006b070 2024-05-08T21:22:55.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f6144b44700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614013ea30 con 0x7f61400b0d80 2024-05-08T21:22:55.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f6144b44700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614013ed10 con 0x7f61400b4d50 2024-05-08T21:22:55.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f613e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614006b070 0x7f61400af1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:55.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f613e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614006b070 0x7f61400af1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45540/0 (socket says 172.21.15.3:45540) 2024-05-08T21:22:55.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f613e59c700 1 -- 172.21.15.3:0/2702094927 learned_addr learned my addr 172.21.15.3:0/2702094927 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:55.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f613dd9b700 1 --2- 172.21.15.3:0/2702094927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61400b0d80 0x7f61401f33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:55.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f613ed9d700 1 --2- 172.21.15.3:0/2702094927 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61400b4d50 0x7f61400adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:55.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f613e59c700 1 -- 172.21.15.3:0/2702094927 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61400b4d50 msgr2=0x7f61400adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:55.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f613e59c700 1 --2- 172.21.15.3:0/2702094927 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61400b4d50 0x7f61400adc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:55.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f613e59c700 1 -- 172.21.15.3:0/2702094927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61400b0d80 msgr2=0x7f61401f33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:55.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f613e59c700 1 --2- 172.21.15.3:0/2702094927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61400b0d80 0x7f61401f33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:55.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f613e59c700 1 -- 172.21.15.3:0/2702094927 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61300094a0 con 0x7f614006b070 2024-05-08T21:22:55.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f613dd9b700 1 --2- 172.21.15.3:0/2702094927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61400b0d80 0x7f61401f33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:55.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.051+0000 7f613e59c700 1 --2- 172.21.15.3:0/2702094927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614006b070 0x7f61400af1e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f61300090b0 tx=0x7f6130004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:55.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.052+0000 7f612effd700 1 -- 172.21.15.3:0/2702094927 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613001a910 con 0x7f614006b070 2024-05-08T21:22:55.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.052+0000 7f612effd700 1 -- 172.21.15.3:0/2702094927 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613001aa70 con 0x7f614006b070 2024-05-08T21:22:55.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.052+0000 7f612effd700 1 -- 172.21.15.3:0/2702094927 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613001ade0 con 0x7f614006b070 2024-05-08T21:22:55.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.052+0000 7f6144b44700 1 -- 172.21.15.3:0/2702094927 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f614013eff0 con 0x7f614006b070 2024-05-08T21:22:55.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.052+0000 7f6144b44700 1 -- 172.21.15.3:0/2702094927 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61400afdd0 con 0x7f614006b070 2024-05-08T21:22:55.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.053+0000 7f6144b44700 1 -- 172.21.15.3:0/2702094927 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614013f510 con 0x7f614006b070 2024-05-08T21:22:55.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.053+0000 7f612effd700 1 -- 172.21.15.3:0/2702094927 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6130022070 con 0x7f614006b070 2024-05-08T21:22:55.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.054+0000 7f612effd700 1 --2- 172.21.15.3:0/2702094927 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f612006ac90 0x7f612006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:55.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.054+0000 7f612effd700 1 -- 172.21.15.3:0/2702094927 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f613008a130 con 0x7f614006b070 2024-05-08T21:22:55.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.054+0000 7f613dd9b700 1 --2- 172.21.15.3:0/2702094927 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f612006ac90 0x7f612006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:55.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.055+0000 7f613dd9b700 1 --2- 172.21.15.3:0/2702094927 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f612006ac90 0x7f612006d150 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f61400b1430 tx=0x7f6134009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:55.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.058+0000 7f612effd700 1 -- 172.21.15.3:0/2702094927 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f61300554d0 con 0x7f614006b070 2024-05-08T21:22:55.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.370+0000 7f6144b44700 1 -- 172.21.15.3:0/2702094927 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f614006aa80 con 0x7f614006b070 2024-05-08T21:22:55.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.374+0000 7f612effd700 1 -- 172.21.15.3:0/2702094927 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+0 (secure 0 0 0) 0x7f6130058af0 con 0x7f614006b070 2024-05-08T21:22:55.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.376+0000 7f6144b44700 1 -- 172.21.15.3:0/2702094927 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f612006ac90 msgr2=0x7f612006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.376+0000 7f6144b44700 1 --2- 172.21.15.3:0/2702094927 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f612006ac90 0x7f612006d150 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f61400b1430 tx=0x7f6134009040 comp rx=0 tx=0).stop 2024-05-08T21:22:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.376+0000 7f6144b44700 1 -- 172.21.15.3:0/2702094927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614006b070 msgr2=0x7f61400af1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.376+0000 7f6144b44700 1 --2- 172.21.15.3:0/2702094927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614006b070 0x7f61400af1e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f61300090b0 tx=0x7f6130004750 comp rx=0 tx=0).stop 2024-05-08T21:22:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.377+0000 7f6144b44700 1 -- 172.21.15.3:0/2702094927 shutdown_connections 2024-05-08T21:22:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.377+0000 7f6144b44700 1 --2- 172.21.15.3:0/2702094927 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61400b4d50 0x7f61400adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.377+0000 7f6144b44700 1 --2- 172.21.15.3:0/2702094927 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f612006ac90 0x7f612006d150 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.377+0000 7f6144b44700 1 --2- 172.21.15.3:0/2702094927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614006b070 0x7f61400af1e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.377+0000 7f6144b44700 1 --2- 172.21.15.3:0/2702094927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61400b0d80 0x7f61401f33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.377+0000 7f6144b44700 1 -- 172.21.15.3:0/2702094927 >> 172.21.15.3:0/2702094927 conn(0x7f6140099a50 msgr2=0x7f61400b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.377+0000 7f6144b44700 1 -- 172.21.15.3:0/2702094927 shutdown_connections 2024-05-08T21:22:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.377+0000 7f6144b44700 1 -- 172.21.15.3:0/2702094927 wait complete. 2024-05-08T21:22:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.0 2024-05-08T21:22:55.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:22:55.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.526+0000 7f07072fe700 1 Processor -- start 2024-05-08T21:22:55.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.527+0000 7f07072fe700 1 -- start start 2024-05-08T21:22:55.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.528+0000 7f07072fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f07000cd4e0 0x7f0700099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:55.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.528+0000 7f07072fe700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070009aca0 con 0x7f0700099a50 2024-05-08T21:22:55.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.528+0000 7f07072fe700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07000cd9e0 con 0x7f070015b810 2024-05-08T21:22:55.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.528+0000 7f07072fe700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f070009a440 con 0x7f07000cd4e0 2024-05-08T21:22:55.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.528+0000 7f070589b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f07000cd4e0 0x7f0700099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:55.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.528+0000 7f0704899700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0700099a50 0x7f07000ccdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59808/0 (socket says 172.21.15.3:59808) 2024-05-08T21:22:55.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.528+0000 7f0704899700 1 -- 172.21.15.3:0/1073202974 learned_addr learned my addr 172.21.15.3:0/1073202974 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:55.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.528+0000 7f070589b700 1 -- 172.21.15.3:0/1073202974 >> v1:172.21.15.186:6789/0 conn(0x7f070015b810 legacy=0x7f07000cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:55.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.528+0000 7f070589b700 1 -- 172.21.15.3:0/1073202974 >> v1:172.21.15.3:6789/0 conn(0x7f0700099a50 legacy=0x7f07000ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:55.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.528+0000 7f070589b700 1 -- 172.21.15.3:0/1073202974 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07000cf220 con 0x7f07000cd4e0 2024-05-08T21:22:55.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.528+0000 7f070589b700 1 --2- 172.21.15.3:0/1073202974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07000cd4e0 0x7f0700099f00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f06f8009d90 tx=0x7f06f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7000c6cb45f55475 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:55.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.529+0000 7f06f7fff700 1 -- 172.21.15.3:0/1073202974 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f800e070 con 0x7f07000cd4e0 2024-05-08T21:22:55.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.529+0000 7f06f7fff700 1 -- 172.21.15.3:0/1073202974 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f800bad0 con 0x7f07000cd4e0 2024-05-08T21:22:55.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.529+0000 7f06f7fff700 1 -- 172.21.15.3:0/1073202974 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f800be80 con 0x7f07000cd4e0 2024-05-08T21:22:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.529+0000 7f07072fe700 1 -- 172.21.15.3:0/1073202974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07000cd4e0 msgr2=0x7f0700099f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.529+0000 7f07072fe700 1 --2- 172.21.15.3:0/1073202974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07000cd4e0 0x7f0700099f00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f06f8009d90 tx=0x7f06f800b330 comp rx=0 tx=0).stop 2024-05-08T21:22:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.529+0000 7f07072fe700 1 -- 172.21.15.3:0/1073202974 shutdown_connections 2024-05-08T21:22:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.529+0000 7f07072fe700 1 --2- 172.21.15.3:0/1073202974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07000cd4e0 0x7f0700099f00 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.529+0000 7f07072fe700 1 -- 172.21.15.3:0/1073202974 >> 172.21.15.3:0/1073202974 conn(0x7f07000b3010 msgr2=0x7f07000b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.529+0000 7f07072fe700 1 -- 172.21.15.3:0/1073202974 shutdown_connections 2024-05-08T21:22:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.529+0000 7f07072fe700 1 -- 172.21.15.3:0/1073202974 wait complete. 2024-05-08T21:22:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.531+0000 7f07072fe700 1 Processor -- start 2024-05-08T21:22:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.531+0000 7f07072fe700 1 -- start start 2024-05-08T21:22:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.531+0000 7f07072fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0700099a50 0x7f070006d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.531+0000 7f07072fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07000cd4e0 0x7f07001f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f07072fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f070015b810 0x7f0700070020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f07072fe700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07000b70e0 con 0x7f07000cd4e0 2024-05-08T21:22:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f07072fe700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07001f3aa0 con 0x7f070015b810 2024-05-08T21:22:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f07072fe700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f070006da60 con 0x7f0700099a50 2024-05-08T21:22:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f0704899700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07000cd4e0 0x7f07001f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f0704899700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07000cd4e0 0x7f07001f3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45554/0 (socket says 172.21.15.3:45554) 2024-05-08T21:22:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f0704899700 1 -- 172.21.15.3:0/369176618 learned_addr learned my addr 172.21.15.3:0/369176618 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f070589b700 1 --2- 172.21.15.3:0/369176618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f070015b810 0x7f0700070020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f070509a700 1 --2- 172.21.15.3:0/369176618 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0700099a50 0x7f070006d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f0704899700 1 -- 172.21.15.3:0/369176618 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0700099a50 msgr2=0x7f070006d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f0704899700 1 --2- 172.21.15.3:0/369176618 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0700099a50 0x7f070006d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f0704899700 1 -- 172.21.15.3:0/369176618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f070015b810 msgr2=0x7f0700070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f0704899700 1 --2- 172.21.15.3:0/369176618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f070015b810 0x7f0700070020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:55.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f0704899700 1 -- 172.21.15.3:0/369176618 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f8009a70 con 0x7f07000cd4e0 2024-05-08T21:22:55.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f070509a700 1 --2- 172.21.15.3:0/369176618 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0700099a50 0x7f070006d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:55.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f0704899700 1 --2- 172.21.15.3:0/369176618 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07000cd4e0 0x7f07001f3f30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f06f000c7b0 tx=0x7f06f000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:55.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f06f5ffb700 1 -- 172.21.15.3:0/369176618 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f0004c80 con 0x7f07000cd4e0 2024-05-08T21:22:55.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f06f5ffb700 1 -- 172.21.15.3:0/369176618 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f0004de0 con 0x7f07000cd4e0 2024-05-08T21:22:55.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f07072fe700 1 -- 172.21.15.3:0/369176618 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07000c74a0 con 0x7f07000cd4e0 2024-05-08T21:22:55.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.532+0000 7f06f5ffb700 1 -- 172.21.15.3:0/369176618 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f00072d0 con 0x7f07000cd4e0 2024-05-08T21:22:55.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.533+0000 7f07072fe700 1 -- 172.21.15.3:0/369176618 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07000c7830 con 0x7f07000cd4e0 2024-05-08T21:22:55.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.534+0000 7f07072fe700 1 -- 172.21.15.3:0/369176618 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0700068000 con 0x7f07000cd4e0 2024-05-08T21:22:55.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.534+0000 7f06f5ffb700 1 -- 172.21.15.3:0/369176618 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f06f0007430 con 0x7f07000cd4e0 2024-05-08T21:22:55.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.534+0000 7f06f5ffb700 1 --2- 172.21.15.3:0/369176618 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f06e406ac90 0x7f06e406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:55.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.535+0000 7f06f5ffb700 1 -- 172.21.15.3:0/369176618 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06f00885b0 con 0x7f07000cd4e0 2024-05-08T21:22:55.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.535+0000 7f070509a700 1 --2- 172.21.15.3:0/369176618 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f06e406ac90 0x7f06e406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:55.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.535+0000 7f070509a700 1 --2- 172.21.15.3:0/369176618 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f06e406ac90 0x7f06e406d150 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f06fc001710 tx=0x7f06fc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:55.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.539+0000 7f06f5ffb700 1 -- 172.21.15.3:0/369176618 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f06f00536f0 con 0x7f07000cd4e0 2024-05-08T21:22:55.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.806+0000 7f07072fe700 1 -- 172.21.15.3:0/369176618 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07001f3be0 con 0x7f07000cd4e0 2024-05-08T21:22:55.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.807+0000 7f06f5ffb700 1 -- 172.21.15.3:0/369176618 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4734 (secure 0 0 0) 0x7f06f0018090 con 0x7f07000cd4e0 2024-05-08T21:22:55.809 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:55.809 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/3571929018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3571929018},{"type":"v1","addr":"172.21.15.186:6836","nonce":3571929018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4760,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3520732046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3520732046},{"type":"v1","addr":"172.21.15.186:6837","nonce":3520732046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4957,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2579650460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":2579650460},{"type":"v1","addr":"172.21.15.3:6835","nonce":2579650460}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:22:50.184245+0000","modified":"2024-05-08T21:22:53.192328+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.3:6834/3599481370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":3599481370},{"type":"v1","addr":"172.21.15.3:6834","nonce":3599481370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-05-08T21:22:55.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.809+0000 7f07072fe700 1 -- 172.21.15.3:0/369176618 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f06e406ac90 msgr2=0x7f06e406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:55.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.809+0000 7f07072fe700 1 --2- 172.21.15.3:0/369176618 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f06e406ac90 0x7f06e406d150 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f06fc001710 tx=0x7f06fc009040 comp rx=0 tx=0).stop 2024-05-08T21:22:55.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.809+0000 7f07072fe700 1 -- 172.21.15.3:0/369176618 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07000cd4e0 msgr2=0x7f07001f3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:55.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.809+0000 7f07072fe700 1 --2- 172.21.15.3:0/369176618 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07000cd4e0 0x7f07001f3f30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f06f000c7b0 tx=0x7f06f000cb70 comp rx=0 tx=0).stop 2024-05-08T21:22:55.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.809+0000 7f07072fe700 1 -- 172.21.15.3:0/369176618 shutdown_connections 2024-05-08T21:22:55.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.809+0000 7f07072fe700 1 --2- 172.21.15.3:0/369176618 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0700099a50 0x7f070006d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:55.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.809+0000 7f07072fe700 1 --2- 172.21.15.3:0/369176618 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f06e406ac90 0x7f06e406d150 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:55.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.809+0000 7f07072fe700 1 --2- 172.21.15.3:0/369176618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f070015b810 0x7f0700070020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:55.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.809+0000 7f07072fe700 1 --2- 172.21.15.3:0/369176618 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07000cd4e0 0x7f07001f3f30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:55.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.809+0000 7f07072fe700 1 -- 172.21.15.3:0/369176618 >> 172.21.15.3:0/369176618 conn(0x7f07000b3010 msgr2=0x7f07000b1860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:55.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.809+0000 7f07072fe700 1 -- 172.21.15.3:0/369176618 shutdown_connections 2024-05-08T21:22:55.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:55.809+0000 7f07072fe700 1 -- 172.21.15.3:0/369176618 wait complete. 2024-05-08T21:22:55.812 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 16 2024-05-08T21:22:56.227 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-08T21:22:50.184245+0000', 'modified': '2024-05-08T21:22:53.192328+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5025}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5025': {'gid': 5025, 'name': 'a', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.3:6834/3599481370', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6832', 'nonce': 3599481370}, {'type': 'v1', 'addr': '172.21.15.3:6834', 'nonce': 3599481370}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-05-08T21:22:56.227 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-08T21:22:56.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-05-08T21:22:56.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.371+0000 7efd10ff4700 1 Processor -- start 2024-05-08T21:22:56.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.371+0000 7efd10ff4700 1 -- start start 2024-05-08T21:22:56.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.372+0000 7efd10ff4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efd0c155da0 0x7efd0c06e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.372+0000 7efd10ff4700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0c06be40 con 0x7efd0c06d4f0 2024-05-08T21:22:56.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.372+0000 7efd10ff4700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0c06c2d0 con 0x7efd0c15f830 2024-05-08T21:22:56.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.372+0000 7efd10ff4700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd0c06dcb0 con 0x7efd0c155da0 2024-05-08T21:22:56.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.372+0000 7efd0a59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efd0c155da0 0x7efd0c06e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.372+0000 7efd0a59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efd0c155da0 0x7efd0c06e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45562/0 (socket says 172.21.15.3:45562) 2024-05-08T21:22:56.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.372+0000 7efd09d9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7efd0c06d4f0 0x7efd0c15f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59810/0 (socket says 172.21.15.3:59810) 2024-05-08T21:22:56.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.372+0000 7efd0a59c700 1 -- 172.21.15.3:0/3203945735 learned_addr learned my addr 172.21.15.3:0/3203945735 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:56.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.373+0000 7efd0a59c700 1 -- 172.21.15.3:0/3203945735 >> v1:172.21.15.186:6789/0 conn(0x7efd0c15f830 legacy=0x7efd0c0ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:56.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.373+0000 7efd0a59c700 1 -- 172.21.15.3:0/3203945735 >> v1:172.21.15.3:6789/0 conn(0x7efd0c06d4f0 legacy=0x7efd0c15f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:56.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.373+0000 7efd0a59c700 1 -- 172.21.15.3:0/3203945735 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0c06b510 con 0x7efd0c155da0 2024-05-08T21:22:56.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.373+0000 7efd0959a700 1 -- 172.21.15.3:0/3203945735 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860592183 0 0) 0x7efd0c06be40 con 0x7efd0c06d4f0 2024-05-08T21:22:56.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.373+0000 7efd0a59c700 1 --2- 172.21.15.3:0/3203945735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd0c155da0 0x7efd0c06e200 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0098f0 tx=0x7efcfc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b71e9c719cbbe4 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:56.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.373+0000 7efd0959a700 1 -- 172.21.15.3:0/3203945735 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc00b8a0 con 0x7efd0c155da0 2024-05-08T21:22:56.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.373+0000 7efd0959a700 1 -- 172.21.15.3:0/3203945735 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc00ba00 con 0x7efd0c155da0 2024-05-08T21:22:56.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.373+0000 7efd0959a700 1 -- 172.21.15.3:0/3203945735 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc00bd90 con 0x7efd0c155da0 2024-05-08T21:22:56.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.374+0000 7efd10ff4700 1 -- 172.21.15.3:0/3203945735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd0c155da0 msgr2=0x7efd0c06e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:56.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.374+0000 7efd10ff4700 1 --2- 172.21.15.3:0/3203945735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd0c155da0 0x7efd0c06e200 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0098f0 tx=0x7efcfc009c00 comp rx=0 tx=0).stop 2024-05-08T21:22:56.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.374+0000 7efd10ff4700 1 -- 172.21.15.3:0/3203945735 shutdown_connections 2024-05-08T21:22:56.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.374+0000 7efd10ff4700 1 --2- 172.21.15.3:0/3203945735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd0c155da0 0x7efd0c06e200 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.374+0000 7efd10ff4700 1 -- 172.21.15.3:0/3203945735 >> 172.21.15.3:0/3203945735 conn(0x7efd0c0cc580 msgr2=0x7efd0c0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:56.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.374+0000 7efd10ff4700 1 -- 172.21.15.3:0/3203945735 shutdown_connections 2024-05-08T21:22:56.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.374+0000 7efd10ff4700 1 -- 172.21.15.3:0/3203945735 wait complete. 2024-05-08T21:22:56.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.375+0000 7efd10ff4700 1 Processor -- start 2024-05-08T21:22:56.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.376+0000 7efd10ff4700 1 -- start start 2024-05-08T21:22:56.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.376+0000 7efd10ff4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd0c06d4f0 0x7efd0c1380f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.376+0000 7efd10ff4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd0c155da0 0x7efd0c0b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.376+0000 7efd10ff4700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efd0c15f830 0x7efd0c0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.376+0000 7efd10ff4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c06e740 con 0x7efd0c06d4f0 2024-05-08T21:22:56.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.376+0000 7efd10ff4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c1e6df0 con 0x7efd0c155da0 2024-05-08T21:22:56.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.376+0000 7efd10ff4700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c1e7280 con 0x7efd0c15f830 2024-05-08T21:22:56.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.376+0000 7efd0a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd0c06d4f0 0x7efd0c1380f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.376+0000 7efd0a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd0c06d4f0 0x7efd0c1380f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45572/0 (socket says 172.21.15.3:45572) 2024-05-08T21:22:56.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.376+0000 7efd0a59c700 1 -- 172.21.15.3:0/3986686612 learned_addr learned my addr 172.21.15.3:0/3986686612 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:56.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.377+0000 7efd0a59c700 1 -- 172.21.15.3:0/3986686612 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efd0c15f830 msgr2=0x7efd0c0b4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:56.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.377+0000 7efd09d9b700 1 --2- 172.21.15.3:0/3986686612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd0c155da0 0x7efd0c0b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.377+0000 7efd0ad9d700 1 --2- 172.21.15.3:0/3986686612 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efd0c15f830 0x7efd0c0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.377+0000 7efd0a59c700 1 --2- 172.21.15.3:0/3986686612 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efd0c15f830 0x7efd0c0b4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.377+0000 7efd0a59c700 1 -- 172.21.15.3:0/3986686612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd0c155da0 msgr2=0x7efd0c0b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:56.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.377+0000 7efd0a59c700 1 --2- 172.21.15.3:0/3986686612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd0c155da0 0x7efd0c0b4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.377+0000 7efd0a59c700 1 -- 172.21.15.3:0/3986686612 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc009550 con 0x7efd0c06d4f0 2024-05-08T21:22:56.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.377+0000 7efd0a59c700 1 --2- 172.21.15.3:0/3986686612 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd0c06d4f0 0x7efd0c1380f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7efd0c0cd140 tx=0x7efcfc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:56.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.377+0000 7efd02ffd700 1 -- 172.21.15.3:0/3986686612 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc011a70 con 0x7efd0c06d4f0 2024-05-08T21:22:56.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.377+0000 7efd02ffd700 1 -- 172.21.15.3:0/3986686612 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc01f070 con 0x7efd0c06d4f0 2024-05-08T21:22:56.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.377+0000 7efd10ff4700 1 -- 172.21.15.3:0/3986686612 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0c138630 con 0x7efd0c06d4f0 2024-05-08T21:22:56.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.377+0000 7efd02ffd700 1 -- 172.21.15.3:0/3986686612 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc011df0 con 0x7efd0c06d4f0 2024-05-08T21:22:56.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.377+0000 7efd10ff4700 1 -- 172.21.15.3:0/3986686612 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0c1e64c0 con 0x7efd0c06d4f0 2024-05-08T21:22:56.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.381+0000 7efd02ffd700 1 -- 172.21.15.3:0/3986686612 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efcfc01a430 con 0x7efd0c06d4f0 2024-05-08T21:22:56.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.381+0000 7efd1d1aeb80 1 Processor -- start 2024-05-08T21:22:56.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.382+0000 7efd1d1aeb80 1 -- start start 2024-05-08T21:22:56.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.382+0000 7efd02ffd700 1 --2- 172.21.15.3:0/3986686612 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efcec06ac90 0x7efcec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.382+0000 7efd02ffd700 1 -- 172.21.15.3:0/3986686612 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcfc021070 con 0x7efd0c06d4f0 2024-05-08T21:22:56.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.382+0000 7efd0ad9d700 1 --2- 172.21.15.3:0/3986686612 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efcec06ac90 0x7efcec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.382+0000 7efd1d1aeb80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 0x55a92802f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.382+0000 7efd1d1aeb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a928031320 0x55a928034050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd1d1aeb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a92805dd80 0x55a92805e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd1d1aeb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a927f8a100 con 0x55a92805dd80 2024-05-08T21:22:56.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd1d1aeb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a927f87ac0 con 0x55a928031320 2024-05-08T21:22:56.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd1d1aeb80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a927f833b0 con 0x55a928056750 2024-05-08T21:22:56.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd09d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a928031320 0x55a928034050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd0a59c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 0x55a92802f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd09d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a928031320 0x55a928034050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:39990/0 (socket says 172.21.15.3:39990) 2024-05-08T21:22:56.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd09d9b700 1 -- 172.21.15.3:0/1821894550 learned_addr learned my addr 172.21.15.3:0/1821894550 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:56.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd0a59c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 0x55a92802f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.3:33332/0 (socket says 172.21.15.3:33332) 2024-05-08T21:22:56.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd0ad9d700 1 --2- 172.21.15.3:0/1821894550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a92805dd80 0x55a92805e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd0ad9d700 1 --2- 172.21.15.3:0/3986686612 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efcec06ac90 0x7efcec06d150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efd040012e0 tx=0x7efd0400d430 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:56.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd0ad9d700 1 -- 172.21.15.3:0/1821894550 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 msgr2=0x55a92802f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:56.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd0ad9d700 1 --2- 172.21.15.3:0/1821894550 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 0x55a92802f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd0ad9d700 1 -- 172.21.15.3:0/1821894550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a928031320 msgr2=0x55a928034050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:56.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd0ad9d700 1 --2- 172.21.15.3:0/1821894550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a928031320 0x55a928034050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd0ad9d700 1 -- 172.21.15.3:0/1821894550 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a927e04360 con 0x55a92805dd80 2024-05-08T21:22:56.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd09d9b700 1 --2- 172.21.15.3:0/1821894550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a928031320 0x55a928034050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:56.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd0a59c700 1 --2- 172.21.15.3:0/1821894550 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 0x55a92802f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:22:56.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efd0ad9d700 1 --2- 172.21.15.3:0/1821894550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a92805dd80 0x55a92805e200 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efd0c141e10 tx=0x7efd04018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b037ed210e85307c server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:56.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efcebfff700 1 -- 172.21.15.3:0/1821894550 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04003c20 con 0x55a92805dd80 2024-05-08T21:22:56.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efcebfff700 1 -- 172.21.15.3:0/1821894550 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd04003d80 con 0x55a92805dd80 2024-05-08T21:22:56.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.383+0000 7efcebfff700 1 -- 172.21.15.3:0/1821894550 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04017ab0 con 0x55a92805dd80 2024-05-08T21:22:56.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.384+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/1821894550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a92805dd80 msgr2=0x55a92805e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:56.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.384+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/1821894550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a92805dd80 0x55a92805e200 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efd0c141e10 tx=0x7efd04018000 comp rx=0 tx=0).stop 2024-05-08T21:22:56.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.384+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/1821894550 shutdown_connections 2024-05-08T21:22:56.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.384+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/1821894550 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 0x55a92802f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.384+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/1821894550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a92805dd80 0x55a92805e200 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.384+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/1821894550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a928031320 0x55a928034050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.384+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/1821894550 >> 172.21.15.3:0/1821894550 conn(0x55a927f911c0 msgr2=0x55a928078d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:56.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.384+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/1821894550 shutdown_connections 2024-05-08T21:22:56.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.384+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/1821894550 wait complete. 2024-05-08T21:22:56.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.385+0000 7efd1d1aeb80 1 -- start start 2024-05-08T21:22:56.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.385+0000 7efd1d1aeb80 1 Processor -- start 2024-05-08T21:22:56.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.386+0000 7efd1d1aeb80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928031320 0x55a92807a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.386+0000 7efd1d1aeb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a928056750 0x55a928055570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.386+0000 7efd1d1aeb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a92805dd80 0x55a9280609e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.386+0000 7efd1d1aeb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a927f1f510 con 0x55a928056750 2024-05-08T21:22:56.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.386+0000 7efd1d1aeb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a927f1f1b0 con 0x55a92805dd80 2024-05-08T21:22:56.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.386+0000 7efd1d1aeb80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a927e06650 con 0x55a928031320 2024-05-08T21:22:56.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.386+0000 7efd1d1aeb80 10 client.0 _ll_register_callbacks cb 0x55a927f28ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-08T21:22:56.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.386+0000 7efd09d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a928056750 0x55a928055570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.386+0000 7efd09d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a928056750 0x55a928055570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45592/0 (socket says 172.21.15.3:45592) 2024-05-08T21:22:56.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.386+0000 7efd09d9b700 1 -- 172.21.15.3:0/2413868784 learned_addr learned my addr 172.21.15.3:0/2413868784 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:56.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.386+0000 7efd0ad9d700 1 --2- 172.21.15.3:0/2413868784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a92805dd80 0x55a9280609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.386+0000 7efd0a59c700 1 --2- 172.21.15.3:0/2413868784 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928031320 0x55a92807a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.387+0000 7efd09d9b700 1 -- 172.21.15.3:0/2413868784 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928031320 msgr2=0x55a92807a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:56.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.387+0000 7efd09d9b700 1 --2- 172.21.15.3:0/2413868784 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928031320 0x55a92807a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.387+0000 7efd09d9b700 1 -- 172.21.15.3:0/2413868784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a92805dd80 msgr2=0x55a9280609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:56.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.387+0000 7efd09d9b700 1 --2- 172.21.15.3:0/2413868784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a92805dd80 0x55a9280609e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.387+0000 7efd09d9b700 1 -- 172.21.15.3:0/2413868784 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0c1e64c0 con 0x55a928056750 2024-05-08T21:22:56.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.387+0000 7efd0ad9d700 1 --2- 172.21.15.3:0/2413868784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a92805dd80 0x55a9280609e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:56.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.387+0000 7efd09d9b700 1 --2- 172.21.15.3:0/2413868784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a928056750 0x55a928055570 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efcf40011e0 tx=0x7efcf4001590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d43e4d793babc0d server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:56.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.387+0000 7efce97fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-05-08T21:22:56.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.387+0000 7efce97fa700 1 -- 172.21.15.3:0/2413868784 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf4004980 con 0x55a928056750 2024-05-08T21:22:56.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.387+0000 7efce97fa700 1 -- 172.21.15.3:0/2413868784 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcf40066f0 con 0x55a928056750 2024-05-08T21:22:56.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.387+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2413868784 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a927edc220 con 0x55a928056750 2024-05-08T21:22:56.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.387+0000 7efce97fa700 1 -- 172.21.15.3:0/2413868784 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7efcf4001d50 con 0x55a928056750 2024-05-08T21:22:56.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.387+0000 7efd1d1aeb80 10 client.5190 fetch_fsmap learned FSMap version 16 2024-05-08T21:22:56.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.387+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2413868784 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x7efd0c06c2d0 con 0x55a928056750 2024-05-08T21:22:56.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.387+0000 7efce97fa700 1 -- 172.21.15.3:0/2413868784 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf4024020 con 0x55a928056750 2024-05-08T21:22:56.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.387+0000 7efce97fa700 1 -- 172.21.15.3:0/2413868784 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 16) v1 ==== 3114+0+0 (secure 0 0 0) 0x7efcf4012040 con 0x55a928056750 2024-05-08T21:22:56.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.388+0000 7efd1d1aeb80 10 client.5190 fetch_fsmap finished waiting for FSMap version 16 2024-05-08T21:22:56.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.388+0000 7efd1d1aeb80 10 client.5190 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-08T21:22:56.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.388+0000 7efd1d1aeb80 20 client.5190 populate_metadata read hostname 'smithi003' 2024-05-08T21:22:56.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.388+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/2413868784 >> [v2:172.21.15.3:6832/3599481370,v1:172.21.15.3:6834/3599481370] conn(0x55a9280885a0 0x55a92805ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.388+0000 7efd1d1aeb80 4 client.5190 mds_command: new command op to 5025 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-08T21:22:56.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.388+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2413868784 --> [v2:172.21.15.3:6832/3599481370,v1:172.21.15.3:6834/3599481370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a928002790 con 0x55a9280885a0 2024-05-08T21:22:56.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.388+0000 7efd0a59c700 1 --2- 172.21.15.3:0/2413868784 >> [v2:172.21.15.3:6832/3599481370,v1:172.21.15.3:6834/3599481370] conn(0x55a9280885a0 0x55a92805ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.388+0000 7efd0a59c700 1 --2- 172.21.15.3:0/2413868784 >> [v2:172.21.15.3:6832/3599481370,v1:172.21.15.3:6834/3599481370] conn(0x55a9280885a0 0x55a92805ef20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65b87dea4fc4d439 server_cookie=b81ec475776fb484 in_seq=0 out_seq=0 2024-05-08T21:22:56.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.388+0000 7efce97fa700 10 client.5190 ms_handle_connect on v2:172.21.15.3:6832/3599481370 2024-05-08T21:22:56.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.389+0000 7efce97fa700 1 -- 172.21.15.3:0/2413868784 <== mds.0 v2:172.21.15.3:6832/3599481370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a928002790 con 0x55a9280885a0 2024-05-08T21:22:56.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.389+0000 7efce97fa700 10 client.5190 handle_command_reply: tid=0 2024-05-08T21:22:56.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.389+0000 7efd1d1aeb80 1 client.5190 shutdown 2024-05-08T21:22:56.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.389+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2413868784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a928056750 msgr2=0x55a928055570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:56.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.389+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/2413868784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a928056750 0x55a928055570 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efcf40011e0 tx=0x7efcf4001590 comp rx=0 tx=0).stop 2024-05-08T21:22:56.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.389+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2413868784 shutdown_connections 2024-05-08T21:22:56.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.389+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/2413868784 >> [v2:172.21.15.3:6832/3599481370,v1:172.21.15.3:6834/3599481370] conn(0x55a9280885a0 0x55a92805ef20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.389+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/2413868784 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928031320 0x55a92807a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.389+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/2413868784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a92805dd80 0x55a9280609e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.389+0000 7efce97fa700 0 client.5190 ms_handle_reset on v2:172.21.15.3:6832/3599481370 2024-05-08T21:22:56.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.389+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/2413868784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a928056750 0x55a928055570 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.389+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2413868784 >> 172.21.15.3:0/2413868784 conn(0x55a927f911c0 msgr2=0x55a92805d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:56.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.389+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2413868784 shutdown_connections 2024-05-08T21:22:56.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.389+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2413868784 wait complete. 2024-05-08T21:22:56.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.389+0000 7efd1d1aeb80 20 client.5190 trim_cache size 0 max 16384 2024-05-08T21:22:56.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.408+0000 7efd1d1aeb80 1 Processor -- start 2024-05-08T21:22:56.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.408+0000 7efd1d1aeb80 1 -- start start 2024-05-08T21:22:56.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.409+0000 7efd1d1aeb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a928031320 0x55a92809ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.409+0000 7efd1d1aeb80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 0x55a9280606b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.409+0000 7efd1d1aeb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a92805dd80 0x55a92805aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.409+0000 7efd1d1aeb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a92802ce30 con 0x55a92805dd80 2024-05-08T21:22:56.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.409+0000 7efd1d1aeb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a92808b240 con 0x55a928031320 2024-05-08T21:22:56.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.409+0000 7efd1d1aeb80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a9280616f0 con 0x55a928056750 2024-05-08T21:22:56.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.409+0000 7efd0ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a92805dd80 0x55a92805aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.409+0000 7efd0ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a92805dd80 0x55a92805aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45594/0 (socket says 172.21.15.3:45594) 2024-05-08T21:22:56.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.409+0000 7efd0ad9d700 1 -- 172.21.15.3:0/2059854554 learned_addr learned my addr 172.21.15.3:0/2059854554 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:56.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.409+0000 7efd0a59c700 1 --2- 172.21.15.3:0/2059854554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a928031320 0x55a92809ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.409+0000 7efd09d9b700 1 --2- 172.21.15.3:0/2059854554 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 0x55a9280606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.409+0000 7efd0ad9d700 1 -- 172.21.15.3:0/2059854554 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 msgr2=0x55a9280606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.409+0000 7efd0ad9d700 1 --2- 172.21.15.3:0/2059854554 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 0x55a9280606b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.409+0000 7efd0ad9d700 1 -- 172.21.15.3:0/2059854554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a928031320 msgr2=0x55a92809ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.409+0000 7efd0ad9d700 1 --2- 172.21.15.3:0/2059854554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a928031320 0x55a92809ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.409+0000 7efd0ad9d700 1 -- 172.21.15.3:0/2059854554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a928088c50 con 0x55a92805dd80 2024-05-08T21:22:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.410+0000 7efd0ad9d700 1 --2- 172.21.15.3:0/2059854554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a92805dd80 0x55a92805aa10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efd0400dcc0 tx=0x7efd04009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3f0b4290fc7c434 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.410+0000 7efd00ff9700 1 -- 172.21.15.3:0/2059854554 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04003c20 con 0x55a92805dd80 2024-05-08T21:22:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.410+0000 7efd00ff9700 1 -- 172.21.15.3:0/2059854554 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd04017870 con 0x55a92805dd80 2024-05-08T21:22:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.410+0000 7efd00ff9700 1 -- 172.21.15.3:0/2059854554 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04017c00 con 0x55a92805dd80 2024-05-08T21:22:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.410+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2059854554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a92805dd80 msgr2=0x55a92805aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.410+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/2059854554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a92805dd80 0x55a92805aa10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efd0400dcc0 tx=0x7efd04009040 comp rx=0 tx=0).stop 2024-05-08T21:22:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.410+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2059854554 shutdown_connections 2024-05-08T21:22:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.410+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/2059854554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a92805dd80 0x55a92805aa10 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.410+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/2059854554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a928031320 0x55a92809ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.410+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/2059854554 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 0x55a9280606b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.410+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2059854554 >> 172.21.15.3:0/2059854554 conn(0x55a927f911c0 msgr2=0x55a9280302b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.410+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2059854554 shutdown_connections 2024-05-08T21:22:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.410+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2059854554 wait complete. 2024-05-08T21:22:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.411+0000 7efd1d1aeb80 1 -- start start 2024-05-08T21:22:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.411+0000 7efd1d1aeb80 1 Processor -- start 2024-05-08T21:22:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.412+0000 7efd1d1aeb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a928031320 0x55a928033e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.412+0000 7efd1d1aeb80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 0x55a9280618a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.412+0000 7efd1d1aeb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a92805dd80 0x55a928061de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.412+0000 7efd1d1aeb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9280785e0 con 0x55a928031320 2024-05-08T21:22:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.412+0000 7efd1d1aeb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9280c7540 con 0x55a92805dd80 2024-05-08T21:22:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.412+0000 7efd1d1aeb80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a9280dc490 con 0x55a928056750 2024-05-08T21:22:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.412+0000 7efd1d1aeb80 10 client.0 _ll_register_callbacks cb 0x7efcf4004e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-08T21:22:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.412+0000 7efd0a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a928031320 0x55a928033e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.412+0000 7efd0a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a928031320 0x55a928033e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45596/0 (socket says 172.21.15.3:45596) 2024-05-08T21:22:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.412+0000 7efd0a59c700 1 -- 172.21.15.3:0/2447962726 learned_addr learned my addr 172.21.15.3:0/2447962726 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.412+0000 7efd0ad9d700 1 --2- 172.21.15.3:0/2447962726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a92805dd80 0x55a928061de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.412+0000 7efd09d9b700 1 --2- 172.21.15.3:0/2447962726 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 0x55a9280618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efd0a59c700 1 -- 172.21.15.3:0/2447962726 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 msgr2=0x55a9280618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efd0a59c700 1 --2- 172.21.15.3:0/2447962726 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 0x55a9280618a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efd0a59c700 1 -- 172.21.15.3:0/2447962726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a92805dd80 msgr2=0x55a928061de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efd0a59c700 1 --2- 172.21.15.3:0/2447962726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a92805dd80 0x55a928061de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efd0a59c700 1 -- 172.21.15.3:0/2447962726 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a927e04360 con 0x55a928031320 2024-05-08T21:22:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efd0a59c700 1 --2- 172.21.15.3:0/2447962726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a928031320 0x55a928033e30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efcfc05b010 tx=0x7efcfc05f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb2e93cb711baf1c server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efce97fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-05-08T21:22:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efce97fa700 1 -- 172.21.15.3:0/2447962726 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc061330 con 0x55a928031320 2024-05-08T21:22:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efce97fa700 1 -- 172.21.15.3:0/2447962726 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcfc0584e0 con 0x55a928031320 2024-05-08T21:22:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2447962726 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd04003d80 con 0x55a928031320 2024-05-08T21:22:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efce97fa700 1 -- 172.21.15.3:0/2447962726 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7efcfc0586a0 con 0x55a928031320 2024-05-08T21:22:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efd1d1aeb80 10 client.5196 fetch_fsmap learned FSMap version 16 2024-05-08T21:22:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2447962726 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55a927db0b10 con 0x55a928031320 2024-05-08T21:22:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efce97fa700 1 -- 172.21.15.3:0/2447962726 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc074080 con 0x55a928031320 2024-05-08T21:22:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efce97fa700 1 -- 172.21.15.3:0/2447962726 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 16) v1 ==== 3114+0+0 (secure 0 0 0) 0x7efcfc060ea0 con 0x55a928031320 2024-05-08T21:22:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efd1d1aeb80 10 client.5196 fetch_fsmap finished waiting for FSMap version 16 2024-05-08T21:22:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efd1d1aeb80 10 client.5196 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-08T21:22:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efd1d1aeb80 20 client.5196 populate_metadata read hostname 'smithi003' 2024-05-08T21:22:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.413+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/2447962726 >> [v2:172.21.15.3:6832/3599481370,v1:172.21.15.3:6834/3599481370] conn(0x55a9280e29c0 0x55a9280915b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efd1d1aeb80 4 client.5196 mds_command: new command op to 5025 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-08T21:22:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2447962726 --> [v2:172.21.15.3:6832/3599481370,v1:172.21.15.3:6834/3599481370] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a927f7be70 con 0x55a9280e29c0 2024-05-08T21:22:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efd09d9b700 1 --2- 172.21.15.3:0/2447962726 >> [v2:172.21.15.3:6832/3599481370,v1:172.21.15.3:6834/3599481370] conn(0x55a9280e29c0 0x55a9280915b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efce97fa700 10 client.5196 ms_handle_connect on v2:172.21.15.3:6832/3599481370 2024-05-08T21:22:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efd09d9b700 1 --2- 172.21.15.3:0/2447962726 >> [v2:172.21.15.3:6832/3599481370,v1:172.21.15.3:6834/3599481370] conn(0x55a9280e29c0 0x55a9280915b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9aa2008c991eb9eb server_cookie=c16bfc4c2b5528dc in_seq=0 out_seq=0 2024-05-08T21:22:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efce97fa700 1 -- 172.21.15.3:0/2447962726 <== mds.0 v2:172.21.15.3:6832/3599481370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55a927f7be70 con 0x55a9280e29c0 2024-05-08T21:22:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efce97fa700 10 client.5196 handle_command_reply: tid=0 2024-05-08T21:22:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efd1d1aeb80 1 client.5196 shutdown 2024-05-08T21:22:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2447962726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a928031320 msgr2=0x55a928033e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/2447962726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a928031320 0x55a928033e30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efcfc05b010 tx=0x7efcfc05f040 comp rx=0 tx=0).stop 2024-05-08T21:22:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2447962726 shutdown_connections 2024-05-08T21:22:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/2447962726 >> [v2:172.21.15.3:6832/3599481370,v1:172.21.15.3:6834/3599481370] conn(0x55a9280e29c0 0x55a9280915b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/2447962726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a92805dd80 0x55a928061de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/2447962726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a928031320 0x55a928033e30 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efd1d1aeb80 1 --2- 172.21.15.3:0/2447962726 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a928056750 0x55a9280618a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efce97fa700 0 client.5196 ms_handle_reset on v2:172.21.15.3:6832/3599481370 2024-05-08T21:22:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.414+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2447962726 >> 172.21.15.3:0/2447962726 conn(0x55a927f911c0 msgr2=0x55a92805e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.415+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2447962726 shutdown_connections 2024-05-08T21:22:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.415+0000 7efd1d1aeb80 1 -- 172.21.15.3:0/2447962726 wait complete. 2024-05-08T21:22:56.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.415+0000 7efd1d1aeb80 20 client.5196 trim_cache size 0 max 16384 2024-05-08T21:22:56.429 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:22:56.429 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "d0f58275-758f-4470-b003-84d6abe25489", 2024-05-08T21:22:56.429 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 0, 2024-05-08T21:22:56.429 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 5025, 2024-05-08T21:22:56.429 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2024-05-08T21:22:56.429 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2024-05-08T21:22:56.429 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2024-05-08T21:22:56.429 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 6.2293338739999999, 2024-05-08T21:22:56.430 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 16, 2024-05-08T21:22:56.430 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 34, 2024-05-08T21:22:56.430 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 34, 2024-05-08T21:22:56.430 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 14.553498567 2024-05-08T21:22:56.430 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:22:56.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.415+0000 7efd10ff4700 1 -- 172.21.15.3:0/3986686612 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efcec06ac90 msgr2=0x7efcec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:56.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.415+0000 7efd10ff4700 1 --2- 172.21.15.3:0/3986686612 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efcec06ac90 0x7efcec06d150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efd040012e0 tx=0x7efd0400d430 comp rx=0 tx=0).stop 2024-05-08T21:22:56.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.415+0000 7efd10ff4700 1 -- 172.21.15.3:0/3986686612 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd0c06d4f0 msgr2=0x7efd0c1380f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:56.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.415+0000 7efd10ff4700 1 --2- 172.21.15.3:0/3986686612 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd0c06d4f0 0x7efd0c1380f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7efd0c0cd140 tx=0x7efcfc004750 comp rx=0 tx=0).stop 2024-05-08T21:22:56.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.415+0000 7efd10ff4700 1 -- 172.21.15.3:0/3986686612 shutdown_connections 2024-05-08T21:22:56.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.415+0000 7efd10ff4700 1 --2- 172.21.15.3:0/3986686612 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efd0c15f830 0x7efd0c0b4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.415+0000 7efd10ff4700 1 --2- 172.21.15.3:0/3986686612 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efcec06ac90 0x7efcec06d150 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.415+0000 7efd10ff4700 1 --2- 172.21.15.3:0/3986686612 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd0c06d4f0 0x7efd0c1380f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.415+0000 7efd10ff4700 1 --2- 172.21.15.3:0/3986686612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd0c155da0 0x7efd0c0b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.415+0000 7efd10ff4700 1 -- 172.21.15.3:0/3986686612 >> 172.21.15.3:0/3986686612 conn(0x7efd0c0cc580 msgr2=0x7efd0c0ca890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:56.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.415+0000 7efd10ff4700 1 -- 172.21.15.3:0/3986686612 shutdown_connections 2024-05-08T21:22:56.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.415+0000 7efd10ff4700 1 -- 172.21.15.3:0/3986686612 wait complete. 2024-05-08T21:22:56.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-08T21:22:56.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.568+0000 7fdf97039700 1 Processor -- start 2024-05-08T21:22:56.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.568+0000 7fdf97039700 1 -- start start 2024-05-08T21:22:56.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf97039700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdf900cf840 0x7fdf900af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf97039700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf9006a960 0x7fdf9015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf97039700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf900cd980 con 0x7fdf900d0eb0 2024-05-08T21:22:56.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf97039700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf900ccdb0 con 0x7fdf900cf840 2024-05-08T21:22:56.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf97039700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf900cd050 con 0x7fdf9006a960 2024-05-08T21:22:56.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf94dd5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdf900cf840 0x7fdf900af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf94dd5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdf900cf840 0x7fdf900af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45604/0 (socket says 172.21.15.3:45604) 2024-05-08T21:22:56.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf94dd5700 1 -- 172.21.15.3:0/672286697 learned_addr learned my addr 172.21.15.3:0/672286697 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:56.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf8ffff700 1 --2- 172.21.15.3:0/672286697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf9006a960 0x7fdf9015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf94dd5700 1 -- 172.21.15.3:0/672286697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf9006a960 msgr2=0x7fdf9015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:56.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf94dd5700 1 --2- 172.21.15.3:0/672286697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf9006a960 0x7fdf9015d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf94dd5700 1 -- 172.21.15.3:0/672286697 >> v1:172.21.15.186:6789/0 conn(0x7fdf900d0eb0 legacy=0x7fdf9006dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:56.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf94dd5700 1 -- 172.21.15.3:0/672286697 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf900cd330 con 0x7fdf900cf840 2024-05-08T21:22:56.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf94dd5700 1 --2- 172.21.15.3:0/672286697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf900cf840 0x7fdf900af440 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdf80009920 tx=0x7fdf80009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b38fd6e88292b04 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:56.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf8f7fe700 1 -- 172.21.15.3:0/672286697 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80004230 con 0x7fdf900cf840 2024-05-08T21:22:56.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf8f7fe700 1 -- 172.21.15.3:0/672286697 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf80004390 con 0x7fdf900cf840 2024-05-08T21:22:56.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.569+0000 7fdf8f7fe700 1 -- 172.21.15.3:0/672286697 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80009e70 con 0x7fdf900cf840 2024-05-08T21:22:56.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.570+0000 7fdf97039700 1 -- 172.21.15.3:0/672286697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf900cf840 msgr2=0x7fdf900af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:56.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.570+0000 7fdf97039700 1 --2- 172.21.15.3:0/672286697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf900cf840 0x7fdf900af440 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdf80009920 tx=0x7fdf80009c30 comp rx=0 tx=0).stop 2024-05-08T21:22:56.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.570+0000 7fdf97039700 1 -- 172.21.15.3:0/672286697 shutdown_connections 2024-05-08T21:22:56.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.570+0000 7fdf97039700 1 --2- 172.21.15.3:0/672286697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf900cf840 0x7fdf900af440 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.570+0000 7fdf97039700 1 --2- 172.21.15.3:0/672286697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf9006a960 0x7fdf9015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.570+0000 7fdf97039700 1 -- 172.21.15.3:0/672286697 >> 172.21.15.3:0/672286697 conn(0x7fdf900d0320 msgr2=0x7fdf900d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:56.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.570+0000 7fdf97039700 1 -- 172.21.15.3:0/672286697 shutdown_connections 2024-05-08T21:22:56.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.570+0000 7fdf97039700 1 -- 172.21.15.3:0/672286697 wait complete. 2024-05-08T21:22:56.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.571+0000 7fdf97039700 1 Processor -- start 2024-05-08T21:22:56.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.572+0000 7fdf97039700 1 -- start start 2024-05-08T21:22:56.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.572+0000 7fdf97039700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf9006a960 0x7fdf9014a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.572+0000 7fdf97039700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf900cf840 0x7fdf900c4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.572+0000 7fdf97039700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf900d0eb0 0x7fdf900c63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.572+0000 7fdf97039700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf900c60c0 con 0x7fdf9006a960 2024-05-08T21:22:56.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.572+0000 7fdf97039700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9014ac30 con 0x7fdf900cf840 2024-05-08T21:22:56.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.572+0000 7fdf97039700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf900c5310 con 0x7fdf900d0eb0 2024-05-08T21:22:56.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf94dd5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf9006a960 0x7fdf9014a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf94dd5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf9006a960 0x7fdf9014a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45610/0 (socket says 172.21.15.3:45610) 2024-05-08T21:22:56.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf94dd5700 1 -- 172.21.15.3:0/1681284353 learned_addr learned my addr 172.21.15.3:0/1681284353 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:56.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf8ffff700 1 --2- 172.21.15.3:0/1681284353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf900cf840 0x7fdf900c4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf955d6700 1 --2- 172.21.15.3:0/1681284353 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf900d0eb0 0x7fdf900c63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf94dd5700 1 -- 172.21.15.3:0/1681284353 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf900d0eb0 msgr2=0x7fdf900c63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:56.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf94dd5700 1 --2- 172.21.15.3:0/1681284353 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf900d0eb0 0x7fdf900c63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf94dd5700 1 -- 172.21.15.3:0/1681284353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf900cf840 msgr2=0x7fdf900c4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:56.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf94dd5700 1 --2- 172.21.15.3:0/1681284353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf900cf840 0x7fdf900c4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf94dd5700 1 -- 172.21.15.3:0/1681284353 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf80009550 con 0x7fdf9006a960 2024-05-08T21:22:56.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf955d6700 1 --2- 172.21.15.3:0/1681284353 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf900d0eb0 0x7fdf900c63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:56.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf94dd5700 1 --2- 172.21.15.3:0/1681284353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf9006a960 0x7fdf9014a6f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdf900704d0 tx=0x7fdf80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:56.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf8d7fa700 1 -- 172.21.15.3:0/1681284353 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8000bae0 con 0x7fdf9006a960 2024-05-08T21:22:56.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf8d7fa700 1 -- 172.21.15.3:0/1681284353 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8001f070 con 0x7fdf9006a960 2024-05-08T21:22:56.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf97039700 1 -- 172.21.15.3:0/1681284353 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf900c5540 con 0x7fdf9006a960 2024-05-08T21:22:56.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf8d7fa700 1 -- 172.21.15.3:0/1681284353 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8000be80 con 0x7fdf9006a960 2024-05-08T21:22:56.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.573+0000 7fdf97039700 1 -- 172.21.15.3:0/1681284353 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf900c5b00 con 0x7fdf9006a960 2024-05-08T21:22:56.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.574+0000 7fdf97039700 1 -- 172.21.15.3:0/1681284353 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf90068000 con 0x7fdf9006a960 2024-05-08T21:22:56.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.575+0000 7fdf8d7fa700 1 -- 172.21.15.3:0/1681284353 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdf80021030 con 0x7fdf9006a960 2024-05-08T21:22:56.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.576+0000 7fdf8d7fa700 1 --2- 172.21.15.3:0/1681284353 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdf7c06ac90 0x7fdf7c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:56.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.576+0000 7fdf8d7fa700 1 -- 172.21.15.3:0/1681284353 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf8008a830 con 0x7fdf9006a960 2024-05-08T21:22:56.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.576+0000 7fdf8ffff700 1 --2- 172.21.15.3:0/1681284353 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdf7c06ac90 0x7fdf7c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:56.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.576+0000 7fdf8ffff700 1 --2- 172.21.15.3:0/1681284353 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdf7c06ac90 0x7fdf7c06d150 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdf900b2380 tx=0x7fdf84008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:56.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.581+0000 7fdf8d7fa700 1 -- 172.21.15.3:0/1681284353 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fdf80055aa0 con 0x7fdf9006a960 2024-05-08T21:22:56.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.851+0000 7fdf97039700 1 -- 172.21.15.3:0/1681284353 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdf90068000 con 0x7fdf9006a960 2024-05-08T21:22:56.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.851+0000 7fdf8d7fa700 1 -- 172.21.15.3:0/1681284353 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7fdf800590c0 con 0x7fdf9006a960 2024-05-08T21:22:56.854 INFO:teuthology.orchestra.run.smithi003.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-08T21:22:56.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.853+0000 7fdf97039700 1 -- 172.21.15.3:0/1681284353 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdf7c06ac90 msgr2=0x7fdf7c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:56.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.853+0000 7fdf97039700 1 --2- 172.21.15.3:0/1681284353 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdf7c06ac90 0x7fdf7c06d150 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdf900b2380 tx=0x7fdf84008040 comp rx=0 tx=0).stop 2024-05-08T21:22:56.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.853+0000 7fdf97039700 1 -- 172.21.15.3:0/1681284353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf9006a960 msgr2=0x7fdf9014a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:56.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.853+0000 7fdf97039700 1 --2- 172.21.15.3:0/1681284353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf9006a960 0x7fdf9014a6f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdf900704d0 tx=0x7fdf80004750 comp rx=0 tx=0).stop 2024-05-08T21:22:56.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.854+0000 7fdf97039700 1 -- 172.21.15.3:0/1681284353 shutdown_connections 2024-05-08T21:22:56.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.854+0000 7fdf97039700 1 --2- 172.21.15.3:0/1681284353 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf900d0eb0 0x7fdf900c63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.854+0000 7fdf97039700 1 --2- 172.21.15.3:0/1681284353 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdf7c06ac90 0x7fdf7c06d150 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.854+0000 7fdf97039700 1 --2- 172.21.15.3:0/1681284353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf9006a960 0x7fdf9014a6f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.854+0000 7fdf97039700 1 --2- 172.21.15.3:0/1681284353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf900cf840 0x7fdf900c4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:56.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.854+0000 7fdf97039700 1 -- 172.21.15.3:0/1681284353 >> 172.21.15.3:0/1681284353 conn(0x7fdf900d0320 msgr2=0x7fdf9006f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:56.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.854+0000 7fdf97039700 1 -- 172.21.15.3:0/1681284353 shutdown_connections 2024-05-08T21:22:56.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:56.854+0000 7fdf97039700 1 -- 172.21.15.3:0/1681284353 wait complete. 2024-05-08T21:22:56.869 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-08T21:22:56.869 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-08T21:22:56.869 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi003.front.sepia.ceph.com 2024-05-08T21:22:56.869 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-08T21:22:56.869 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-08T21:22:56.869 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-08T21:22:56.869 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-05-08T21:22:56.869 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-08T21:22:56.869 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-08T21:22:56.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:56.870 DEBUG:teuthology.orchestra.run.smithi003:> ip addr 2024-05-08T21:22:56.889 INFO:teuthology.orchestra.run.smithi003.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-08T21:22:56.889 INFO:teuthology.orchestra.run.smithi003.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-08T21:22:56.889 INFO:teuthology.orchestra.run.smithi003.stdout: inet 127.0.0.1/8 scope host lo 2024-05-08T21:22:56.890 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:56.890 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 ::1/128 scope host 2024-05-08T21:22:56.890 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:56.890 INFO:teuthology.orchestra.run.smithi003.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:22:56.890 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ac brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:56.890 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f0 2024-05-08T21:22:56.890 INFO:teuthology.orchestra.run.smithi003.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:22:56.890 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ad brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:56.890 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f1 2024-05-08T21:22:56.890 INFO:teuthology.orchestra.run.smithi003.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-08T21:22:56.890 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dc brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:56.890 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f0 2024-05-08T21:22:56.890 INFO:teuthology.orchestra.run.smithi003.stdout: inet 172.21.15.3/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-08T21:22:56.890 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft 42155sec preferred_lft 42155sec 2024-05-08T21:22:56.891 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::ec4:7aff:febd:15dc/64 scope link 2024-05-08T21:22:56.891 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:56.891 INFO:teuthology.orchestra.run.smithi003.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:22:56.891 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dd brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:56.891 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f1 2024-05-08T21:22:56.891 INFO:teuthology.orchestra.run.smithi003.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-08T21:22:56.891 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether a6:6a:ef:07:5e:cb brd ff:ff:ff:ff:ff:ff 2024-05-08T21:22:56.891 INFO:teuthology.orchestra.run.smithi003.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-08T21:22:56.891 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:56.891 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::a89e:98ff:fec6:3929/64 scope link 2024-05-08T21:22:56.891 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:56.891 INFO:teuthology.orchestra.run.smithi003.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-08T21:22:56.891 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether ee:e2:a2:93:f0:94 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-08T21:22:56.891 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::ece2:a2ff:fe93:f094/64 scope link 2024-05-08T21:22:56.892 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:56.892 INFO:teuthology.orchestra.run.smithi003.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-08T21:22:56.892 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether a6:6a:ef:07:5e:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-08T21:22:56.892 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::a46a:efff:fe07:5ecb/64 scope link 2024-05-08T21:22:56.892 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:22:56.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:56.892 DEBUG:teuthology.orchestra.run.smithi003:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-08T21:22:56.960 INFO:teuthology.orchestra.run.smithi003.stdout:1 2024-05-08T21:22:57.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:57.048 DEBUG:teuthology.orchestra.run.smithi003:> route 2024-05-08T21:22:57.113 INFO:teuthology.orchestra.run.smithi003.stdout:Kernel IP routing table 2024-05-08T21:22:57.113 INFO:teuthology.orchestra.run.smithi003.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-08T21:22:57.113 INFO:teuthology.orchestra.run.smithi003.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-08T21:22:57.113 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-08T21:22:57.113 INFO:teuthology.orchestra.run.smithi003.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-08T21:22:57.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:57.113 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2024-05-08T21:22:57.113 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-05-08T21:22:57.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-08T21:22:57.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-08T21:22:57.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-08T21:22:57.114 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-05-08T21:22:57.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:57.238 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2024-05-08T21:22:57.262 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-08T21:22:57.262 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-08T21:22:57.262 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-08T21:22:57.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:57.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set brx.0 up 2024-05-08T21:22:57.338 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-08T21:22:57.338 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-08T21:22:57.399 INFO:teuthology.orchestra.run.smithi003.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-08T21:22:57.400 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-08T21:22:57.400 DEBUG:teuthology.orchestra.run.smithi003:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-08T21:22:57.460 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-05-08T21:22:57.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:57.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-08T21:22:57.558 INFO:teuthology.orchestra.run.smithi003.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-08T21:22:57.559 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-05-08T21:22:57.559 INFO:teuthology.orchestra.run.smithi003.stdout:invalid new device string format 2024-05-08T21:22:57.569 INFO:teuthology.orchestra.run.smithi003.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-08T21:22:57.569 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-05-08T21:22:57.569 INFO:teuthology.orchestra.run.smithi003.stdout:invalid new device string format 2024-05-08T21:22:57.569 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: resolved to: "172.21.15.3:3300,172.21.15.186:3300,172.21.15.186:3301" 2024-05-08T21:22:57.569 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-08T21:22:57.590 INFO:teuthology.orchestra.run.smithi003.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-08T21:22:57.590 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-05-08T21:22:57.590 INFO:teuthology.orchestra.run.smithi003.stdout:invalid new device string format 2024-05-08T21:22:57.590 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: resolved to: "172.21.15.3:3300,172.21.15.186:3300,172.21.15.186:3301" 2024-05-08T21:22:57.591 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-08T21:22:57.591 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: resolved to: "172.21.15.3:6789,172.21.15.186:6789,172.21.15.186:6790" 2024-05-08T21:22:57.591 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.3:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2024-05-08T21:22:57.591 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=d0f58275-758f-4470-b003-84d6abe25489" will pass to kernel 2024-05-08T21:22:57.594 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-08T21:22:57.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:57.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-08T21:22:57.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:57.626 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-08T21:22:57.728 INFO:teuthology.orchestra.run.smithi003.stdout:client5137 2024-05-08T21:22:57.728 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:22:57.729 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:57.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:57.751 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-05-08T21:22:57.821 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:22:57.821 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:57.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:22:57.884 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=subdir/sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-05-08T21:22:58.008 INFO:teuthology.orchestra.run.smithi003.stderr:6+0 records in 2024-05-08T21:22:58.009 INFO:teuthology.orchestra.run.smithi003.stderr:6+0 records out 2024-05-08T21:22:58.009 INFO:teuthology.orchestra.run.smithi003.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0606491 s, 104 MB/s 2024-05-08T21:22:58.010 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits daemon-helper kill python3 -c ' 2024-05-08T21:22:58.010 DEBUG:teuthology.orchestra.run.smithi003:> import os 2024-05-08T21:22:58.010 DEBUG:teuthology.orchestra.run.smithi003:> import stat 2024-05-08T21:22:58.010 DEBUG:teuthology.orchestra.run.smithi003:> import json 2024-05-08T21:22:58.010 DEBUG:teuthology.orchestra.run.smithi003:> import sys 2024-05-08T21:22:58.010 DEBUG:teuthology.orchestra.run.smithi003:> 2024-05-08T21:22:58.010 DEBUG:teuthology.orchestra.run.smithi003:> try: 2024-05-08T21:22:58.010 DEBUG:teuthology.orchestra.run.smithi003:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/sixmegs'"'"') 2024-05-08T21:22:58.010 DEBUG:teuthology.orchestra.run.smithi003:> except OSError as e: 2024-05-08T21:22:58.011 DEBUG:teuthology.orchestra.run.smithi003:> sys.exit(e.errno) 2024-05-08T21:22:58.011 DEBUG:teuthology.orchestra.run.smithi003:> 2024-05-08T21:22:58.011 DEBUG:teuthology.orchestra.run.smithi003:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-05-08T21:22:58.011 DEBUG:teuthology.orchestra.run.smithi003:> print(json.dumps( 2024-05-08T21:22:58.011 DEBUG:teuthology.orchestra.run.smithi003:> dict([(a, getattr(s, a)) for a in attrs]), 2024-05-08T21:22:58.011 DEBUG:teuthology.orchestra.run.smithi003:> indent=2)) 2024-05-08T21:22:58.011 DEBUG:teuthology.orchestra.run.smithi003:> ' 2024-05-08T21:22:58.092 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:22:58.092 INFO:teuthology.orchestra.run.smithi003.stdout: "st_mode": 33204, 2024-05-08T21:22:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "st_ino": 1099511627777, 2024-05-08T21:22:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "st_dev": 47, 2024-05-08T21:22:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "st_nlink": 1, 2024-05-08T21:22:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "st_uid": 1000, 2024-05-08T21:22:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "st_gid": 1304, 2024-05-08T21:22:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "st_size": 6291456, 2024-05-08T21:22:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "st_atime": 1715203377.9458632, 2024-05-08T21:22:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "st_mtime": 1715203377.9838626, 2024-05-08T21:22:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "st_ctime": 1715203377.9838626 2024-05-08T21:22:58.093 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:22:58.277 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:22:58.277 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:58.339 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:22:58.339 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:22:58.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:22:58.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.585+0000 7ff27d34f700 1 Processor -- start 2024-05-08T21:22:58.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.585+0000 7ff27d34f700 1 -- start start 2024-05-08T21:22:58.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.586+0000 7ff27d34f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff27815db60 0x7ff2780cf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:58.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.586+0000 7ff27d34f700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff278161b80 0x7ff2780d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:58.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.586+0000 7ff27d34f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff27806c9c0 con 0x7ff27815d780 2024-05-08T21:22:58.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.586+0000 7ff27d34f700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff27806cca0 con 0x7ff278161b80 2024-05-08T21:22:58.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.586+0000 7ff27d34f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff27806cf80 con 0x7ff27815db60 2024-05-08T21:22:58.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.586+0000 7ff276ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff27815d780 0x7ff278161470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59816/0 (socket says 172.21.15.3:59816) 2024-05-08T21:22:58.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.586+0000 7ff276ffd700 1 -- 172.21.15.3:0/200018858 learned_addr learned my addr 172.21.15.3:0/200018858 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:58.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff2777fe700 1 --2- 172.21.15.3:0/200018858 >> v2:172.21.15.186:3300/0 conn(0x7ff278161b80 0x7ff2780d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:58.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff2767fc700 1 --2- 172.21.15.3:0/200018858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff27815db60 0x7ff2780cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:58.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff275ffb700 1 -- 172.21.15.3:0/200018858 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026758547 0 0) 0x7ff27806c9c0 con 0x7ff27815d780 2024-05-08T21:22:58.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff275ffb700 1 -- 172.21.15.3:0/200018858 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2600035b0 con 0x7ff27815d780 2024-05-08T21:22:58.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff275ffb700 1 -- 172.21.15.3:0/200018858 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2000195009 0 0) 0x7ff2600035b0 con 0x7ff27815d780 2024-05-08T21:22:58.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff275ffb700 1 -- 172.21.15.3:0/200018858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff27815db60 msgr2=0x7ff2780cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:58.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff275ffb700 1 --2- 172.21.15.3:0/200018858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff27815db60 0x7ff2780cf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:58.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff275ffb700 1 -- 172.21.15.3:0/200018858 >> v2:172.21.15.186:3300/0 conn(0x7ff278161b80 msgr2=0x7ff2780d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:58.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff275ffb700 1 --2- 172.21.15.3:0/200018858 >> v2:172.21.15.186:3300/0 conn(0x7ff278161b80 0x7ff2780d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:58.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff275ffb700 1 -- 172.21.15.3:0/200018858 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27806d260 con 0x7ff27815d780 2024-05-08T21:22:58.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff275ffb700 1 -- 172.21.15.3:0/200018858 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7ff268001670 con 0x7ff27815d780 2024-05-08T21:22:58.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff275ffb700 1 -- 172.21.15.3:0/200018858 >> v1:172.21.15.3:6789/0 conn(0x7ff27815d780 legacy=0x7ff278161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:58.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff275ffb700 1 --2- 172.21.15.3:0/200018858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff27815db60 0x7ff260003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:58.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff275ffb700 1 --2- 172.21.15.3:0/200018858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff260003f80 0x7ff260004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:58.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff275ffb700 1 --2- 172.21.15.3:0/200018858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff260004d90 0x7ff260005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:58.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff275ffb700 1 -- 172.21.15.3:0/200018858 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27806cca0 con 0x7ff260004d90 2024-05-08T21:22:58.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff275ffb700 1 -- 172.21.15.3:0/200018858 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27806cf80 con 0x7ff260003f80 2024-05-08T21:22:58.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff275ffb700 1 -- 172.21.15.3:0/200018858 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26000ba10 con 0x7ff27815db60 2024-05-08T21:22:58.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff2777fe700 1 --2- 172.21.15.3:0/200018858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff260004d90 0x7ff260005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:58.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff2767fc700 1 --2- 172.21.15.3:0/200018858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff260003f80 0x7ff260004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:58.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.587+0000 7ff276ffd700 1 --2- 172.21.15.3:0/200018858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff27815db60 0x7ff260003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:58.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.588+0000 7ff2777fe700 1 -- 172.21.15.3:0/200018858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff27815db60 msgr2=0x7ff260003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:58.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.588+0000 7ff2777fe700 1 --2- 172.21.15.3:0/200018858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff27815db60 0x7ff260003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:58.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.588+0000 7ff2777fe700 1 -- 172.21.15.3:0/200018858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff260003f80 msgr2=0x7ff260004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:58.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.588+0000 7ff2777fe700 1 --2- 172.21.15.3:0/200018858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff260003f80 0x7ff260004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:58.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.588+0000 7ff2777fe700 1 -- 172.21.15.3:0/200018858 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2600035b0 con 0x7ff260004d90 2024-05-08T21:22:58.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.588+0000 7ff2777fe700 1 --2- 172.21.15.3:0/200018858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff260004d90 0x7ff260005210 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff278073070 tx=0x7ff260005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc227e0208828fe server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:58.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.588+0000 7ff275ffb700 1 -- 172.21.15.3:0/200018858 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27000eb70 con 0x7ff260004d90 2024-05-08T21:22:58.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.588+0000 7ff275ffb700 1 -- 172.21.15.3:0/200018858 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27000f040 con 0x7ff260004d90 2024-05-08T21:22:58.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.588+0000 7ff275ffb700 1 -- 172.21.15.3:0/200018858 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff270014480 con 0x7ff260004d90 2024-05-08T21:22:58.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.589+0000 7ff27d34f700 1 -- 172.21.15.3:0/200018858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff260004d90 msgr2=0x7ff260005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:58.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.589+0000 7ff27d34f700 1 --2- 172.21.15.3:0/200018858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff260004d90 0x7ff260005210 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff278073070 tx=0x7ff260005880 comp rx=0 tx=0).stop 2024-05-08T21:22:58.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.589+0000 7ff2777fe700 1 -- 172.21.15.3:0/200018858 reap_dead start 2024-05-08T21:22:58.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.589+0000 7ff27d34f700 1 -- 172.21.15.3:0/200018858 shutdown_connections 2024-05-08T21:22:58.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.589+0000 7ff27d34f700 1 -- 172.21.15.3:0/200018858 >> 172.21.15.3:0/200018858 conn(0x7ff27808afe0 msgr2=0x7ff2780d3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:58.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.589+0000 7ff27d34f700 1 -- 172.21.15.3:0/200018858 shutdown_connections 2024-05-08T21:22:58.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.589+0000 7ff27d34f700 1 -- 172.21.15.3:0/200018858 wait complete. 2024-05-08T21:22:58.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.590+0000 7ff27d34f700 1 Processor -- start 2024-05-08T21:22:58.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.590+0000 7ff27d34f700 1 -- start start 2024-05-08T21:22:58.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff27d34f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff278155410 0x7ff2780cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:58.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff27d34f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2780cc650 0x7ff2780b6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:58.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff27d34f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2780b6ec0 0x7ff2780b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:58.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff27d34f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2781557f0 con 0x7ff2780cc650 2024-05-08T21:22:58.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff27d34f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2780ccad0 con 0x7ff278155410 2024-05-08T21:22:58.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff27d34f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff278152330 con 0x7ff2780b6ec0 2024-05-08T21:22:58.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff2767fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2780cc650 0x7ff2780b6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:58.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff2767fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2780cc650 0x7ff2780b6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45634/0 (socket says 172.21.15.3:45634) 2024-05-08T21:22:58.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff2767fc700 1 -- 172.21.15.3:0/2036363485 learned_addr learned my addr 172.21.15.3:0/2036363485 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:58.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff276ffd700 1 --2- 172.21.15.3:0/2036363485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff278155410 0x7ff2780cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:58.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff2777fe700 1 --2- 172.21.15.3:0/2036363485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2780b6ec0 0x7ff2780b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:58.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff2767fc700 1 -- 172.21.15.3:0/2036363485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2780b6ec0 msgr2=0x7ff2780b4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:58.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff2767fc700 1 --2- 172.21.15.3:0/2036363485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2780b6ec0 0x7ff2780b4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:58.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff2767fc700 1 -- 172.21.15.3:0/2036363485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff278155410 msgr2=0x7ff2780cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff2767fc700 1 --2- 172.21.15.3:0/2036363485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff278155410 0x7ff2780cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff2767fc700 1 -- 172.21.15.3:0/2036363485 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2780b7670 con 0x7ff2780cc650 2024-05-08T21:22:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.591+0000 7ff276ffd700 1 --2- 172.21.15.3:0/2036363485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff278155410 0x7ff2780cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.592+0000 7ff2767fc700 1 --2- 172.21.15.3:0/2036363485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2780cc650 0x7ff2780b6980 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff26c011940 tx=0x7ff26c011d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.592+0000 7ff25f7fe700 1 -- 172.21.15.3:0/2036363485 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c011ee0 con 0x7ff2780cc650 2024-05-08T21:22:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.592+0000 7ff25f7fe700 1 -- 172.21.15.3:0/2036363485 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26c004240 con 0x7ff2780cc650 2024-05-08T21:22:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.592+0000 7ff25f7fe700 1 -- 172.21.15.3:0/2036363485 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c0036a0 con 0x7ff2780cc650 2024-05-08T21:22:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.592+0000 7ff27d34f700 1 -- 172.21.15.3:0/2036363485 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2780b7950 con 0x7ff2780cc650 2024-05-08T21:22:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.592+0000 7ff27d34f700 1 -- 172.21.15.3:0/2036363485 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2780b4a30 con 0x7ff2780cc650 2024-05-08T21:22:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.593+0000 7ff27d34f700 1 -- 172.21.15.3:0/2036363485 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2780b4bf0 con 0x7ff2780cc650 2024-05-08T21:22:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.593+0000 7ff25f7fe700 1 -- 172.21.15.3:0/2036363485 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff26c003800 con 0x7ff2780cc650 2024-05-08T21:22:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.594+0000 7ff25f7fe700 1 --2- 172.21.15.3:0/2036363485 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff26406ac90 0x7ff26406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.594+0000 7ff25f7fe700 1 -- 172.21.15.3:0/2036363485 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff26c089760 con 0x7ff2780cc650 2024-05-08T21:22:58.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.594+0000 7ff276ffd700 1 --2- 172.21.15.3:0/2036363485 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff26406ac90 0x7ff26406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:58.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.595+0000 7ff276ffd700 1 --2- 172.21.15.3:0/2036363485 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff26406ac90 0x7ff26406d150 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff2680048d0 tx=0x7ff268009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:58.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.598+0000 7ff25f7fe700 1 -- 172.21.15.3:0/2036363485 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff26c0549d0 con 0x7ff2780cc650 2024-05-08T21:22:58.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.818+0000 7ff27d34f700 1 -- 172.21.15.3:0/2036363485 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2780678f0 con 0x7ff2780cc650 2024-05-08T21:22:58.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.819+0000 7ff25f7fe700 1 -- 172.21.15.3:0/2036363485 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16053 (secure 0 0 0) 0x7ff26c019090 con 0x7ff2780cc650 2024-05-08T21:22:58.821 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:22:58.821 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":34,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:22:51.186147+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:22:47.853786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-08T21:22:49.153759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:22:58.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.822+0000 7ff27d34f700 1 -- 172.21.15.3:0/2036363485 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff26406ac90 msgr2=0x7ff26406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:58.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.822+0000 7ff27d34f700 1 --2- 172.21.15.3:0/2036363485 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff26406ac90 0x7ff26406d150 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff2680048d0 tx=0x7ff268009040 comp rx=0 tx=0).stop 2024-05-08T21:22:58.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.822+0000 7ff27d34f700 1 -- 172.21.15.3:0/2036363485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2780cc650 msgr2=0x7ff2780b6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:58.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.822+0000 7ff27d34f700 1 --2- 172.21.15.3:0/2036363485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2780cc650 0x7ff2780b6980 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff26c011940 tx=0x7ff26c011d00 comp rx=0 tx=0).stop 2024-05-08T21:22:58.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.822+0000 7ff27d34f700 1 -- 172.21.15.3:0/2036363485 shutdown_connections 2024-05-08T21:22:58.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.822+0000 7ff27d34f700 1 --2- 172.21.15.3:0/2036363485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2780b6ec0 0x7ff2780b4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:58.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.822+0000 7ff27d34f700 1 --2- 172.21.15.3:0/2036363485 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff26406ac90 0x7ff26406d150 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:58.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.822+0000 7ff27d34f700 1 --2- 172.21.15.3:0/2036363485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff278155410 0x7ff2780cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:58.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.822+0000 7ff27d34f700 1 --2- 172.21.15.3:0/2036363485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2780cc650 0x7ff2780b6980 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:58.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.822+0000 7ff27d34f700 1 -- 172.21.15.3:0/2036363485 >> 172.21.15.3:0/2036363485 conn(0x7ff27808afe0 msgr2=0x7ff2780cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:58.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.822+0000 7ff27d34f700 1 -- 172.21.15.3:0/2036363485 shutdown_connections 2024-05-08T21:22:58.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.822+0000 7ff27d34f700 1 -- 172.21.15.3:0/2036363485 wait complete. 2024-05-08T21:22:58.837 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-08T21:22:47.853786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:22:49.153759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:22:58.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:22:58.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.976+0000 7fe55e3b1700 1 Processor -- start 2024-05-08T21:22:58.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.976+0000 7fe55e3b1700 1 -- start start 2024-05-08T21:22:58.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.977+0000 7fe55e3b1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe55806bd60 0x7fe5580cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:58.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.977+0000 7fe55e3b1700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5580b1200 con 0x7fe5580d13a0 2024-05-08T21:22:58.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.977+0000 7fe55e3b1700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5580cfcc0 con 0x7fe5580cdc90 2024-05-08T21:22:58.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.977+0000 7fe55e3b1700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5580ce5c0 con 0x7fe55806bd60 2024-05-08T21:22:58.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.977+0000 7fe557fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe5580d13a0 0x7fe5580d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59828/0 (socket says 172.21.15.3:59828) 2024-05-08T21:22:58.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.977+0000 7fe557fff700 1 -- 172.21.15.3:0/3307743069 learned_addr learned my addr 172.21.15.3:0/3307743069 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:58.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.977+0000 7fe5577fe700 1 --2- 172.21.15.3:0/3307743069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe55806bd60 0x7fe5580cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:58.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.977+0000 7fe556ffd700 1 -- 172.21.15.3:0/3307743069 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776169644 0 0) 0x7fe5580b1200 con 0x7fe5580d13a0 2024-05-08T21:22:58.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.977+0000 7fe556ffd700 1 -- 172.21.15.3:0/3307743069 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5440035b0 con 0x7fe5580d13a0 2024-05-08T21:22:58.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.978+0000 7fe556ffd700 1 -- 172.21.15.3:0/3307743069 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812462951 0 0) 0x7fe5580cfcc0 con 0x7fe5580cdc90 2024-05-08T21:22:58.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.978+0000 7fe5577fe700 1 -- 172.21.15.3:0/3307743069 >> v1:172.21.15.186:6789/0 conn(0x7fe5580cdc90 legacy=0x7fe5580b19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:58.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.978+0000 7fe5577fe700 1 -- 172.21.15.3:0/3307743069 >> v1:172.21.15.3:6789/0 conn(0x7fe5580d13a0 legacy=0x7fe5580d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:58.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.978+0000 7fe5577fe700 1 -- 172.21.15.3:0/3307743069 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5580ce8a0 con 0x7fe55806bd60 2024-05-08T21:22:58.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.978+0000 7fe5577fe700 1 --2- 172.21.15.3:0/3307743069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe55806bd60 0x7fe5580cf780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe540009870 tx=0x7fe540009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=888eb44b7f8517f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:58.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.978+0000 7fe556ffd700 1 -- 172.21.15.3:0/3307743069 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe540011830 con 0x7fe55806bd60 2024-05-08T21:22:58.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.978+0000 7fe556ffd700 1 -- 172.21.15.3:0/3307743069 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe540011990 con 0x7fe55806bd60 2024-05-08T21:22:58.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.978+0000 7fe556ffd700 1 -- 172.21.15.3:0/3307743069 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe540011d00 con 0x7fe55806bd60 2024-05-08T21:22:58.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.979+0000 7fe55e3b1700 1 -- 172.21.15.3:0/3307743069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe55806bd60 msgr2=0x7fe5580cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:58.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.979+0000 7fe55e3b1700 1 --2- 172.21.15.3:0/3307743069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe55806bd60 0x7fe5580cf780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe540009870 tx=0x7fe540009b80 comp rx=0 tx=0).stop 2024-05-08T21:22:58.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.979+0000 7fe55e3b1700 1 -- 172.21.15.3:0/3307743069 shutdown_connections 2024-05-08T21:22:58.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.979+0000 7fe55e3b1700 1 --2- 172.21.15.3:0/3307743069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe55806bd60 0x7fe5580cf780 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:58.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.979+0000 7fe55e3b1700 1 -- 172.21.15.3:0/3307743069 >> 172.21.15.3:0/3307743069 conn(0x7fe55808afe0 msgr2=0x7fe55808b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:58.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.979+0000 7fe55e3b1700 1 -- 172.21.15.3:0/3307743069 shutdown_connections 2024-05-08T21:22:58.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.979+0000 7fe55e3b1700 1 -- 172.21.15.3:0/3307743069 wait complete. 2024-05-08T21:22:58.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.980+0000 7fe55e3b1700 1 Processor -- start 2024-05-08T21:22:58.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.980+0000 7fe55e3b1700 1 -- start start 2024-05-08T21:22:58.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.981+0000 7fe55e3b1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe55806bd60 0x7fe5580afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:58.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.981+0000 7fe55e3b1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5580cdc90 0x7fe5580b6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:58.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.981+0000 7fe55e3b1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5580d13a0 0x7fe5580b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:58.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.981+0000 7fe55e3b1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55815cda0 con 0x7fe5580cdc90 2024-05-08T21:22:58.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.981+0000 7fe55e3b1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5580b5330 con 0x7fe5580d13a0 2024-05-08T21:22:58.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.981+0000 7fe55e3b1700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5580b03f0 con 0x7fe55806bd60 2024-05-08T21:22:58.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.981+0000 7fe5577fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5580cdc90 0x7fe5580b6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:58.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.981+0000 7fe5577fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5580cdc90 0x7fe5580b6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45648/0 (socket says 172.21.15.3:45648) 2024-05-08T21:22:58.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.981+0000 7fe5577fe700 1 -- 172.21.15.3:0/264821365 learned_addr learned my addr 172.21.15.3:0/264821365 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:58.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.981+0000 7fe557fff700 1 --2- 172.21.15.3:0/264821365 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe55806bd60 0x7fe5580afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:58.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.981+0000 7fe55c94e700 1 --2- 172.21.15.3:0/264821365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5580d13a0 0x7fe5580b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:58.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.982+0000 7fe5577fe700 1 -- 172.21.15.3:0/264821365 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe55806bd60 msgr2=0x7fe5580afd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:58.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.982+0000 7fe5577fe700 1 --2- 172.21.15.3:0/264821365 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe55806bd60 0x7fe5580afd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:58.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.982+0000 7fe5577fe700 1 -- 172.21.15.3:0/264821365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5580d13a0 msgr2=0x7fe5580b4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:58.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.982+0000 7fe5577fe700 1 --2- 172.21.15.3:0/264821365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5580d13a0 0x7fe5580b4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:58.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.982+0000 7fe5577fe700 1 -- 172.21.15.3:0/264821365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5400094a0 con 0x7fe5580cdc90 2024-05-08T21:22:58.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.982+0000 7fe5577fe700 1 --2- 172.21.15.3:0/264821365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5580cdc90 0x7fe5580b6850 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe540014040 tx=0x7fe540004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:58.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.982+0000 7fe554ff9700 1 -- 172.21.15.3:0/264821365 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe540004550 con 0x7fe5580cdc90 2024-05-08T21:22:58.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.982+0000 7fe554ff9700 1 -- 172.21.15.3:0/264821365 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54001e070 con 0x7fe5580cdc90 2024-05-08T21:22:58.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.982+0000 7fe55e3b1700 1 -- 172.21.15.3:0/264821365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5580b4030 con 0x7fe5580cdc90 2024-05-08T21:22:58.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.982+0000 7fe554ff9700 1 -- 172.21.15.3:0/264821365 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5400038e0 con 0x7fe5580cdc90 2024-05-08T21:22:58.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.982+0000 7fe55e3b1700 1 -- 172.21.15.3:0/264821365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5580b45f0 con 0x7fe5580cdc90 2024-05-08T21:22:58.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.983+0000 7fe55e3b1700 1 -- 172.21.15.3:0/264821365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe558074bc0 con 0x7fe5580cdc90 2024-05-08T21:22:58.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.984+0000 7fe554ff9700 1 -- 172.21.15.3:0/264821365 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe540003a40 con 0x7fe5580cdc90 2024-05-08T21:22:58.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.984+0000 7fe554ff9700 1 --2- 172.21.15.3:0/264821365 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe53806acc0 0x7fe53806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:58.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.984+0000 7fe554ff9700 1 -- 172.21.15.3:0/264821365 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe54008a100 con 0x7fe5580cdc90 2024-05-08T21:22:58.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.984+0000 7fe557fff700 1 --2- 172.21.15.3:0/264821365 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe53806acc0 0x7fe53806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:58.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.985+0000 7fe557fff700 1 --2- 172.21.15.3:0/264821365 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe53806acc0 0x7fe53806d180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe548007fd0 tx=0x7fe548009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:58.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:58.989+0000 7fe554ff9700 1 -- 172.21.15.3:0/264821365 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe540055370 con 0x7fe5580cdc90 2024-05-08T21:22:59.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.206+0000 7fe55e3b1700 1 -- 172.21.15.3:0/264821365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe5580b7280 con 0x7fe5580cdc90 2024-05-08T21:22:59.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.206+0000 7fe554ff9700 1 -- 172.21.15.3:0/264821365 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe540020020 con 0x7fe5580cdc90 2024-05-08T21:22:59.208 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:22:59.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.208+0000 7fe55e3b1700 1 -- 172.21.15.3:0/264821365 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe53806acc0 msgr2=0x7fe53806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:59.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.208+0000 7fe55e3b1700 1 --2- 172.21.15.3:0/264821365 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe53806acc0 0x7fe53806d180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe548007fd0 tx=0x7fe548009040 comp rx=0 tx=0).stop 2024-05-08T21:22:59.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.208+0000 7fe55e3b1700 1 -- 172.21.15.3:0/264821365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5580cdc90 msgr2=0x7fe5580b6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:59.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.208+0000 7fe55e3b1700 1 --2- 172.21.15.3:0/264821365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5580cdc90 0x7fe5580b6850 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe540014040 tx=0x7fe540004750 comp rx=0 tx=0).stop 2024-05-08T21:22:59.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.208+0000 7fe55e3b1700 1 -- 172.21.15.3:0/264821365 shutdown_connections 2024-05-08T21:22:59.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.208+0000 7fe55e3b1700 1 --2- 172.21.15.3:0/264821365 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe55806bd60 0x7fe5580afd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.208+0000 7fe55e3b1700 1 --2- 172.21.15.3:0/264821365 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe53806acc0 0x7fe53806d180 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.208+0000 7fe55e3b1700 1 --2- 172.21.15.3:0/264821365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5580d13a0 0x7fe5580b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.208+0000 7fe55e3b1700 1 --2- 172.21.15.3:0/264821365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5580cdc90 0x7fe5580b6850 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.208+0000 7fe55e3b1700 1 -- 172.21.15.3:0/264821365 >> 172.21.15.3:0/264821365 conn(0x7fe55808afe0 msgr2=0x7fe5580cf740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:59.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.209+0000 7fe55e3b1700 1 -- 172.21.15.3:0/264821365 shutdown_connections 2024-05-08T21:22:59.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.209+0000 7fe55e3b1700 1 -- 172.21.15.3:0/264821365 wait complete. 2024-05-08T21:22:59.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-08T21:22:59.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.361+0000 7fa529127700 1 Processor -- start 2024-05-08T21:22:59.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.361+0000 7fa529127700 1 -- start start 2024-05-08T21:22:59.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.362+0000 7fa529127700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa5240d13a0 0x7fa5240d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:59.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.362+0000 7fa529127700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa5240cdc90 0x7fa5240b19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:59.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.362+0000 7fa529127700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5240b1200 con 0x7fa52406bd60 2024-05-08T21:22:59.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.362+0000 7fa529127700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5240d1cc0 con 0x7fa5240cdc90 2024-05-08T21:22:59.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.362+0000 7fa529127700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5240ce5c0 con 0x7fa5240d13a0 2024-05-08T21:22:59.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.362+0000 7fa52359e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa5240cdc90 0x7fa5240b19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:59.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.362+0000 7fa52259c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa52406bd60 0x7fa5240cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59838/0 (socket says 172.21.15.3:59838) 2024-05-08T21:22:59.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.362+0000 7fa52259c700 1 -- 172.21.15.3:0/1825516339 learned_addr learned my addr 172.21.15.3:0/1825516339 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:22:59.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.362+0000 7fa522d9d700 1 --2- 172.21.15.3:0/1825516339 >> v2:172.21.15.186:3300/0 conn(0x7fa5240d13a0 0x7fa5240d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:59.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.362+0000 7fa52359e700 1 -- 172.21.15.3:0/1825516339 >> v2:172.21.15.186:3300/0 conn(0x7fa5240d13a0 msgr2=0x7fa5240d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:59.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.362+0000 7fa52359e700 1 --2- 172.21.15.3:0/1825516339 >> v2:172.21.15.186:3300/0 conn(0x7fa5240d13a0 0x7fa5240d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.362+0000 7fa52359e700 1 -- 172.21.15.3:0/1825516339 >> v1:172.21.15.3:6789/0 conn(0x7fa52406bd60 legacy=0x7fa5240cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:59.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.362+0000 7fa52359e700 1 -- 172.21.15.3:0/1825516339 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5240ce8a0 con 0x7fa5240cdc90 2024-05-08T21:22:59.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.362+0000 7fa521d9b700 1 -- 172.21.15.3:0/1825516339 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624245066 0 0) 0x7fa5240b1200 con 0x7fa52406bd60 2024-05-08T21:22:59.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.363+0000 7fa52359e700 1 --2- 172.21.15.3:0/1825516339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5240cdc90 0x7fa5240b19c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa52406e520 tx=0x7fa51c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b812b7c6c99a221 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:59.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.363+0000 7fa521d9b700 1 -- 172.21.15.3:0/1825516339 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c00e070 con 0x7fa5240cdc90 2024-05-08T21:22:59.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.363+0000 7fa521d9b700 1 -- 172.21.15.3:0/1825516339 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51c00bb20 con 0x7fa5240cdc90 2024-05-08T21:22:59.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.363+0000 7fa521d9b700 1 -- 172.21.15.3:0/1825516339 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c012430 con 0x7fa5240cdc90 2024-05-08T21:22:59.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.363+0000 7fa529127700 1 -- 172.21.15.3:0/1825516339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5240cdc90 msgr2=0x7fa5240b19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:59.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.363+0000 7fa529127700 1 --2- 172.21.15.3:0/1825516339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5240cdc90 0x7fa5240b19c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa52406e520 tx=0x7fa51c00b330 comp rx=0 tx=0).stop 2024-05-08T21:22:59.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.363+0000 7fa529127700 1 -- 172.21.15.3:0/1825516339 shutdown_connections 2024-05-08T21:22:59.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.363+0000 7fa529127700 1 --2- 172.21.15.3:0/1825516339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5240cdc90 0x7fa5240b19c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.363+0000 7fa529127700 1 --2- 172.21.15.3:0/1825516339 >> v2:172.21.15.186:3300/0 conn(0x7fa5240d13a0 0x7fa5240d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.363+0000 7fa529127700 1 -- 172.21.15.3:0/1825516339 >> 172.21.15.3:0/1825516339 conn(0x7fa52408afe0 msgr2=0x7fa52408b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:59.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.364+0000 7fa529127700 1 -- 172.21.15.3:0/1825516339 shutdown_connections 2024-05-08T21:22:59.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.364+0000 7fa529127700 1 -- 172.21.15.3:0/1825516339 wait complete. 2024-05-08T21:22:59.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.365+0000 7fa529127700 1 Processor -- start 2024-05-08T21:22:59.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.365+0000 7fa529127700 1 -- start start 2024-05-08T21:22:59.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.366+0000 7fa529127700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa52406bd60 0x7fa5240b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:59.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.366+0000 7fa529127700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5240cdc90 0x7fa52409c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:59.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.366+0000 7fa529127700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5240d13a0 0x7fa5240af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:59.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.366+0000 7fa529127700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5240b6760 con 0x7fa5240cdc90 2024-05-08T21:22:59.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.366+0000 7fa529127700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52409c7b0 con 0x7fa52406bd60 2024-05-08T21:22:59.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.366+0000 7fa529127700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52409ef50 con 0x7fa5240d13a0 2024-05-08T21:22:59.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.366+0000 7fa52259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5240cdc90 0x7fa52409c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:59.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.366+0000 7fa52259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5240cdc90 0x7fa52409c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45658/0 (socket says 172.21.15.3:45658) 2024-05-08T21:22:59.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.366+0000 7fa52259c700 1 -- 172.21.15.3:0/2537263273 learned_addr learned my addr 172.21.15.3:0/2537263273 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:59.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.366+0000 7fa522d9d700 1 --2- 172.21.15.3:0/2537263273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa52406bd60 0x7fa5240b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:59.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.366+0000 7fa52359e700 1 --2- 172.21.15.3:0/2537263273 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5240d13a0 0x7fa5240af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:59.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.366+0000 7fa52259c700 1 -- 172.21.15.3:0/2537263273 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5240d13a0 msgr2=0x7fa5240af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:59.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.366+0000 7fa52259c700 1 --2- 172.21.15.3:0/2537263273 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5240d13a0 0x7fa5240af070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.366+0000 7fa52259c700 1 -- 172.21.15.3:0/2537263273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa52406bd60 msgr2=0x7fa5240b6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:59.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.366+0000 7fa52259c700 1 --2- 172.21.15.3:0/2537263273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa52406bd60 0x7fa5240b6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.367+0000 7fa52259c700 1 -- 172.21.15.3:0/2537263273 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c009a70 con 0x7fa5240cdc90 2024-05-08T21:22:59.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.367+0000 7fa52359e700 1 --2- 172.21.15.3:0/2537263273 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5240d13a0 0x7fa5240af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:22:59.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.367+0000 7fa52259c700 1 --2- 172.21.15.3:0/2537263273 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5240cdc90 0x7fa52409c270 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa51800c550 tx=0x7fa51800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:59.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.367+0000 7fa50b7fe700 1 -- 172.21.15.3:0/2537263273 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa518004c80 con 0x7fa5240cdc90 2024-05-08T21:22:59.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.367+0000 7fa50b7fe700 1 -- 172.21.15.3:0/2537263273 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa518004e00 con 0x7fa5240cdc90 2024-05-08T21:22:59.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.367+0000 7fa50b7fe700 1 -- 172.21.15.3:0/2537263273 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa518007310 con 0x7fa5240cdc90 2024-05-08T21:22:59.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.367+0000 7fa529127700 1 -- 172.21.15.3:0/2537263273 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52409dc10 con 0x7fa5240cdc90 2024-05-08T21:22:59.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.367+0000 7fa529127700 1 -- 172.21.15.3:0/2537263273 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52409e190 con 0x7fa5240cdc90 2024-05-08T21:22:59.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.368+0000 7fa529127700 1 -- 172.21.15.3:0/2537263273 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa524074bc0 con 0x7fa5240cdc90 2024-05-08T21:22:59.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.368+0000 7fa50b7fe700 1 -- 172.21.15.3:0/2537263273 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa518007470 con 0x7fa5240cdc90 2024-05-08T21:22:59.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.369+0000 7fa50b7fe700 1 --2- 172.21.15.3:0/2537263273 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa51006ac90 0x7fa51006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:59.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.369+0000 7fa50b7fe700 1 -- 172.21.15.3:0/2537263273 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa518088ed0 con 0x7fa5240cdc90 2024-05-08T21:22:59.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.369+0000 7fa522d9d700 1 --2- 172.21.15.3:0/2537263273 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa51006ac90 0x7fa51006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:59.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.370+0000 7fa522d9d700 1 --2- 172.21.15.3:0/2537263273 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa51006ac90 0x7fa51006d150 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa514009be0 tx=0x7fa514009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:59.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.374+0000 7fa50b7fe700 1 -- 172.21.15.3:0/2537263273 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fa518054140 con 0x7fa5240cdc90 2024-05-08T21:22:59.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.587+0000 7fa529127700 1 -- 172.21.15.3:0/2537263273 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa52406c260 con 0x7fa5240cdc90 2024-05-08T21:22:59.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.588+0000 7fa50b7fe700 1 -- 172.21.15.3:0/2537263273 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa518057760 con 0x7fa5240cdc90 2024-05-08T21:22:59.590 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:22:59.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.590+0000 7fa529127700 1 -- 172.21.15.3:0/2537263273 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa51006ac90 msgr2=0x7fa51006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.590+0000 7fa529127700 1 --2- 172.21.15.3:0/2537263273 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa51006ac90 0x7fa51006d150 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa514009be0 tx=0x7fa514009210 comp rx=0 tx=0).stop 2024-05-08T21:22:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.590+0000 7fa529127700 1 -- 172.21.15.3:0/2537263273 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5240cdc90 msgr2=0x7fa52409c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.590+0000 7fa529127700 1 --2- 172.21.15.3:0/2537263273 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5240cdc90 0x7fa52409c270 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa51800c550 tx=0x7fa51800c860 comp rx=0 tx=0).stop 2024-05-08T21:22:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.590+0000 7fa529127700 1 -- 172.21.15.3:0/2537263273 shutdown_connections 2024-05-08T21:22:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.590+0000 7fa529127700 1 --2- 172.21.15.3:0/2537263273 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5240d13a0 0x7fa5240af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.590+0000 7fa529127700 1 --2- 172.21.15.3:0/2537263273 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa51006ac90 0x7fa51006d150 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.590+0000 7fa529127700 1 --2- 172.21.15.3:0/2537263273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa52406bd60 0x7fa5240b6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.590+0000 7fa529127700 1 --2- 172.21.15.3:0/2537263273 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5240cdc90 0x7fa52409c270 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.590+0000 7fa529127700 1 -- 172.21.15.3:0/2537263273 >> 172.21.15.3:0/2537263273 conn(0x7fa52408afe0 msgr2=0x7fa5240ca4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.591+0000 7fa529127700 1 -- 172.21.15.3:0/2537263273 shutdown_connections 2024-05-08T21:22:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.591+0000 7fa529127700 1 -- 172.21.15.3:0/2537263273 wait complete. 2024-05-08T21:22:59.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:22:59.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.743+0000 7fc7b0ef4700 1 Processor -- start 2024-05-08T21:22:59.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.743+0000 7fc7b0ef4700 1 -- start start 2024-05-08T21:22:59.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.744+0000 7fc7b0ef4700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7ac0b44e0 0x7fc7ac06b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:59.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.744+0000 7fc7b0ef4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc7ac0b48c0 0x7fc7ac06beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:59.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.744+0000 7fc7b0ef4700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ac15f3c0 con 0x7fc7ac15bf60 2024-05-08T21:22:59.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.744+0000 7fc7b0ef4700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ac15cad0 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.744+0000 7fc7b0ef4700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0a84b0 con 0x7fc7ac0b44e0 2024-05-08T21:22:59.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.745+0000 7fc7a9d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc7ac0b48c0 0x7fc7ac06beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:59.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.745+0000 7fc7a9d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc7ac0b48c0 0x7fc7ac06beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45664/0 (socket says 172.21.15.3:45664) 2024-05-08T21:22:59.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.745+0000 7fc7a9d9b700 1 -- 172.21.15.3:0/3113231469 learned_addr learned my addr 172.21.15.3:0/3113231469 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:59.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.745+0000 7fc7aa59c700 1 --2- 172.21.15.3:0/3113231469 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7ac0b44e0 0x7fc7ac06b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:59.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.745+0000 7fc7a9d9b700 1 -- 172.21.15.3:0/3113231469 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7ac0b44e0 msgr2=0x7fc7ac06b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:59.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.745+0000 7fc7a9d9b700 1 --2- 172.21.15.3:0/3113231469 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7ac0b44e0 0x7fc7ac06b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.745+0000 7fc7a9d9b700 1 -- 172.21.15.3:0/3113231469 >> v1:172.21.15.186:6789/0 conn(0x7fc7ac15bf60 legacy=0x7fc7ac15c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:22:59.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.745+0000 7fc7a9d9b700 1 -- 172.21.15.3:0/3113231469 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac160160 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.745+0000 7fc7a9d9b700 1 --2- 172.21.15.3:0/3113231469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7ac0b48c0 0x7fc7ac06beb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00098f0 tx=0x7fc7a0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd68520b22276fe5 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:59.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.745+0000 7fc7a959a700 1 -- 172.21.15.3:0/3113231469 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0004230 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.745+0000 7fc7a959a700 1 -- 172.21.15.3:0/3113231469 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a0004390 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.745+0000 7fc7a959a700 1 -- 172.21.15.3:0/3113231469 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a00036a0 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.746+0000 7fc7b0ef4700 1 -- 172.21.15.3:0/3113231469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7ac0b48c0 msgr2=0x7fc7ac06beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:59.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.746+0000 7fc7b0ef4700 1 --2- 172.21.15.3:0/3113231469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7ac0b48c0 0x7fc7ac06beb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00098f0 tx=0x7fc7a0009c00 comp rx=0 tx=0).stop 2024-05-08T21:22:59.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.746+0000 7fc7b0ef4700 1 -- 172.21.15.3:0/3113231469 shutdown_connections 2024-05-08T21:22:59.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.746+0000 7fc7b0ef4700 1 --2- 172.21.15.3:0/3113231469 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7ac0b44e0 0x7fc7ac06b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.746+0000 7fc7b0ef4700 1 --2- 172.21.15.3:0/3113231469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7ac0b48c0 0x7fc7ac06beb0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.746+0000 7fc7b0ef4700 1 -- 172.21.15.3:0/3113231469 >> 172.21.15.3:0/3113231469 conn(0x7fc7ac0713c0 msgr2=0x7fc7ac0717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:59.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.746+0000 7fc7b0ef4700 1 -- 172.21.15.3:0/3113231469 shutdown_connections 2024-05-08T21:22:59.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.746+0000 7fc7b0ef4700 1 -- 172.21.15.3:0/3113231469 wait complete. 2024-05-08T21:22:59.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.748+0000 7fc7b0ef4700 1 Processor -- start 2024-05-08T21:22:59.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.748+0000 7fc7b0ef4700 1 -- start start 2024-05-08T21:22:59.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7b0ef4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7ac0b44e0 0x7fc7ac14ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:59.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7b0ef4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7ac0b48c0 0x7fc7ac13d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:59.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7b0ef4700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7ac15bf60 0x7fc7ac14e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:59.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7b0ef4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac14fb90 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7b0ef4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac14ef10 con 0x7fc7ac0b44e0 2024-05-08T21:22:59.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7b0ef4700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac14f1f0 con 0x7fc7ac15bf60 2024-05-08T21:22:59.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7a9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7ac0b48c0 0x7fc7ac13d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:59.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7a9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7ac0b48c0 0x7fc7ac13d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45670/0 (socket says 172.21.15.3:45670) 2024-05-08T21:22:59.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7a9d9b700 1 -- 172.21.15.3:0/3485338234 learned_addr learned my addr 172.21.15.3:0/3485338234 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:22:59.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7aa59c700 1 --2- 172.21.15.3:0/3485338234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7ac0b44e0 0x7fc7ac14ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:59.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7aad9d700 1 --2- 172.21.15.3:0/3485338234 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7ac15bf60 0x7fc7ac14e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:59.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7a9d9b700 1 -- 172.21.15.3:0/3485338234 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7ac15bf60 msgr2=0x7fc7ac14e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:59.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7a9d9b700 1 --2- 172.21.15.3:0/3485338234 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7ac15bf60 0x7fc7ac14e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7a9d9b700 1 -- 172.21.15.3:0/3485338234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7ac0b44e0 msgr2=0x7fc7ac14ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:59.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7a9d9b700 1 --2- 172.21.15.3:0/3485338234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7ac0b44e0 0x7fc7ac14ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7a9d9b700 1 -- 172.21.15.3:0/3485338234 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a0009550 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.749+0000 7fc7aa59c700 1 --2- 172.21.15.3:0/3485338234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7ac0b44e0 0x7fc7ac14ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:22:59.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.750+0000 7fc7a9d9b700 1 --2- 172.21.15.3:0/3485338234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7ac0b48c0 0x7fc7ac13d1d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0009130 tx=0x7fc7a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:59.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.750+0000 7fc79affd700 1 -- 172.21.15.3:0/3485338234 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0003c00 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.750+0000 7fc79affd700 1 -- 172.21.15.3:0/3485338234 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a001f070 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.750+0000 7fc7b0ef4700 1 -- 172.21.15.3:0/3485338234 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ac14f4d0 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.750+0000 7fc79affd700 1 -- 172.21.15.3:0/3485338234 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a000b660 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.750+0000 7fc7b0ef4700 1 -- 172.21.15.3:0/3485338234 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ac1504f0 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.751+0000 7fc7b0ef4700 1 -- 172.21.15.3:0/3485338234 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ac14fcd0 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.752+0000 7fc79affd700 1 -- 172.21.15.3:0/3485338234 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc7a0021030 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.752+0000 7fc79affd700 1 --2- 172.21.15.3:0/3485338234 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc78c06ac90 0x7fc78c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:22:59.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.752+0000 7fc79affd700 1 -- 172.21.15.3:0/3485338234 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7a008a8b0 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.752+0000 7fc7aa59c700 1 --2- 172.21.15.3:0/3485338234 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc78c06ac90 0x7fc78c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:22:59.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.753+0000 7fc7aa59c700 1 --2- 172.21.15.3:0/3485338234 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc78c06ac90 0x7fc78c06d150 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc79c009820 tx=0x7fc79c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:22:59.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.757+0000 7fc79affd700 1 -- 172.21.15.3:0/3485338234 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc7ac14fcd0 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.974+0000 7fc7b0ef4700 1 -- 172.21.15.3:0/3485338234 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc7ac14fcd0 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.975+0000 7fc79affd700 1 -- 172.21.15.3:0/3485338234 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc7ac14fcd0 con 0x7fc7ac0b48c0 2024-05-08T21:22:59.976 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:22:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.977+0000 7fc7b0ef4700 1 -- 172.21.15.3:0/3485338234 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc78c06ac90 msgr2=0x7fc78c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.977+0000 7fc7b0ef4700 1 --2- 172.21.15.3:0/3485338234 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc78c06ac90 0x7fc78c06d150 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc79c009820 tx=0x7fc79c008040 comp rx=0 tx=0).stop 2024-05-08T21:22:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.977+0000 7fc7b0ef4700 1 -- 172.21.15.3:0/3485338234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7ac0b48c0 msgr2=0x7fc7ac13d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:22:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.977+0000 7fc7b0ef4700 1 --2- 172.21.15.3:0/3485338234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7ac0b48c0 0x7fc7ac13d1d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0009130 tx=0x7fc7a0004750 comp rx=0 tx=0).stop 2024-05-08T21:22:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.977+0000 7fc7b0ef4700 1 -- 172.21.15.3:0/3485338234 shutdown_connections 2024-05-08T21:22:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.977+0000 7fc7b0ef4700 1 --2- 172.21.15.3:0/3485338234 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7ac15bf60 0x7fc7ac14e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.977+0000 7fc7b0ef4700 1 --2- 172.21.15.3:0/3485338234 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc78c06ac90 0x7fc78c06d150 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.977+0000 7fc7b0ef4700 1 --2- 172.21.15.3:0/3485338234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7ac0b44e0 0x7fc7ac14ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.977+0000 7fc7b0ef4700 1 --2- 172.21.15.3:0/3485338234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7ac0b48c0 0x7fc7ac13d1d0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:22:59.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.977+0000 7fc7b0ef4700 1 -- 172.21.15.3:0/3485338234 >> 172.21.15.3:0/3485338234 conn(0x7fc7ac0713c0 msgr2=0x7fc7ac15d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:22:59.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.977+0000 7fc7b0ef4700 1 -- 172.21.15.3:0/3485338234 shutdown_connections 2024-05-08T21:22:59.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:22:59.977+0000 7fc7b0ef4700 1 -- 172.21.15.3:0/3485338234 wait complete. 2024-05-08T21:22:59.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-08T21:23:00.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.128+0000 7ff028af2700 1 Processor -- start 2024-05-08T21:23:00.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.128+0000 7ff028af2700 1 -- start start 2024-05-08T21:23:00.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.129+0000 7ff028af2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff02415b900 0x7ff0240cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:00.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.129+0000 7ff028af2700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff024099a50 0x7ff024099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:00.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.129+0000 7ff028af2700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff02409a8d0 con 0x7ff02409a440 2024-05-08T21:23:00.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.129+0000 7ff028af2700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0240b1410 con 0x7ff02415b900 2024-05-08T21:23:00.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.129+0000 7ff028af2700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0240b0e30 con 0x7ff024099a50 2024-05-08T21:23:00.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.129+0000 7ff02259c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff02415b900 0x7ff0240cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:00.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.129+0000 7ff02259c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff02415b900 0x7ff0240cf9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45678/0 (socket says 172.21.15.3:45678) 2024-05-08T21:23:00.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.129+0000 7ff02259c700 1 -- 172.21.15.3:0/3758900140 learned_addr learned my addr 172.21.15.3:0/3758900140 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:00.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.129+0000 7ff021d9b700 1 --2- 172.21.15.3:0/3758900140 >> v2:172.21.15.186:3300/0 conn(0x7ff024099a50 0x7ff024099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:00.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.129+0000 7ff02259c700 1 -- 172.21.15.3:0/3758900140 >> v2:172.21.15.186:3300/0 conn(0x7ff024099a50 msgr2=0x7ff024099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:00.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.129+0000 7ff02259c700 1 --2- 172.21.15.3:0/3758900140 >> v2:172.21.15.186:3300/0 conn(0x7ff024099a50 0x7ff024099f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.130+0000 7ff02259c700 1 -- 172.21.15.3:0/3758900140 >> v1:172.21.15.186:6789/0 conn(0x7ff02409a440 legacy=0x7ff0240cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:00.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.130+0000 7ff02259c700 1 -- 172.21.15.3:0/3758900140 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0240cffd0 con 0x7ff02415b900 2024-05-08T21:23:00.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.130+0000 7ff021d9b700 1 --2- 172.21.15.3:0/3758900140 >> v2:172.21.15.186:3300/0 conn(0x7ff024099a50 0x7ff024099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:00.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.130+0000 7ff02259c700 1 --2- 172.21.15.3:0/3758900140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff02415b900 0x7ff0240cf9e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff0240d0a70 tx=0x7ff014009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba715321e287a4f server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:00.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.130+0000 7ff02159a700 1 -- 172.21.15.3:0/3758900140 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff014004230 con 0x7ff02415b900 2024-05-08T21:23:00.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.130+0000 7ff02159a700 1 -- 172.21.15.3:0/3758900140 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff014004390 con 0x7ff02415b900 2024-05-08T21:23:00.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.130+0000 7ff02159a700 1 -- 172.21.15.3:0/3758900140 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0140036a0 con 0x7ff02415b900 2024-05-08T21:23:00.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.130+0000 7ff028af2700 1 -- 172.21.15.3:0/3758900140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff02415b900 msgr2=0x7ff0240cf9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:00.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.130+0000 7ff028af2700 1 --2- 172.21.15.3:0/3758900140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff02415b900 0x7ff0240cf9e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff0240d0a70 tx=0x7ff014009bd0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.131+0000 7ff028af2700 1 -- 172.21.15.3:0/3758900140 shutdown_connections 2024-05-08T21:23:00.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.131+0000 7ff028af2700 1 --2- 172.21.15.3:0/3758900140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff02415b900 0x7ff0240cf9e0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.131+0000 7ff028af2700 1 --2- 172.21.15.3:0/3758900140 >> v2:172.21.15.186:3300/0 conn(0x7ff024099a50 0x7ff024099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.131+0000 7ff028af2700 1 -- 172.21.15.3:0/3758900140 >> 172.21.15.3:0/3758900140 conn(0x7ff0240b2f20 msgr2=0x7ff0240b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:00.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.131+0000 7ff028af2700 1 -- 172.21.15.3:0/3758900140 shutdown_connections 2024-05-08T21:23:00.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.131+0000 7ff028af2700 1 -- 172.21.15.3:0/3758900140 wait complete. 2024-05-08T21:23:00.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.132+0000 7ff028af2700 1 Processor -- start 2024-05-08T21:23:00.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.133+0000 7ff028af2700 1 -- start start 2024-05-08T21:23:00.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.133+0000 7ff028af2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff024099a50 0x7ff02415e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:00.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff028af2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff02409a440 0x7ff0240b4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:00.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff028af2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff02415b900 0x7ff0240b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:00.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff028af2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240b59c0 con 0x7ff02409a440 2024-05-08T21:23:00.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff028af2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240b4b90 con 0x7ff024099a50 2024-05-08T21:23:00.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff028af2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240b4e40 con 0x7ff02415b900 2024-05-08T21:23:00.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff021d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff02409a440 0x7ff0240b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:00.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff021d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff02409a440 0x7ff0240b4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45694/0 (socket says 172.21.15.3:45694) 2024-05-08T21:23:00.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff021d9b700 1 -- 172.21.15.3:0/2778780181 learned_addr learned my addr 172.21.15.3:0/2778780181 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:00.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff02259c700 1 --2- 172.21.15.3:0/2778780181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff024099a50 0x7ff02415e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:00.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff022d9d700 1 --2- 172.21.15.3:0/2778780181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff02415b900 0x7ff0240b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:00.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff021d9b700 1 -- 172.21.15.3:0/2778780181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff02415b900 msgr2=0x7ff0240b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:00.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff021d9b700 1 --2- 172.21.15.3:0/2778780181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff02415b900 0x7ff0240b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff021d9b700 1 -- 172.21.15.3:0/2778780181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff024099a50 msgr2=0x7ff02415e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:00.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff021d9b700 1 --2- 172.21.15.3:0/2778780181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff024099a50 0x7ff02415e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff021d9b700 1 -- 172.21.15.3:0/2778780181 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff014009550 con 0x7ff02409a440 2024-05-08T21:23:00.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff02259c700 1 --2- 172.21.15.3:0/2778780181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff024099a50 0x7ff02415e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:00.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff021d9b700 1 --2- 172.21.15.3:0/2778780181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff02409a440 0x7ff0240b4110 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff01800c7f0 tx=0x7ff01800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:00.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff012ffd700 1 -- 172.21.15.3:0/2778780181 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff018011800 con 0x7ff02409a440 2024-05-08T21:23:00.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff012ffd700 1 -- 172.21.15.3:0/2778780181 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff018011960 con 0x7ff02409a440 2024-05-08T21:23:00.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff028af2700 1 -- 172.21.15.3:0/2778780181 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0240b5120 con 0x7ff02409a440 2024-05-08T21:23:00.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff012ffd700 1 -- 172.21.15.3:0/2778780181 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff018011d10 con 0x7ff02409a440 2024-05-08T21:23:00.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.134+0000 7ff028af2700 1 -- 172.21.15.3:0/2778780181 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0240b56e0 con 0x7ff02409a440 2024-05-08T21:23:00.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.135+0000 7ff028af2700 1 -- 172.21.15.3:0/2778780181 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff024068000 con 0x7ff02409a440 2024-05-08T21:23:00.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.136+0000 7ff012ffd700 1 -- 172.21.15.3:0/2778780181 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff018018360 con 0x7ff02409a440 2024-05-08T21:23:00.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.136+0000 7ff012ffd700 1 --2- 172.21.15.3:0/2778780181 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff00406ac90 0x7ff00406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:00.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.136+0000 7ff012ffd700 1 -- 172.21.15.3:0/2778780181 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff018089330 con 0x7ff02409a440 2024-05-08T21:23:00.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.137+0000 7ff02259c700 1 --2- 172.21.15.3:0/2778780181 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff00406ac90 0x7ff00406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:00.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.137+0000 7ff02259c700 1 --2- 172.21.15.3:0/2778780181 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff00406ac90 0x7ff00406d150 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff014009130 tx=0x7ff014008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:00.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.141+0000 7ff012ffd700 1 -- 172.21.15.3:0/2778780181 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff018054480 con 0x7ff02409a440 2024-05-08T21:23:00.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.375+0000 7ff028af2700 1 -- 172.21.15.3:0/2778780181 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff0240cde00 con 0x7ff02409a440 2024-05-08T21:23:00.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.375+0000 7ff012ffd700 1 -- 172.21.15.3:0/2778780181 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff018057aa0 con 0x7ff02409a440 2024-05-08T21:23:00.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.378+0000 7ff028af2700 1 -- 172.21.15.3:0/2778780181 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff00406ac90 msgr2=0x7ff00406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:00.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.378+0000 7ff028af2700 1 --2- 172.21.15.3:0/2778780181 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff00406ac90 0x7ff00406d150 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff014009130 tx=0x7ff014008030 comp rx=0 tx=0).stop 2024-05-08T21:23:00.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.378+0000 7ff028af2700 1 -- 172.21.15.3:0/2778780181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff02409a440 msgr2=0x7ff0240b4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:00.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.378+0000 7ff028af2700 1 --2- 172.21.15.3:0/2778780181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff02409a440 0x7ff0240b4110 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff01800c7f0 tx=0x7ff01800cbb0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.378+0000 7ff028af2700 1 -- 172.21.15.3:0/2778780181 shutdown_connections 2024-05-08T21:23:00.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.378+0000 7ff028af2700 1 --2- 172.21.15.3:0/2778780181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff02415b900 0x7ff0240b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.378+0000 7ff028af2700 1 --2- 172.21.15.3:0/2778780181 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff00406ac90 0x7ff00406d150 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.378+0000 7ff028af2700 1 --2- 172.21.15.3:0/2778780181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff024099a50 0x7ff02415e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.378+0000 7ff028af2700 1 --2- 172.21.15.3:0/2778780181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff02409a440 0x7ff0240b4110 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.378+0000 7ff028af2700 1 -- 172.21.15.3:0/2778780181 >> 172.21.15.3:0/2778780181 conn(0x7ff0240b2f20 msgr2=0x7ff0240b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:00.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.378+0000 7ff028af2700 1 -- 172.21.15.3:0/2778780181 shutdown_connections 2024-05-08T21:23:00.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.378+0000 7ff028af2700 1 -- 172.21.15.3:0/2778780181 wait complete. 2024-05-08T21:23:00.381 INFO:teuthology.orchestra.run.smithi003.stderr:listed 0 entries 2024-05-08T21:23:00.393 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-08T21:23:00.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:00.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-08T21:23:00.467 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi003.front.sepia.ceph.com 2024-05-08T21:23:00.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:00.467 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-08T21:23:00.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:23:00.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.680+0000 7f5cea1af700 1 Processor -- start 2024-05-08T21:23:00.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.680+0000 7f5cea1af700 1 -- start start 2024-05-08T21:23:00.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.681+0000 7f5cea1af700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5ce40ccf20 0x7f5ce409a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:00.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.681+0000 7f5cea1af700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce40cbee0 con 0x7f5ce4099a50 2024-05-08T21:23:00.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.681+0000 7f5cea1af700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce415ca20 con 0x7f5ce4099e30 2024-05-08T21:23:00.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.681+0000 7f5cea1af700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce40cc160 con 0x7f5ce40ccf20 2024-05-08T21:23:00.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.681+0000 7f5ce37fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f5ce4099a50 0x7f5ce40cf010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59848/0 (socket says 172.21.15.3:59848) 2024-05-08T21:23:00.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.681+0000 7f5ce37fe700 1 -- 172.21.15.3:0/119462112 learned_addr learned my addr 172.21.15.3:0/119462112 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:00.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.681+0000 7f5ce2ffd700 1 --2- 172.21.15.3:0/119462112 >> v2:172.21.15.186:3300/0 conn(0x7f5ce40ccf20 0x7f5ce409a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:00.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.682+0000 7f5ce27fc700 1 -- 172.21.15.3:0/119462112 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414680822 0 0) 0x7f5ce40cbee0 con 0x7f5ce4099a50 2024-05-08T21:23:00.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.682+0000 7f5ce27fc700 1 -- 172.21.15.3:0/119462112 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ccc0035b0 con 0x7f5ce4099a50 2024-05-08T21:23:00.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.682+0000 7f5ce27fc700 1 -- 172.21.15.3:0/119462112 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764731735 0 0) 0x7f5ce415ca20 con 0x7f5ce4099e30 2024-05-08T21:23:00.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.682+0000 7f5ce27fc700 1 -- 172.21.15.3:0/119462112 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce40cbee0 con 0x7f5ce4099e30 2024-05-08T21:23:00.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.682+0000 7f5ce2ffd700 1 -- 172.21.15.3:0/119462112 >> v1:172.21.15.186:6789/0 conn(0x7f5ce4099e30 legacy=0x7f5ce406ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:00.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.682+0000 7f5ce27fc700 1 -- 172.21.15.3:0/119462112 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2563475033 0 0) 0x7f5ccc0035b0 con 0x7f5ce4099a50 2024-05-08T21:23:00.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.682+0000 7f5ce2ffd700 1 -- 172.21.15.3:0/119462112 >> v1:172.21.15.3:6789/0 conn(0x7f5ce4099a50 legacy=0x7f5ce40cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:00.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.682+0000 7f5ce2ffd700 1 -- 172.21.15.3:0/119462112 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce40cc440 con 0x7f5ce40ccf20 2024-05-08T21:23:00.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.682+0000 7f5ce2ffd700 1 --2- 172.21.15.3:0/119462112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce40ccf20 0x7f5ce409a970 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009840 tx=0x7f5cd8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb2e3edc258c82f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:00.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.682+0000 7f5ce27fc700 1 -- 172.21.15.3:0/119462112 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4001670 con 0x7f5ce40ccf20 2024-05-08T21:23:00.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.682+0000 7f5ce27fc700 1 -- 172.21.15.3:0/119462112 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd8004030 con 0x7f5ce40ccf20 2024-05-08T21:23:00.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.682+0000 7f5ce27fc700 1 -- 172.21.15.3:0/119462112 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd80043c0 con 0x7f5ce40ccf20 2024-05-08T21:23:00.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.683+0000 7f5cea1af700 1 -- 172.21.15.3:0/119462112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce40ccf20 msgr2=0x7f5ce409a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:00.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.683+0000 7f5cea1af700 1 --2- 172.21.15.3:0/119462112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce40ccf20 0x7f5ce409a970 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009840 tx=0x7f5cd8009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:00.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.683+0000 7f5cea1af700 1 -- 172.21.15.3:0/119462112 shutdown_connections 2024-05-08T21:23:00.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.683+0000 7f5cea1af700 1 --2- 172.21.15.3:0/119462112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce40ccf20 0x7f5ce409a970 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.683+0000 7f5cea1af700 1 -- 172.21.15.3:0/119462112 >> 172.21.15.3:0/119462112 conn(0x7f5ce40b2f30 msgr2=0x7f5ce40b3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:00.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.683+0000 7f5cea1af700 1 -- 172.21.15.3:0/119462112 shutdown_connections 2024-05-08T21:23:00.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.683+0000 7f5cea1af700 1 -- 172.21.15.3:0/119462112 wait complete. 2024-05-08T21:23:00.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.685+0000 7f5cea1af700 1 Processor -- start 2024-05-08T21:23:00.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.685+0000 7f5cea1af700 1 -- start start 2024-05-08T21:23:00.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.686+0000 7f5cea1af700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce4099a50 0x7f5ce406d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:00.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.686+0000 7f5cea1af700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce4099e30 0x7f5ce40b1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:00.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.686+0000 7f5cea1af700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ce40ccf20 0x7f5ce40b1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:00.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.686+0000 7f5cea1af700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40c5550 con 0x7f5ce4099e30 2024-05-08T21:23:00.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.686+0000 7f5cea1af700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce41584b0 con 0x7f5ce4099a50 2024-05-08T21:23:00.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.686+0000 7f5cea1af700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40c9fc0 con 0x7f5ce40ccf20 2024-05-08T21:23:00.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.686+0000 7f5ce2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce4099e30 0x7f5ce40b1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:00.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.686+0000 7f5ce2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce4099e30 0x7f5ce40b1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45702/0 (socket says 172.21.15.3:45702) 2024-05-08T21:23:00.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.686+0000 7f5ce2ffd700 1 -- 172.21.15.3:0/250500761 learned_addr learned my addr 172.21.15.3:0/250500761 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:00.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.686+0000 7f5ce37fe700 1 --2- 172.21.15.3:0/250500761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce4099a50 0x7f5ce406d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:00.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.686+0000 7f5ce3fff700 1 --2- 172.21.15.3:0/250500761 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ce40ccf20 0x7f5ce40b1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:00.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.687+0000 7f5ce2ffd700 1 -- 172.21.15.3:0/250500761 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ce40ccf20 msgr2=0x7f5ce40b1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:00.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.687+0000 7f5ce2ffd700 1 --2- 172.21.15.3:0/250500761 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ce40ccf20 0x7f5ce40b1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.687+0000 7f5ce2ffd700 1 -- 172.21.15.3:0/250500761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce4099a50 msgr2=0x7f5ce406d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:00.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.687+0000 7f5ce2ffd700 1 --2- 172.21.15.3:0/250500761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce4099a50 0x7f5ce406d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.687+0000 7f5ce2ffd700 1 -- 172.21.15.3:0/250500761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd80094a0 con 0x7f5ce4099e30 2024-05-08T21:23:00.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.687+0000 7f5ce37fe700 1 --2- 172.21.15.3:0/250500761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce4099a50 0x7f5ce406d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:00.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.687+0000 7f5ce2ffd700 1 --2- 172.21.15.3:0/250500761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce4099e30 0x7f5ce40b1620 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40727e0 tx=0x7f5cd8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:00.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.687+0000 7f5ccbfff700 1 -- 172.21.15.3:0/250500761 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8003d30 con 0x7f5ce4099e30 2024-05-08T21:23:00.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.687+0000 7f5ccbfff700 1 -- 172.21.15.3:0/250500761 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd801f070 con 0x7f5ce4099e30 2024-05-08T21:23:00.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.687+0000 7f5cea1af700 1 -- 172.21.15.3:0/250500761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce40ca210 con 0x7f5ce4099e30 2024-05-08T21:23:00.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.687+0000 7f5ccbfff700 1 -- 172.21.15.3:0/250500761 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8011870 con 0x7f5ce4099e30 2024-05-08T21:23:00.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.687+0000 7f5cea1af700 1 -- 172.21.15.3:0/250500761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce40ca7d0 con 0x7f5ce4099e30 2024-05-08T21:23:00.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.688+0000 7f5cea1af700 1 -- 172.21.15.3:0/250500761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce40c5690 con 0x7f5ce4099e30 2024-05-08T21:23:00.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.688+0000 7f5ccbfff700 1 -- 172.21.15.3:0/250500761 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5cd8021030 con 0x7f5ce4099e30 2024-05-08T21:23:00.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.689+0000 7f5ccbfff700 1 --2- 172.21.15.3:0/250500761 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f5cd006ac90 0x7f5cd006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:00.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.689+0000 7f5ccbfff700 1 -- 172.21.15.3:0/250500761 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cd808a6a0 con 0x7f5ce4099e30 2024-05-08T21:23:00.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.689+0000 7f5ce37fe700 1 --2- 172.21.15.3:0/250500761 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f5cd006ac90 0x7f5cd006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:00.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.690+0000 7f5ce37fe700 1 --2- 172.21.15.3:0/250500761 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f5cd006ac90 0x7f5cd006d150 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4001380 tx=0x7f5cd4001630 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:00.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.694+0000 7f5ccbfff700 1 -- 172.21.15.3:0/250500761 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f5ce40c5690 con 0x7f5ce4099e30 2024-05-08T21:23:00.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.961+0000 7f5cea1af700 1 -- 172.21.15.3:0/250500761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ce40c5690 con 0x7f5ce4099e30 2024-05-08T21:23:00.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.961+0000 7f5ccbfff700 1 -- 172.21.15.3:0/250500761 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4734 (secure 0 0 0) 0x7f5ce40c5690 con 0x7f5ce4099e30 2024-05-08T21:23:00.963 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:00.963 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4757,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/3571929018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3571929018},{"type":"v1","addr":"172.21.15.186:6836","nonce":3571929018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4760,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3520732046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3520732046},{"type":"v1","addr":"172.21.15.186:6837","nonce":3520732046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4957,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2579650460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":2579650460},{"type":"v1","addr":"172.21.15.3:6835","nonce":2579650460}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:22:50.184245+0000","modified":"2024-05-08T21:22:53.192328+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.3:6834/3599481370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":3599481370},{"type":"v1","addr":"172.21.15.3:6834","nonce":3599481370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-05-08T21:23:00.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.963+0000 7f5cea1af700 1 -- 172.21.15.3:0/250500761 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f5cd006ac90 msgr2=0x7f5cd006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:00.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.963+0000 7f5cea1af700 1 --2- 172.21.15.3:0/250500761 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f5cd006ac90 0x7f5cd006d150 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4001380 tx=0x7f5cd4001630 comp rx=0 tx=0).stop 2024-05-08T21:23:00.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.964+0000 7f5cea1af700 1 -- 172.21.15.3:0/250500761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce4099e30 msgr2=0x7f5ce40b1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:00.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.964+0000 7f5cea1af700 1 --2- 172.21.15.3:0/250500761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce4099e30 0x7f5ce40b1620 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40727e0 tx=0x7f5cd8004750 comp rx=0 tx=0).stop 2024-05-08T21:23:00.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.964+0000 7f5cea1af700 1 -- 172.21.15.3:0/250500761 shutdown_connections 2024-05-08T21:23:00.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.964+0000 7f5cea1af700 1 --2- 172.21.15.3:0/250500761 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ce40ccf20 0x7f5ce40b1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.964+0000 7f5cea1af700 1 --2- 172.21.15.3:0/250500761 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f5cd006ac90 0x7f5cd006d150 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.964+0000 7f5cea1af700 1 --2- 172.21.15.3:0/250500761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce4099a50 0x7f5ce406d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.964+0000 7f5cea1af700 1 --2- 172.21.15.3:0/250500761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce4099e30 0x7f5ce40b1620 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:00.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.964+0000 7f5cea1af700 1 -- 172.21.15.3:0/250500761 >> 172.21.15.3:0/250500761 conn(0x7f5ce40b2f30 msgr2=0x7f5ce40cd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:00.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.964+0000 7f5cea1af700 1 -- 172.21.15.3:0/250500761 shutdown_connections 2024-05-08T21:23:00.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:00.964+0000 7f5cea1af700 1 -- 172.21.15.3:0/250500761 wait complete. 2024-05-08T21:23:00.967 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 16 2024-05-08T21:23:00.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-05-08T21:23:01.148 INFO:teuthology.orchestra.run.smithi003.stdout:{"message":"","return_code":0} 2024-05-08T21:23:01.149 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-05-08T21:23:01.149 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-08T21:23:01.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.284+0000 7f410c3af700 1 Processor -- start 2024-05-08T21:23:01.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.284+0000 7f410c3af700 1 -- start start 2024-05-08T21:23:01.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.285+0000 7f410c3af700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41040c90e0 0x7f41041612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.285+0000 7f410c3af700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41040c94c0 0x7f410406b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.285+0000 7f410c3af700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4104146a10 0x7f410406b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.285+0000 7f410c3af700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f410406a960 con 0x7f4104146a10 2024-05-08T21:23:01.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.285+0000 7f410c3af700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4104161800 con 0x7f41040c90e0 2024-05-08T21:23:01.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.285+0000 7f410c3af700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4104161a80 con 0x7f41040c94c0 2024-05-08T21:23:01.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.286+0000 7f410a94c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4104146a10 0x7f410406b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.286+0000 7f410a94c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4104146a10 0x7f410406b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45712/0 (socket says 172.21.15.3:45712) 2024-05-08T21:23:01.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.286+0000 7f410a94c700 1 -- 172.21.15.3:0/2959831104 learned_addr learned my addr 172.21.15.3:0/2959831104 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.286+0000 7f410a14b700 1 --2- 172.21.15.3:0/2959831104 >> v2:172.21.15.186:3300/0 conn(0x7f41040c90e0 0x7f41041612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.286+0000 7f410994a700 1 --2- 172.21.15.3:0/2959831104 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41040c94c0 0x7f410406b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.286+0000 7f410a94c700 1 -- 172.21.15.3:0/2959831104 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41040c94c0 msgr2=0x7f410406b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.286+0000 7f410a94c700 1 --2- 172.21.15.3:0/2959831104 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41040c94c0 0x7f410406b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.286+0000 7f410a94c700 1 -- 172.21.15.3:0/2959831104 >> v2:172.21.15.186:3300/0 conn(0x7f41040c90e0 msgr2=0x7f41041612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.286+0000 7f410a94c700 1 --2- 172.21.15.3:0/2959831104 >> v2:172.21.15.186:3300/0 conn(0x7f41040c90e0 0x7f41041612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.286+0000 7f410a94c700 1 -- 172.21.15.3:0/2959831104 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4104161d60 con 0x7f4104146a10 2024-05-08T21:23:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.286+0000 7f410994a700 1 --2- 172.21.15.3:0/2959831104 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41040c94c0 0x7f410406b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.286+0000 7f410a14b700 1 --2- 172.21.15.3:0/2959831104 >> v2:172.21.15.186:3300/0 conn(0x7f41040c90e0 0x7f41041612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.286+0000 7f410a94c700 1 --2- 172.21.15.3:0/2959831104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4104146a10 0x7f410406b5d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f41040cc4e0 tx=0x7f40fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a21938ebc393dfb server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:01.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.286+0000 7f4109149700 1 -- 172.21.15.3:0/2959831104 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc00e070 con 0x7f4104146a10 2024-05-08T21:23:01.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.286+0000 7f4109149700 1 -- 172.21.15.3:0/2959831104 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc00b9c0 con 0x7f4104146a10 2024-05-08T21:23:01.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.286+0000 7f4109149700 1 -- 172.21.15.3:0/2959831104 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc00bd70 con 0x7f4104146a10 2024-05-08T21:23:01.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.287+0000 7f410c3af700 1 -- 172.21.15.3:0/2959831104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4104146a10 msgr2=0x7f410406b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.287+0000 7f410c3af700 1 --2- 172.21.15.3:0/2959831104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4104146a10 0x7f410406b5d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f41040cc4e0 tx=0x7f40fc00b330 comp rx=0 tx=0).stop 2024-05-08T21:23:01.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.287+0000 7f410c3af700 1 -- 172.21.15.3:0/2959831104 shutdown_connections 2024-05-08T21:23:01.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.287+0000 7f410c3af700 1 --2- 172.21.15.3:0/2959831104 >> v2:172.21.15.186:3300/0 conn(0x7f41040c90e0 0x7f41041612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.287+0000 7f410c3af700 1 --2- 172.21.15.3:0/2959831104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4104146a10 0x7f410406b5d0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.287+0000 7f410c3af700 1 --2- 172.21.15.3:0/2959831104 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41040c94c0 0x7f410406b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.287+0000 7f410c3af700 1 -- 172.21.15.3:0/2959831104 >> 172.21.15.3:0/2959831104 conn(0x7f41040cc580 msgr2=0x7f41040cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:01.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.287+0000 7f410c3af700 1 -- 172.21.15.3:0/2959831104 shutdown_connections 2024-05-08T21:23:01.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.287+0000 7f410c3af700 1 -- 172.21.15.3:0/2959831104 wait complete. 2024-05-08T21:23:01.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.289+0000 7f410c3af700 1 Processor -- start 2024-05-08T21:23:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.289+0000 7f410c3af700 1 -- start start 2024-05-08T21:23:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410c3af700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41040c90e0 0x7f4104160840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410c3af700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41040c94c0 0x7f410413e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410c3af700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4104146a10 0x7f410413e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410c3af700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410414cd00 con 0x7f41040c90e0 2024-05-08T21:23:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410c3af700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410414cfa0 con 0x7f4104146a10 2024-05-08T21:23:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410c3af700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f410414d280 con 0x7f41040c94c0 2024-05-08T21:23:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410a14b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41040c90e0 0x7f4104160840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410a14b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41040c90e0 0x7f4104160840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45726/0 (socket says 172.21.15.3:45726) 2024-05-08T21:23:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410a14b700 1 -- 172.21.15.3:0/4061140141 learned_addr learned my addr 172.21.15.3:0/4061140141 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410a94c700 1 --2- 172.21.15.3:0/4061140141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4104146a10 0x7f410413e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410994a700 1 --2- 172.21.15.3:0/4061140141 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41040c94c0 0x7f410413e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410a14b700 1 -- 172.21.15.3:0/4061140141 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41040c94c0 msgr2=0x7f410413e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410a14b700 1 --2- 172.21.15.3:0/4061140141 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41040c94c0 0x7f410413e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410a14b700 1 -- 172.21.15.3:0/4061140141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4104146a10 msgr2=0x7f410413e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410a14b700 1 --2- 172.21.15.3:0/4061140141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4104146a10 0x7f410413e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410a14b700 1 -- 172.21.15.3:0/4061140141 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40fc009b30 con 0x7f41040c90e0 2024-05-08T21:23:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.290+0000 7f410a94c700 1 --2- 172.21.15.3:0/4061140141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4104146a10 0x7f410413e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.291+0000 7f410a14b700 1 --2- 172.21.15.3:0/4061140141 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41040c90e0 0x7f4104160840 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f40f800d800 tx=0x7f40f800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.291+0000 7f40f6ffd700 1 -- 172.21.15.3:0/4061140141 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f8004030 con 0x7f41040c90e0 2024-05-08T21:23:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.291+0000 7f40f6ffd700 1 -- 172.21.15.3:0/4061140141 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f8004190 con 0x7f41040c90e0 2024-05-08T21:23:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.291+0000 7f410c3af700 1 -- 172.21.15.3:0/4061140141 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410414d560 con 0x7f41040c90e0 2024-05-08T21:23:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.291+0000 7f40f6ffd700 1 -- 172.21.15.3:0/4061140141 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f8004520 con 0x7f41040c90e0 2024-05-08T21:23:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.291+0000 7f410c3af700 1 -- 172.21.15.3:0/4061140141 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f410415dea0 con 0x7f41040c90e0 2024-05-08T21:23:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.292+0000 7f410c3af700 1 -- 172.21.15.3:0/4061140141 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f410414d8d0 con 0x7f41040c90e0 2024-05-08T21:23:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.292+0000 7f40f6ffd700 1 -- 172.21.15.3:0/4061140141 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f40f80036a0 con 0x7f41040c90e0 2024-05-08T21:23:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.293+0000 7f40f6ffd700 1 --2- 172.21.15.3:0/4061140141 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f40e806ac90 0x7f40e806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.293+0000 7f40f6ffd700 1 -- 172.21.15.3:0/4061140141 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40f80898d0 con 0x7f41040c90e0 2024-05-08T21:23:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.293+0000 7f410994a700 1 --2- 172.21.15.3:0/4061140141 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f40e806ac90 0x7f40e806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.293+0000 7f410994a700 1 --2- 172.21.15.3:0/4061140141 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f40e806ac90 0x7f40e806d150 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f410414cb30 tx=0x7f4100008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.297+0000 7f40f6ffd700 1 -- 172.21.15.3:0/4061140141 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f40f8054ab0 con 0x7f41040c90e0 2024-05-08T21:23:01.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.562+0000 7f410c3af700 1 -- 172.21.15.3:0/4061140141 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4104068000 con 0x7f41040c90e0 2024-05-08T21:23:01.564 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-08T21:23:01.563+0000 7f74f7652700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-08T21:23:01.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.570+0000 7f40f6ffd700 1 -- 172.21.15.3:0/4061140141 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) v1 ==== 161+0+0 (secure 0 0 0) 0x7f40f80580d0 con 0x7f41040c90e0 2024-05-08T21:23:01.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.574+0000 7f410c3af700 1 -- 172.21.15.3:0/4061140141 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f40e806ac90 msgr2=0x7f40e806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.574+0000 7f410c3af700 1 --2- 172.21.15.3:0/4061140141 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f40e806ac90 0x7f40e806d150 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f410414cb30 tx=0x7f4100008040 comp rx=0 tx=0).stop 2024-05-08T21:23:01.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.574+0000 7f410c3af700 1 -- 172.21.15.3:0/4061140141 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41040c90e0 msgr2=0x7f4104160840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.574+0000 7f410c3af700 1 --2- 172.21.15.3:0/4061140141 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41040c90e0 0x7f4104160840 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f40f800d800 tx=0x7f40f800dbc0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.574+0000 7f410c3af700 1 -- 172.21.15.3:0/4061140141 shutdown_connections 2024-05-08T21:23:01.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.574+0000 7f410c3af700 1 --2- 172.21.15.3:0/4061140141 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f40e806ac90 0x7f40e806d150 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.574+0000 7f410c3af700 1 --2- 172.21.15.3:0/4061140141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4104146a10 0x7f410413e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.574+0000 7f410c3af700 1 --2- 172.21.15.3:0/4061140141 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41040c90e0 0x7f4104160840 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.574+0000 7f410c3af700 1 --2- 172.21.15.3:0/4061140141 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41040c94c0 0x7f410413e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.574+0000 7f410c3af700 1 -- 172.21.15.3:0/4061140141 >> 172.21.15.3:0/4061140141 conn(0x7f41040cc580 msgr2=0x7f41040af920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:01.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.574+0000 7f410c3af700 1 -- 172.21.15.3:0/4061140141 shutdown_connections 2024-05-08T21:23:01.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.574+0000 7f410c3af700 1 -- 172.21.15.3:0/4061140141 wait complete. 2024-05-08T21:23:01.577 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-08T21:23:01.577 INFO:tasks.ceph.mds.a.smithi003.stderr: -5448> 2024-05-08T21:22:52.704+0000 7ff476db9700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-08T21:23:01.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-08T21:23:01.596 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-08T21:23:01.617 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2024-05-08T21:23:01.648 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2024-05-08T21:23:01.648 INFO:teuthology.orchestra.run.smithi003.stdout: chdir = "" 2024-05-08T21:23:01.648 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-08T21:23:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: auth supported = cephx 2024-05-08T21:23:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: filestore xattr use omap = true 2024-05-08T21:23:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: mon clock drift allowed = 1.000 2024-05-08T21:23:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: osd crush chooseleaf type = 0 2024-05-08T21:23:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: auth debug = true 2024-05-08T21:23:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on old message = true 2024-05-08T21:23:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on bug = true 2024-05-08T21:23:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-08T21:23:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: mon pg warn max object skew = 0 2024-05-08T21:23:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: # disable pg_autoscaler by default for new pools 2024-05-08T21:23:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-08T21:23:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default size = 2 2024-05-08T21:23:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow primary affinity = true 2024-05-08T21:23:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow pg remap = true 2024-05-08T21:23:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on legacy crush tunables = false 2024-05-08T21:23:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on crush straw calc version zero = false 2024-05-08T21:23:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on no sortbitwise = false 2024-05-08T21:23:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on osd down out interval zero = false 2024-05-08T21:23:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on too few osds = false 2024-05-08T21:23:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-08T21:23:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-08T21:23:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: mon_allow_pool_size_one = true 2024-05-08T21:23:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-08T21:23:01.651 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.651 INFO:teuthology.orchestra.run.smithi003.stdout: osd default data pool replay window = 5 2024-05-08T21:23:01.651 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.651 INFO:teuthology.orchestra.run.smithi003.stdout: mon allow pool delete = true 2024-05-08T21:23:01.651 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.651 INFO:teuthology.orchestra.run.smithi003.stdout: mon cluster log file level = debug 2024-05-08T21:23:01.651 INFO:teuthology.orchestra.run.smithi003.stdout: debug asserts on shutdown = true 2024-05-08T21:23:01.651 INFO:teuthology.orchestra.run.smithi003.stdout: mon health detail to clog = false 2024-05-08T21:23:01.651 INFO:teuthology.orchestra.run.smithi003.stdout: mon host = "172.21.15.3,172.21.15.186,[v2:172.21.15.186:3301,v1:172.21.15.186:6790]" 2024-05-08T21:23:01.651 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on skipped message = False 2024-05-08T21:23:01.651 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = d0f58275-758f-4470-b003-84d6abe25489 2024-05-08T21:23:01.651 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.651 INFO:teuthology.orchestra.run.smithi003.stdout:[osd] 2024-05-08T21:23:01.651 INFO:teuthology.orchestra.run.smithi003.stdout: osd journal size = 100 2024-05-08T21:23:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub load threshold = 5.0 2024-05-08T21:23:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub max interval = 600 2024-05-08T21:23:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: osd mclock profile = high_recovery_ops 2024-05-08T21:23:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: osd recover clone overlap = true 2024-05-08T21:23:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: osd recovery max chunk = 1048576 2024-05-08T21:23:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug shutdown = true 2024-05-08T21:23:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug op order = true 2024-05-08T21:23:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify stray on activate = true 2024-05-08T21:23:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: osd open classes on start = true 2024-05-08T21:23:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug pg log writeout = true 2024-05-08T21:23:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: osd deep scrub update digest min age = 30 2024-05-08T21:23:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: osd map max advance = 10 2024-05-08T21:23:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: journal zero on create = true 2024-05-08T21:23:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: filestore ondisk finisher threads = 3 2024-05-08T21:23:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: filestore apply finisher threads = 3 2024-05-08T21:23:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: bdev debug aio = true 2024-05-08T21:23:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug misdirected ops = true 2024-05-08T21:23:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: bdev async discard = True 2024-05-08T21:23:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: bdev enable discard = True 2024-05-08T21:23:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore allocator = bitmap 2024-05-08T21:23:01.654 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore block size = 96636764160 2024-05-08T21:23:01.654 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore fsck on mount = True 2024-05-08T21:23:01.654 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluefs = 1/20 2024-05-08T21:23:01.654 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluestore = 1/20 2024-05-08T21:23:01.654 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-08T21:23:01.654 INFO:teuthology.orchestra.run.smithi003.stdout: debug osd = 20 2024-05-08T21:23:01.654 INFO:teuthology.orchestra.run.smithi003.stdout: debug rocksdb = 4/10 2024-05-08T21:23:01.654 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd backfillfull_ratio = 0.85 2024-05-08T21:23:01.654 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd full ratio = 0.9 2024-05-08T21:23:01.654 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd nearfull ratio = 0.8 2024-05-08T21:23:01.654 INFO:teuthology.orchestra.run.smithi003.stdout: osd failsafe full ratio = 0.95 2024-05-08T21:23:01.654 INFO:teuthology.orchestra.run.smithi003.stdout: osd objectstore = bluestore 2024-05-08T21:23:01.654 INFO:teuthology.orchestra.run.smithi003.stdout: osd op complaint time = 180 2024-05-08T21:23:01.655 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.655 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr] 2024-05-08T21:23:01.655 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-08T21:23:01.655 INFO:teuthology.orchestra.run.smithi003.stdout: debug mgr = 20 2024-05-08T21:23:01.655 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2024-05-08T21:23:01.655 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2024-05-08T21:23:01.655 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2024-05-08T21:23:01.655 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2024-05-08T21:23:01.655 INFO:teuthology.orchestra.run.smithi003.stdout: mgr/telemetry/nag = false 2024-05-08T21:23:01.655 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.655 INFO:teuthology.orchestra.run.smithi003.stdout:[mon] 2024-05-08T21:23:01.655 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-08T21:23:01.655 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2024-05-08T21:23:01.656 INFO:teuthology.orchestra.run.smithi003.stdout: debug paxos = 20 2024-05-08T21:23:01.656 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2024-05-08T21:23:01.656 INFO:teuthology.orchestra.run.smithi003.stdout: mon data avail warn = 5 2024-05-08T21:23:01.656 INFO:teuthology.orchestra.run.smithi003.stdout: mon mgr mkfs grace = 240 2024-05-08T21:23:01.656 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2024-05-08T21:23:01.656 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd reporter subtree level = osd 2024-05-08T21:23:01.656 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd prime pg temp = true 2024-05-08T21:23:01.656 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2024-05-08T21:23:01.656 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.656 INFO:teuthology.orchestra.run.smithi003.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-08T21:23:01.656 INFO:teuthology.orchestra.run.smithi003.stdout: auth mon ticket ttl = 660 # 11m 2024-05-08T21:23:01.656 INFO:teuthology.orchestra.run.smithi003.stdout: auth service ticket ttl = 240 # 4m 2024-05-08T21:23:01.656 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.656 INFO:teuthology.orchestra.run.smithi003.stdout: # don't complain about insecure global_id in the test suite 2024-05-08T21:23:01.657 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-08T21:23:01.657 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-08T21:23:01.657 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.657 INFO:teuthology.orchestra.run.smithi003.stdout: # 1m isn't quite enough 2024-05-08T21:23:01.657 INFO:teuthology.orchestra.run.smithi003.stdout: mon_down_mkfs_grace = 2m 2024-05-08T21:23:01.657 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.657 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_filestore_osds = false 2024-05-08T21:23:01.657 INFO:teuthology.orchestra.run.smithi003.stdout: mon op complaint time = 120 2024-05-08T21:23:01.657 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.657 INFO:teuthology.orchestra.run.smithi003.stdout:[client] 2024-05-08T21:23:01.657 INFO:teuthology.orchestra.run.smithi003.stdout: rgw cache enabled = true 2024-05-08T21:23:01.657 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable ops log = true 2024-05-08T21:23:01.657 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable usage log = true 2024-05-08T21:23:01.658 INFO:teuthology.orchestra.run.smithi003.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-08T21:23:01.658 INFO:teuthology.orchestra.run.smithi003.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-08T21:23:01.658 INFO:teuthology.orchestra.run.smithi003.stdout: client mount timeout = 600 2024-05-08T21:23:01.658 INFO:teuthology.orchestra.run.smithi003.stdout: debug client = 20 2024-05-08T21:23:01.658 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-08T21:23:01.658 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = "" 2024-05-08T21:23:01.658 INFO:teuthology.orchestra.run.smithi003.stdout: rados mon op timeout = 900 2024-05-08T21:23:01.658 INFO:teuthology.orchestra.run.smithi003.stdout: rados osd op timeout = 900 2024-05-08T21:23:01.658 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.a] 2024-05-08T21:23:01.658 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.b] 2024-05-08T21:23:01.658 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.c] 2024-05-08T21:23:01.659 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.a] 2024-05-08T21:23:01.659 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.c] 2024-05-08T21:23:01.659 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.b] 2024-05-08T21:23:01.659 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.d] 2024-05-08T21:23:01.659 INFO:teuthology.orchestra.run.smithi003.stdout:[mds] 2024-05-08T21:23:01.659 INFO:teuthology.orchestra.run.smithi003.stdout: debug mds = 20 2024-05-08T21:23:01.659 INFO:teuthology.orchestra.run.smithi003.stdout: debug mds balancer = 20 2024-05-08T21:23:01.659 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-08T21:23:01.659 INFO:teuthology.orchestra.run.smithi003.stdout: mds debug frag = True 2024-05-08T21:23:01.659 INFO:teuthology.orchestra.run.smithi003.stdout: mds debug scatterstat = True 2024-05-08T21:23:01.659 INFO:teuthology.orchestra.run.smithi003.stdout: mds op complaint time = 180 2024-05-08T21:23:01.659 INFO:teuthology.orchestra.run.smithi003.stdout: mds verify scatter = False 2024-05-08T21:23:01.659 INFO:teuthology.orchestra.run.smithi003.stdout: osd op complaint time = 180 2024-05-08T21:23:01.659 INFO:teuthology.orchestra.run.smithi003.stdout: rados mon op timeout = 900 2024-05-08T21:23:01.659 INFO:teuthology.orchestra.run.smithi003.stdout: rados osd op timeout = 900 2024-05-08T21:23:01.660 INFO:teuthology.orchestra.run.smithi186.stdout:[global] 2024-05-08T21:23:01.660 INFO:teuthology.orchestra.run.smithi186.stdout: chdir = "" 2024-05-08T21:23:01.660 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-08T21:23:01.660 INFO:teuthology.orchestra.run.smithi186.stdout: auth supported = cephx 2024-05-08T21:23:01.660 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.660 INFO:teuthology.orchestra.run.smithi186.stdout: filestore xattr use omap = true 2024-05-08T21:23:01.660 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.660 INFO:teuthology.orchestra.run.smithi186.stdout: mon clock drift allowed = 1.000 2024-05-08T21:23:01.660 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.661 INFO:teuthology.orchestra.run.smithi186.stdout: osd crush chooseleaf type = 0 2024-05-08T21:23:01.661 INFO:teuthology.orchestra.run.smithi186.stdout: auth debug = true 2024-05-08T21:23:01.661 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.661 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on old message = true 2024-05-08T21:23:01.661 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on bug = true 2024-05-08T21:23:01.661 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.661 INFO:teuthology.orchestra.run.smithi186.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-08T21:23:01.661 INFO:teuthology.orchestra.run.smithi186.stdout: mon pg warn max object skew = 0 2024-05-08T21:23:01.661 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.661 INFO:teuthology.orchestra.run.smithi186.stdout: # disable pg_autoscaler by default for new pools 2024-05-08T21:23:01.661 INFO:teuthology.orchestra.run.smithi186.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-08T21:23:01.661 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.661 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default size = 2 2024-05-08T21:23:01.661 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.662 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow primary affinity = true 2024-05-08T21:23:01.662 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow pg remap = true 2024-05-08T21:23:01.662 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on legacy crush tunables = false 2024-05-08T21:23:01.662 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on crush straw calc version zero = false 2024-05-08T21:23:01.662 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on no sortbitwise = false 2024-05-08T21:23:01.662 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on osd down out interval zero = false 2024-05-08T21:23:01.662 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on too few osds = false 2024-05-08T21:23:01.662 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-08T21:23:01.662 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-08T21:23:01.662 INFO:teuthology.orchestra.run.smithi186.stdout: mon_allow_pool_size_one = true 2024-05-08T21:23:01.662 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.662 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-08T21:23:01.662 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.662 INFO:teuthology.orchestra.run.smithi186.stdout: osd default data pool replay window = 5 2024-05-08T21:23:01.662 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.662 INFO:teuthology.orchestra.run.smithi186.stdout: mon allow pool delete = true 2024-05-08T21:23:01.663 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.663 INFO:teuthology.orchestra.run.smithi186.stdout: mon cluster log file level = debug 2024-05-08T21:23:01.663 INFO:teuthology.orchestra.run.smithi186.stdout: debug asserts on shutdown = true 2024-05-08T21:23:01.663 INFO:teuthology.orchestra.run.smithi186.stdout: mon health detail to clog = false 2024-05-08T21:23:01.663 INFO:teuthology.orchestra.run.smithi186.stdout: mon host = "172.21.15.3,172.21.15.186,[v2:172.21.15.186:3301,v1:172.21.15.186:6790]" 2024-05-08T21:23:01.663 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on skipped message = False 2024-05-08T21:23:01.663 INFO:teuthology.orchestra.run.smithi186.stdout: fsid = d0f58275-758f-4470-b003-84d6abe25489 2024-05-08T21:23:01.663 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.663 INFO:teuthology.orchestra.run.smithi186.stdout:[osd] 2024-05-08T21:23:01.663 INFO:teuthology.orchestra.run.smithi186.stdout: osd journal size = 100 2024-05-08T21:23:01.663 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.663 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub load threshold = 5.0 2024-05-08T21:23:01.663 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub max interval = 600 2024-05-08T21:23:01.663 INFO:teuthology.orchestra.run.smithi186.stdout: osd mclock profile = high_recovery_ops 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: osd recover clone overlap = true 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: osd recovery max chunk = 1048576 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug shutdown = true 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug op order = true 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify stray on activate = true 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: osd open classes on start = true 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug pg log writeout = true 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: osd deep scrub update digest min age = 30 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: osd map max advance = 10 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: journal zero on create = true 2024-05-08T21:23:01.664 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.665 INFO:teuthology.orchestra.run.smithi186.stdout: filestore ondisk finisher threads = 3 2024-05-08T21:23:01.665 INFO:teuthology.orchestra.run.smithi186.stdout: filestore apply finisher threads = 3 2024-05-08T21:23:01.665 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.665 INFO:teuthology.orchestra.run.smithi186.stdout: bdev debug aio = true 2024-05-08T21:23:01.665 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug misdirected ops = true 2024-05-08T21:23:01.665 INFO:teuthology.orchestra.run.smithi186.stdout: bdev async discard = True 2024-05-08T21:23:01.665 INFO:teuthology.orchestra.run.smithi186.stdout: bdev enable discard = True 2024-05-08T21:23:01.665 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore allocator = bitmap 2024-05-08T21:23:01.665 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore block size = 96636764160 2024-05-08T21:23:01.665 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore fsck on mount = True 2024-05-08T21:23:01.665 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluefs = 1/20 2024-05-08T21:23:01.665 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluestore = 1/20 2024-05-08T21:23:01.665 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-08T21:23:01.665 INFO:teuthology.orchestra.run.smithi186.stdout: debug osd = 20 2024-05-08T21:23:01.665 INFO:teuthology.orchestra.run.smithi186.stdout: debug rocksdb = 4/10 2024-05-08T21:23:01.666 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd backfillfull_ratio = 0.85 2024-05-08T21:23:01.666 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd full ratio = 0.9 2024-05-08T21:23:01.666 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd nearfull ratio = 0.8 2024-05-08T21:23:01.666 INFO:teuthology.orchestra.run.smithi186.stdout: osd failsafe full ratio = 0.95 2024-05-08T21:23:01.666 INFO:teuthology.orchestra.run.smithi186.stdout: osd objectstore = bluestore 2024-05-08T21:23:01.666 INFO:teuthology.orchestra.run.smithi186.stdout: osd op complaint time = 180 2024-05-08T21:23:01.666 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.666 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr] 2024-05-08T21:23:01.666 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-08T21:23:01.666 INFO:teuthology.orchestra.run.smithi186.stdout: debug mgr = 20 2024-05-08T21:23:01.666 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-05-08T21:23:01.666 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-05-08T21:23:01.666 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-05-08T21:23:01.666 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-05-08T21:23:01.666 INFO:teuthology.orchestra.run.smithi186.stdout: mgr/telemetry/nag = false 2024-05-08T21:23:01.667 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.667 INFO:teuthology.orchestra.run.smithi186.stdout:[mon] 2024-05-08T21:23:01.667 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-08T21:23:01.667 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-05-08T21:23:01.667 INFO:teuthology.orchestra.run.smithi186.stdout: debug paxos = 20 2024-05-08T21:23:01.667 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-05-08T21:23:01.667 INFO:teuthology.orchestra.run.smithi186.stdout: mon data avail warn = 5 2024-05-08T21:23:01.667 INFO:teuthology.orchestra.run.smithi186.stdout: mon mgr mkfs grace = 240 2024-05-08T21:23:01.667 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-05-08T21:23:01.667 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd reporter subtree level = osd 2024-05-08T21:23:01.667 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd prime pg temp = true 2024-05-08T21:23:01.667 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-05-08T21:23:01.667 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.667 INFO:teuthology.orchestra.run.smithi186.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-08T21:23:01.667 INFO:teuthology.orchestra.run.smithi186.stdout: auth mon ticket ttl = 660 # 11m 2024-05-08T21:23:01.667 INFO:teuthology.orchestra.run.smithi186.stdout: auth service ticket ttl = 240 # 4m 2024-05-08T21:23:01.668 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.668 INFO:teuthology.orchestra.run.smithi186.stdout: # don't complain about insecure global_id in the test suite 2024-05-08T21:23:01.668 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-08T21:23:01.668 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-08T21:23:01.668 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.668 INFO:teuthology.orchestra.run.smithi186.stdout: # 1m isn't quite enough 2024-05-08T21:23:01.668 INFO:teuthology.orchestra.run.smithi186.stdout: mon_down_mkfs_grace = 2m 2024-05-08T21:23:01.668 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.668 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_filestore_osds = false 2024-05-08T21:23:01.668 INFO:teuthology.orchestra.run.smithi186.stdout: mon op complaint time = 120 2024-05-08T21:23:01.668 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.668 INFO:teuthology.orchestra.run.smithi186.stdout:[client] 2024-05-08T21:23:01.668 INFO:teuthology.orchestra.run.smithi186.stdout: rgw cache enabled = true 2024-05-08T21:23:01.668 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable ops log = true 2024-05-08T21:23:01.668 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable usage log = true 2024-05-08T21:23:01.669 INFO:teuthology.orchestra.run.smithi186.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-08T21:23:01.669 INFO:teuthology.orchestra.run.smithi186.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-08T21:23:01.669 INFO:teuthology.orchestra.run.smithi186.stdout: client mount timeout = 600 2024-05-08T21:23:01.669 INFO:teuthology.orchestra.run.smithi186.stdout: debug client = 20 2024-05-08T21:23:01.669 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-08T21:23:01.669 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = "" 2024-05-08T21:23:01.669 INFO:teuthology.orchestra.run.smithi186.stdout: rados mon op timeout = 900 2024-05-08T21:23:01.669 INFO:teuthology.orchestra.run.smithi186.stdout: rados osd op timeout = 900 2024-05-08T21:23:01.669 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.a] 2024-05-08T21:23:01.669 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.b] 2024-05-08T21:23:01.669 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.c] 2024-05-08T21:23:01.669 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.a] 2024-05-08T21:23:01.669 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.c] 2024-05-08T21:23:01.669 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.b] 2024-05-08T21:23:01.669 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.d] 2024-05-08T21:23:01.669 INFO:teuthology.orchestra.run.smithi186.stdout:[mds] 2024-05-08T21:23:01.670 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds = 20 2024-05-08T21:23:01.670 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds balancer = 20 2024-05-08T21:23:01.670 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-08T21:23:01.670 INFO:teuthology.orchestra.run.smithi186.stdout: mds debug frag = True 2024-05-08T21:23:01.670 INFO:teuthology.orchestra.run.smithi186.stdout: mds debug scatterstat = True 2024-05-08T21:23:01.670 INFO:teuthology.orchestra.run.smithi186.stdout: mds op complaint time = 180 2024-05-08T21:23:01.670 INFO:teuthology.orchestra.run.smithi186.stdout: mds verify scatter = False 2024-05-08T21:23:01.670 INFO:teuthology.orchestra.run.smithi186.stdout: osd op complaint time = 180 2024-05-08T21:23:01.670 INFO:teuthology.orchestra.run.smithi186.stdout: rados mon op timeout = 900 2024-05-08T21:23:01.670 INFO:teuthology.orchestra.run.smithi186.stdout: rados osd op timeout = 900 2024-05-08T21:23:01.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-08T21:23:01.714 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-08T21:23:01.763 INFO:teuthology.orchestra.run.smithi186.stdout:[global] 2024-05-08T21:23:01.763 INFO:teuthology.orchestra.run.smithi186.stdout: chdir = "" 2024-05-08T21:23:01.763 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-08T21:23:01.763 INFO:teuthology.orchestra.run.smithi186.stdout: auth supported = cephx 2024-05-08T21:23:01.763 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.763 INFO:teuthology.orchestra.run.smithi186.stdout: filestore xattr use omap = true 2024-05-08T21:23:01.763 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: mon clock drift allowed = 1.000 2024-05-08T21:23:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: osd crush chooseleaf type = 0 2024-05-08T21:23:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: auth debug = true 2024-05-08T21:23:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on old message = true 2024-05-08T21:23:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on bug = true 2024-05-08T21:23:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-08T21:23:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: mon pg warn max object skew = 0 2024-05-08T21:23:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: # disable pg_autoscaler by default for new pools 2024-05-08T21:23:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-08T21:23:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default size = 2 2024-05-08T21:23:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow primary affinity = true 2024-05-08T21:23:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow pg remap = true 2024-05-08T21:23:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on legacy crush tunables = false 2024-05-08T21:23:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on crush straw calc version zero = false 2024-05-08T21:23:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on no sortbitwise = false 2024-05-08T21:23:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on osd down out interval zero = false 2024-05-08T21:23:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on too few osds = false 2024-05-08T21:23:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-08T21:23:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-08T21:23:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: mon_allow_pool_size_one = true 2024-05-08T21:23:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-08T21:23:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: osd default data pool replay window = 5 2024-05-08T21:23:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: mon allow pool delete = true 2024-05-08T21:23:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: mon cluster log file level = debug 2024-05-08T21:23:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: debug asserts on shutdown = true 2024-05-08T21:23:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: mon health detail to clog = false 2024-05-08T21:23:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: mon host = "172.21.15.3,172.21.15.186,[v2:172.21.15.186:3301,v1:172.21.15.186:6790]" 2024-05-08T21:23:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on skipped message = False 2024-05-08T21:23:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: fsid = d0f58275-758f-4470-b003-84d6abe25489 2024-05-08T21:23:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.766 INFO:teuthology.orchestra.run.smithi186.stdout:[osd] 2024-05-08T21:23:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: osd journal size = 100 2024-05-08T21:23:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub load threshold = 5.0 2024-05-08T21:23:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub max interval = 600 2024-05-08T21:23:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: osd mclock profile = high_recovery_ops 2024-05-08T21:23:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: osd recover clone overlap = true 2024-05-08T21:23:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: osd recovery max chunk = 1048576 2024-05-08T21:23:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug shutdown = true 2024-05-08T21:23:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug op order = true 2024-05-08T21:23:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify stray on activate = true 2024-05-08T21:23:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: osd open classes on start = true 2024-05-08T21:23:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug pg log writeout = true 2024-05-08T21:23:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: osd deep scrub update digest min age = 30 2024-05-08T21:23:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: osd map max advance = 10 2024-05-08T21:23:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: journal zero on create = true 2024-05-08T21:23:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: filestore ondisk finisher threads = 3 2024-05-08T21:23:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: filestore apply finisher threads = 3 2024-05-08T21:23:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: bdev debug aio = true 2024-05-08T21:23:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug misdirected ops = true 2024-05-08T21:23:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: bdev async discard = True 2024-05-08T21:23:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: bdev enable discard = True 2024-05-08T21:23:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore allocator = bitmap 2024-05-08T21:23:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore block size = 96636764160 2024-05-08T21:23:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore fsck on mount = True 2024-05-08T21:23:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluefs = 1/20 2024-05-08T21:23:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluestore = 1/20 2024-05-08T21:23:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-08T21:23:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: debug osd = 20 2024-05-08T21:23:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: debug rocksdb = 4/10 2024-05-08T21:23:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd backfillfull_ratio = 0.85 2024-05-08T21:23:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd full ratio = 0.9 2024-05-08T21:23:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd nearfull ratio = 0.8 2024-05-08T21:23:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: osd failsafe full ratio = 0.95 2024-05-08T21:23:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: osd objectstore = bluestore 2024-05-08T21:23:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: osd op complaint time = 180 2024-05-08T21:23:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.769 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr] 2024-05-08T21:23:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-08T21:23:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: debug mgr = 20 2024-05-08T21:23:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-05-08T21:23:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-05-08T21:23:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-05-08T21:23:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-05-08T21:23:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: mgr/telemetry/nag = false 2024-05-08T21:23:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.770 INFO:teuthology.orchestra.run.smithi186.stdout:[mon] 2024-05-08T21:23:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-08T21:23:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-05-08T21:23:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: debug paxos = 20 2024-05-08T21:23:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-05-08T21:23:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: mon data avail warn = 5 2024-05-08T21:23:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: mon mgr mkfs grace = 240 2024-05-08T21:23:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-05-08T21:23:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd reporter subtree level = osd 2024-05-08T21:23:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd prime pg temp = true 2024-05-08T21:23:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-05-08T21:23:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-08T21:23:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: auth mon ticket ttl = 660 # 11m 2024-05-08T21:23:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: auth service ticket ttl = 240 # 4m 2024-05-08T21:23:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: # don't complain about insecure global_id in the test suite 2024-05-08T21:23:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-08T21:23:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-08T21:23:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: # 1m isn't quite enough 2024-05-08T21:23:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: mon_down_mkfs_grace = 2m 2024-05-08T21:23:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_filestore_osds = false 2024-05-08T21:23:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: mon op complaint time = 120 2024-05-08T21:23:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-08T21:23:01.772 INFO:teuthology.orchestra.run.smithi186.stdout:[client] 2024-05-08T21:23:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: rgw cache enabled = true 2024-05-08T21:23:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable ops log = true 2024-05-08T21:23:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable usage log = true 2024-05-08T21:23:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-08T21:23:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-08T21:23:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: client mount timeout = 600 2024-05-08T21:23:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: debug client = 20 2024-05-08T21:23:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-08T21:23:01.773 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = "" 2024-05-08T21:23:01.773 INFO:teuthology.orchestra.run.smithi186.stdout: rados mon op timeout = 900 2024-05-08T21:23:01.773 INFO:teuthology.orchestra.run.smithi186.stdout: rados osd op timeout = 900 2024-05-08T21:23:01.773 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.a] 2024-05-08T21:23:01.773 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.b] 2024-05-08T21:23:01.773 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.c] 2024-05-08T21:23:01.773 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.a] 2024-05-08T21:23:01.773 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.c] 2024-05-08T21:23:01.773 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.b] 2024-05-08T21:23:01.773 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.d] 2024-05-08T21:23:01.773 INFO:teuthology.orchestra.run.smithi186.stdout:[mds] 2024-05-08T21:23:01.773 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds = 20 2024-05-08T21:23:01.773 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds balancer = 20 2024-05-08T21:23:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-08T21:23:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: mds debug frag = True 2024-05-08T21:23:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: mds debug scatterstat = False 2024-05-08T21:23:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: mds op complaint time = 180 2024-05-08T21:23:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: mds verify scatter = False 2024-05-08T21:23:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: osd op complaint time = 180 2024-05-08T21:23:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: rados mon op timeout = 900 2024-05-08T21:23:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: rados osd op timeout = 900 2024-05-08T21:23:01.774 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2024-05-08T21:23:01.775 INFO:teuthology.orchestra.run.smithi003.stdout: chdir = "" 2024-05-08T21:23:01.775 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-08T21:23:01.775 INFO:teuthology.orchestra.run.smithi003.stdout: auth supported = cephx 2024-05-08T21:23:01.775 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.775 INFO:teuthology.orchestra.run.smithi003.stdout: filestore xattr use omap = true 2024-05-08T21:23:01.775 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.775 INFO:teuthology.orchestra.run.smithi003.stdout: mon clock drift allowed = 1.000 2024-05-08T21:23:01.775 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.775 INFO:teuthology.orchestra.run.smithi003.stdout: osd crush chooseleaf type = 0 2024-05-08T21:23:01.775 INFO:teuthology.orchestra.run.smithi003.stdout: auth debug = true 2024-05-08T21:23:01.775 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.775 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on old message = true 2024-05-08T21:23:01.776 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on bug = true 2024-05-08T21:23:01.776 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.776 INFO:teuthology.orchestra.run.smithi003.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-08T21:23:01.776 INFO:teuthology.orchestra.run.smithi003.stdout: mon pg warn max object skew = 0 2024-05-08T21:23:01.776 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.776 INFO:teuthology.orchestra.run.smithi003.stdout: # disable pg_autoscaler by default for new pools 2024-05-08T21:23:01.776 INFO:teuthology.orchestra.run.smithi003.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-08T21:23:01.776 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.776 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default size = 2 2024-05-08T21:23:01.776 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.776 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow primary affinity = true 2024-05-08T21:23:01.776 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow pg remap = true 2024-05-08T21:23:01.776 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on legacy crush tunables = false 2024-05-08T21:23:01.776 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on crush straw calc version zero = false 2024-05-08T21:23:01.776 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on no sortbitwise = false 2024-05-08T21:23:01.777 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on osd down out interval zero = false 2024-05-08T21:23:01.777 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on too few osds = false 2024-05-08T21:23:01.777 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-08T21:23:01.777 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-08T21:23:01.777 INFO:teuthology.orchestra.run.smithi003.stdout: mon_allow_pool_size_one = true 2024-05-08T21:23:01.777 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.777 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-08T21:23:01.777 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.777 INFO:teuthology.orchestra.run.smithi003.stdout: osd default data pool replay window = 5 2024-05-08T21:23:01.777 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.777 INFO:teuthology.orchestra.run.smithi003.stdout: mon allow pool delete = true 2024-05-08T21:23:01.777 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.777 INFO:teuthology.orchestra.run.smithi003.stdout: mon cluster log file level = debug 2024-05-08T21:23:01.777 INFO:teuthology.orchestra.run.smithi003.stdout: debug asserts on shutdown = true 2024-05-08T21:23:01.777 INFO:teuthology.orchestra.run.smithi003.stdout: mon health detail to clog = false 2024-05-08T21:23:01.777 INFO:teuthology.orchestra.run.smithi003.stdout: mon host = "172.21.15.3,172.21.15.186,[v2:172.21.15.186:3301,v1:172.21.15.186:6790]" 2024-05-08T21:23:01.778 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on skipped message = False 2024-05-08T21:23:01.778 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = d0f58275-758f-4470-b003-84d6abe25489 2024-05-08T21:23:01.778 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.778 INFO:teuthology.orchestra.run.smithi003.stdout:[osd] 2024-05-08T21:23:01.778 INFO:teuthology.orchestra.run.smithi003.stdout: osd journal size = 100 2024-05-08T21:23:01.778 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.778 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub load threshold = 5.0 2024-05-08T21:23:01.778 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub max interval = 600 2024-05-08T21:23:01.778 INFO:teuthology.orchestra.run.smithi003.stdout: osd mclock profile = high_recovery_ops 2024-05-08T21:23:01.778 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.778 INFO:teuthology.orchestra.run.smithi003.stdout: osd recover clone overlap = true 2024-05-08T21:23:01.778 INFO:teuthology.orchestra.run.smithi003.stdout: osd recovery max chunk = 1048576 2024-05-08T21:23:01.778 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.778 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug shutdown = true 2024-05-08T21:23:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug op order = true 2024-05-08T21:23:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify stray on activate = true 2024-05-08T21:23:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: osd open classes on start = true 2024-05-08T21:23:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug pg log writeout = true 2024-05-08T21:23:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: osd deep scrub update digest min age = 30 2024-05-08T21:23:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: osd map max advance = 10 2024-05-08T21:23:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: journal zero on create = true 2024-05-08T21:23:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: filestore ondisk finisher threads = 3 2024-05-08T21:23:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: filestore apply finisher threads = 3 2024-05-08T21:23:01.779 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.780 INFO:teuthology.orchestra.run.smithi003.stdout: bdev debug aio = true 2024-05-08T21:23:01.780 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug misdirected ops = true 2024-05-08T21:23:01.780 INFO:teuthology.orchestra.run.smithi003.stdout: bdev async discard = True 2024-05-08T21:23:01.780 INFO:teuthology.orchestra.run.smithi003.stdout: bdev enable discard = True 2024-05-08T21:23:01.780 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore allocator = bitmap 2024-05-08T21:23:01.780 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore block size = 96636764160 2024-05-08T21:23:01.780 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore fsck on mount = True 2024-05-08T21:23:01.780 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluefs = 1/20 2024-05-08T21:23:01.780 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluestore = 1/20 2024-05-08T21:23:01.780 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-08T21:23:01.780 INFO:teuthology.orchestra.run.smithi003.stdout: debug osd = 20 2024-05-08T21:23:01.780 INFO:teuthology.orchestra.run.smithi003.stdout: debug rocksdb = 4/10 2024-05-08T21:23:01.780 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd backfillfull_ratio = 0.85 2024-05-08T21:23:01.780 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd full ratio = 0.9 2024-05-08T21:23:01.780 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd nearfull ratio = 0.8 2024-05-08T21:23:01.780 INFO:teuthology.orchestra.run.smithi003.stdout: osd failsafe full ratio = 0.95 2024-05-08T21:23:01.781 INFO:teuthology.orchestra.run.smithi003.stdout: osd objectstore = bluestore 2024-05-08T21:23:01.781 INFO:teuthology.orchestra.run.smithi003.stdout: osd op complaint time = 180 2024-05-08T21:23:01.781 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.781 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr] 2024-05-08T21:23:01.781 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-08T21:23:01.781 INFO:teuthology.orchestra.run.smithi003.stdout: debug mgr = 20 2024-05-08T21:23:01.781 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2024-05-08T21:23:01.781 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2024-05-08T21:23:01.781 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2024-05-08T21:23:01.781 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2024-05-08T21:23:01.781 INFO:teuthology.orchestra.run.smithi003.stdout: mgr/telemetry/nag = false 2024-05-08T21:23:01.781 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.781 INFO:teuthology.orchestra.run.smithi003.stdout:[mon] 2024-05-08T21:23:01.781 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-08T21:23:01.782 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2024-05-08T21:23:01.782 INFO:teuthology.orchestra.run.smithi003.stdout: debug paxos = 20 2024-05-08T21:23:01.782 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2024-05-08T21:23:01.782 INFO:teuthology.orchestra.run.smithi003.stdout: mon data avail warn = 5 2024-05-08T21:23:01.782 INFO:teuthology.orchestra.run.smithi003.stdout: mon mgr mkfs grace = 240 2024-05-08T21:23:01.782 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2024-05-08T21:23:01.782 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd reporter subtree level = osd 2024-05-08T21:23:01.782 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd prime pg temp = true 2024-05-08T21:23:01.782 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2024-05-08T21:23:01.782 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.782 INFO:teuthology.orchestra.run.smithi003.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-08T21:23:01.782 INFO:teuthology.orchestra.run.smithi003.stdout: auth mon ticket ttl = 660 # 11m 2024-05-08T21:23:01.782 INFO:teuthology.orchestra.run.smithi003.stdout: auth service ticket ttl = 240 # 4m 2024-05-08T21:23:01.782 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.783 INFO:teuthology.orchestra.run.smithi003.stdout: # don't complain about insecure global_id in the test suite 2024-05-08T21:23:01.783 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-08T21:23:01.783 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-08T21:23:01.783 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.783 INFO:teuthology.orchestra.run.smithi003.stdout: # 1m isn't quite enough 2024-05-08T21:23:01.783 INFO:teuthology.orchestra.run.smithi003.stdout: mon_down_mkfs_grace = 2m 2024-05-08T21:23:01.783 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.783 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_filestore_osds = false 2024-05-08T21:23:01.783 INFO:teuthology.orchestra.run.smithi003.stdout: mon op complaint time = 120 2024-05-08T21:23:01.783 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:01.783 INFO:teuthology.orchestra.run.smithi003.stdout:[client] 2024-05-08T21:23:01.783 INFO:teuthology.orchestra.run.smithi003.stdout: rgw cache enabled = true 2024-05-08T21:23:01.783 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable ops log = true 2024-05-08T21:23:01.783 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable usage log = true 2024-05-08T21:23:01.783 INFO:teuthology.orchestra.run.smithi003.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-08T21:23:01.783 INFO:teuthology.orchestra.run.smithi003.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-08T21:23:01.784 INFO:teuthology.orchestra.run.smithi003.stdout: client mount timeout = 600 2024-05-08T21:23:01.784 INFO:teuthology.orchestra.run.smithi003.stdout: debug client = 20 2024-05-08T21:23:01.784 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-08T21:23:01.784 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = "" 2024-05-08T21:23:01.784 INFO:teuthology.orchestra.run.smithi003.stdout: rados mon op timeout = 900 2024-05-08T21:23:01.784 INFO:teuthology.orchestra.run.smithi003.stdout: rados osd op timeout = 900 2024-05-08T21:23:01.784 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.a] 2024-05-08T21:23:01.784 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.b] 2024-05-08T21:23:01.784 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.c] 2024-05-08T21:23:01.784 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.a] 2024-05-08T21:23:01.784 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.c] 2024-05-08T21:23:01.784 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.b] 2024-05-08T21:23:01.784 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.d] 2024-05-08T21:23:01.784 INFO:teuthology.orchestra.run.smithi003.stdout:[mds] 2024-05-08T21:23:01.784 INFO:teuthology.orchestra.run.smithi003.stdout: debug mds = 20 2024-05-08T21:23:01.784 INFO:teuthology.orchestra.run.smithi003.stdout: debug mds balancer = 20 2024-05-08T21:23:01.785 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-05-08T21:23:01.785 INFO:teuthology.orchestra.run.smithi003.stdout: mds debug frag = True 2024-05-08T21:23:01.785 INFO:teuthology.orchestra.run.smithi003.stdout: mds debug scatterstat = False 2024-05-08T21:23:01.785 INFO:teuthology.orchestra.run.smithi003.stdout: mds op complaint time = 180 2024-05-08T21:23:01.785 INFO:teuthology.orchestra.run.smithi003.stdout: mds verify scatter = False 2024-05-08T21:23:01.785 INFO:teuthology.orchestra.run.smithi003.stdout: osd op complaint time = 180 2024-05-08T21:23:01.785 INFO:teuthology.orchestra.run.smithi003.stdout: rados mon op timeout = 900 2024-05-08T21:23:01.785 INFO:teuthology.orchestra.run.smithi003.stdout: rados osd op timeout = 900 2024-05-08T21:23:01.787 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-08T21:23:01.787 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-08T21:23:01.787 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-08T21:23:01.787 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:23:01.787 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-08T21:23:01.787 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-08T21:23:01.787 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-08T21:23:01.788 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:23:01.788 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-08T21:23:01.788 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-08T21:23:01.788 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-08T21:23:01.788 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:23:01.788 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-08T21:23:01.788 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-08T21:23:01.789 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-08T21:23:01.789 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:23:01.799 INFO:tasks.ceph.mds.a:Stopped 2024-05-08T21:23:01.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-08T21:23:01.799 INFO:tasks.ceph.mds.c:Stopped 2024-05-08T21:23:01.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-08T21:23:01.801 INFO:tasks.ceph.mds.d:Stopped 2024-05-08T21:23:01.801 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-08T21:23:01.801 INFO:tasks.ceph.mds.b:Stopped 2024-05-08T21:23:01.802 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-08T21:23:01.842 INFO:tasks.ceph.mds.a:Started 2024-05-08T21:23:01.842 DEBUG:teuthology.parallel:result is None 2024-05-08T21:23:01.843 INFO:tasks.ceph.mds.c:Started 2024-05-08T21:23:01.843 DEBUG:teuthology.parallel:result is None 2024-05-08T21:23:01.844 INFO:tasks.ceph.mds.d:Started 2024-05-08T21:23:01.844 DEBUG:teuthology.parallel:result is None 2024-05-08T21:23:01.845 INFO:tasks.ceph.mds.b:Started 2024-05-08T21:23:01.845 DEBUG:teuthology.parallel:result is None 2024-05-08T21:23:01.845 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph purge cephfs_metadata --yes-i-really-really-mean-it 2024-05-08T21:23:01.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.896+0000 7f0eacb60080 1 Processor -- start 2024-05-08T21:23:01.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.897+0000 7f0eacb60080 1 -- start start 2024-05-08T21:23:01.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.898+0000 7f0eacb60080 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5587792bf780 0x5587792ee330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.898+0000 7f0eacb60080 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587793082c0 0x5587792bccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.898+0000 7f0eacb60080 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558779393f30 0x558779396010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.898+0000 7f0eacb60080 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587792698b0 con 0x5587793082c0 2024-05-08T21:23:01.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.898+0000 7f0eacb60080 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587792651a0 con 0x558779393f30 2024-05-08T21:23:01.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.898+0000 7f0eacb60080 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x5587792006d0 con 0x5587792bf780 2024-05-08T21:23:01.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.898+0000 7f0e9bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587793082c0 0x5587792bccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.898+0000 7f0e9bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587793082c0 0x5587792bccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53990/0 (socket says 172.21.15.3:53990) 2024-05-08T21:23:01.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.898+0000 7f0e9bfff700 1 -- 172.21.15.3:0/721906991 learned_addr learned my addr 172.21.15.3:0/721906991 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:01.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.898+0000 7f0ea2c32700 1 --2- 172.21.15.3:0/721906991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5587792bf780 0x5587792ee330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.898+0000 7f0ea3433700 1 --2- 172.21.15.3:0/721906991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558779393f30 0x558779396010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.898+0000 7f0e9bfff700 1 -- 172.21.15.3:0/721906991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5587792bf780 msgr2=0x5587792ee330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:01.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.898+0000 7f0e9bfff700 1 --2- 172.21.15.3:0/721906991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5587792bf780 0x5587792ee330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.898+0000 7f0e9bfff700 1 -- 172.21.15.3:0/721906991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558779393f30 msgr2=0x558779396010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:01.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.898+0000 7f0e9bfff700 1 --2- 172.21.15.3:0/721906991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558779393f30 0x558779396010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.898+0000 7f0e9bfff700 1 -- 172.21.15.3:0/721906991 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e94005d20 con 0x5587793082c0 2024-05-08T21:23:01.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.899+0000 7f0e9bfff700 1 --2- 172.21.15.3:0/721906991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587793082c0 0x5587792bccf0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c001e80 tx=0x7f0e8c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6c92bc667b9e0b server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:01.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.899+0000 7f0ea1c30700 1 -- 172.21.15.3:0/721906991 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c00f3a0 con 0x5587793082c0 2024-05-08T21:23:01.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.899+0000 7f0ea1c30700 1 -- 172.21.15.3:0/721906991 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e8c00a040 con 0x5587793082c0 2024-05-08T21:23:01.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.899+0000 7f0ea1c30700 1 -- 172.21.15.3:0/721906991 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c00f740 con 0x5587793082c0 2024-05-08T21:23:01.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.900+0000 7f0eacb60080 1 -- 172.21.15.3:0/721906991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587793082c0 msgr2=0x5587792bccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.900+0000 7f0eacb60080 1 --2- 172.21.15.3:0/721906991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587793082c0 0x5587792bccf0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c001e80 tx=0x7f0e8c00b4a0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.900+0000 7f0eacb60080 1 -- 172.21.15.3:0/721906991 shutdown_connections 2024-05-08T21:23:01.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.900+0000 7f0eacb60080 1 --2- 172.21.15.3:0/721906991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5587792bf780 0x5587792ee330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.900+0000 7f0eacb60080 1 --2- 172.21.15.3:0/721906991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558779393f30 0x558779396010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.900+0000 7f0eacb60080 1 --2- 172.21.15.3:0/721906991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587793082c0 0x5587792bccf0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.900+0000 7f0eacb60080 1 -- 172.21.15.3:0/721906991 >> 172.21.15.3:0/721906991 conn(0x5587792e92d0 msgr2=0x558779394310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:01.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.900+0000 7f0eacb60080 1 -- 172.21.15.3:0/721906991 shutdown_connections 2024-05-08T21:23:01.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.900+0000 7f0eacb60080 1 -- 172.21.15.3:0/721906991 wait complete. 2024-05-08T21:23:01.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.901+0000 7f0eacb60080 1 Processor -- start 2024-05-08T21:23:01.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.901+0000 7f0eacb60080 1 -- start start 2024-05-08T21:23:01.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.902+0000 7f0eacb60080 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587792bf780 0x55877941fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.902+0000 7f0eacb60080 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5587793082c0 0x558779383a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.902+0000 7f0eacb60080 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558779393f30 0x5587792ee150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.902+0000 7f0eacb60080 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558779200370 con 0x558779393f30 2024-05-08T21:23:01.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.902+0000 7f0eacb60080 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587791f7680 con 0x5587792bf780 2024-05-08T21:23:01.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.902+0000 7f0eacb60080 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55877938c580 con 0x5587793082c0 2024-05-08T21:23:01.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.903+0000 7f0ea2c32700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587792bf780 0x55877941fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.903+0000 7f0e9bfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5587793082c0 0x558779383a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.903+0000 7f0ea2c32700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587792bf780 0x55877941fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:38942/0 (socket says 172.21.15.3:38942) 2024-05-08T21:23:01.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.903+0000 7f0ea2c32700 1 -- 172.21.15.3:0/4157376603 learned_addr learned my addr 172.21.15.3:0/4157376603 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:01.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.903+0000 7f0ea3433700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558779393f30 0x5587792ee150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.903+0000 7f0ea2c32700 1 -- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5587793082c0 msgr2=0x558779383a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.903+0000 7f0ea2c32700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5587793082c0 0x558779383a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.903+0000 7f0ea2c32700 1 -- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558779393f30 msgr2=0x5587792ee150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.903+0000 7f0ea2c32700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558779393f30 0x5587792ee150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.903+0000 7f0ea2c32700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8c001ab0 con 0x5587792bf780 2024-05-08T21:23:01.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.903+0000 7f0ea2c32700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587792bf780 0x55877941fa30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0e94002000 tx=0x7f0e94002fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:01.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.903+0000 7f0ea0c2e700 1 -- 172.21.15.3:0/4157376603 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94013690 con 0x5587792bf780 2024-05-08T21:23:01.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.903+0000 7f0ea0c2e700 1 -- 172.21.15.3:0/4157376603 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e94013810 con 0x5587792bf780 2024-05-08T21:23:01.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.903+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558779202ed0 con 0x5587792bf780 2024-05-08T21:23:01.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.904+0000 7f0ea0c2e700 1 -- 172.21.15.3:0/4157376603 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94013bc0 con 0x5587792bf780 2024-05-08T21:23:01.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.904+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558779202930 con 0x5587792bf780 2024-05-08T21:23:01.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.905+0000 7f0ea0c2e700 1 -- 172.21.15.3:0/4157376603 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0e94013d20 con 0x5587792bf780 2024-05-08T21:23:01.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.906+0000 7f0ea0c2e700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0e840695e0 0x7f0e8406baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.906+0000 7f0ea0c2e700 1 -- 172.21.15.3:0/4157376603 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0e94089c40 con 0x5587792bf780 2024-05-08T21:23:01.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.906+0000 7f0e9bfff700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0e840695e0 0x7f0e8406baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.909 INFO:teuthology.orchestra.run.smithi003.stdout:Warning: using slow linear search 2024-05-08T21:23:01.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.906+0000 7f0eacb60080 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x5587792ee690 0x55877938c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.906+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:1 4.0 4:00000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x5587792ee690 2024-05-08T21:23:01.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.906+0000 7f0ea3433700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x5587792ee690 0x55877938c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.906+0000 7f0e9bfff700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0e840695e0 0x7f0e8406baa0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x5587792c1240 tx=0x7f0e8c003000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:01.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.907+0000 7f0ea3433700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x5587792ee690 0x55877938c9a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:01.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.907+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 1 ==== osd_op_reply(1 [pgnls start_epoch 35 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e9c010560 con 0x5587792ee690 2024-05-08T21:23:01.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.907+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:2 4.10 4:08000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x5587792ee690 2024-05-08T21:23:01.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.908+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 2 ==== osd_op_reply(2 [pgnls start_epoch 35 out=177b] v34'10 uv10 ondisk = 1) v8 ==== 144+0+177 (crc 0 0 0) 0x7f0e9c0180c0 con 0x5587792ee690 2024-05-08T21:23:01.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.908+0000 7f0ea3433700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x7f0e9c001630 0x7f0e9c001ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.908+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:3 4.8 4:10000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x7f0e9c001630 2024-05-08T21:23:01.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.908+0000 7f0ea2c32700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x7f0e9c001630 0x7f0e9c001ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.908+0000 7f0ea2c32700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x7f0e9c001630 0x7f0e9c001ab0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:01.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.911+0000 7f0ea2c32700 1 -- 172.21.15.3:0/4157376603 <== osd.5 v2:172.21.15.186:6800/47340 1 ==== osd_op_reply(3 [pgnls start_epoch 35 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e94025070 con 0x7f0e9c001630 2024-05-08T21:23:01.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.911+0000 7f0ea2c32700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x7f0e94028030 0x7f0e94057a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.911+0000 7f0ea2c32700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:4 4.18 4:18000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x7f0e94028030 2024-05-08T21:23:01.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.911+0000 7f0e9bfff700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x7f0e94028030 0x7f0e94057a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.911+0000 7f0e9bfff700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x7f0e94028030 0x7f0e94057a50 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:01.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.912+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.4 v2:172.21.15.186:6810/47344 1 ==== osd_op_reply(4 [pgnls start_epoch 35 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e94028030 2024-05-08T21:23:01.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.912+0000 7f0e9bfff700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x7f0e8c010bd0 0x7f0e8c01e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.912+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:5 4.4 4:20000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x7f0e8c010bd0 2024-05-08T21:23:01.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.912+0000 7f0ea3433700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x7f0e8c010bd0 0x7f0e8c01e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.912+0000 7f0ea3433700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x7f0e8c010bd0 0x7f0e8c01e920 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:01.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.915+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.1 v2:172.21.15.3:6816/47119 1 ==== osd_op_reply(5 [pgnls start_epoch 35 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e9c0180c0 con 0x7f0e8c010bd0 2024-05-08T21:23:01.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.915+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:6 4.14 4:28000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x5587792ee690 2024-05-08T21:23:01.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.916+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 3 ==== osd_op_reply(6 [pgnls start_epoch 35 out=149b] v34'18 uv18 ondisk = 1) v8 ==== 144+0+149 (crc 0 0 0) 0x7f0e9c0180c0 con 0x5587792ee690 2024-05-08T21:23:01.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.916+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:7 4.c 4:30000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x7f0e94028030 2024-05-08T21:23:01.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.916+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.4 v2:172.21.15.186:6810/47344 2 ==== osd_op_reply(7 [pgnls start_epoch 35 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e94028030 2024-05-08T21:23:01.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.917+0000 7f0e9bfff700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x7f0e8c029910 0x7f0e8c00f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.917+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:8 4.1c 4:38000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x7f0e8c029910 2024-05-08T21:23:01.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.917+0000 7f0ea2c32700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x7f0e8c029910 0x7f0e8c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.917+0000 7f0ea2c32700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x7f0e8c029910 0x7f0e8c00f3a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:01.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.920+0000 7f0ea2c32700 1 -- 172.21.15.3:0/4157376603 <== osd.2 v2:172.21.15.3:6808/47117 1 ==== osd_op_reply(8 [pgnls start_epoch 35 out=73b] v33'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x7f0e94025070 con 0x7f0e8c029910 2024-05-08T21:23:01.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.920+0000 7f0ea2c32700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:9 4.2 4:40000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x7f0e8c010bd0 2024-05-08T21:23:01.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.920+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.1 v2:172.21.15.3:6816/47119 2 ==== osd_op_reply(9 [pgnls start_epoch 35 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e9c0180c0 con 0x7f0e8c010bd0 2024-05-08T21:23:01.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.920+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:10 4.12 4:48000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x7f0e8c010bd0 2024-05-08T21:23:01.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.921+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.1 v2:172.21.15.3:6816/47119 3 ==== osd_op_reply(10 [pgnls start_epoch 35 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e9c0180c0 con 0x7f0e8c010bd0 2024-05-08T21:23:01.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.921+0000 7f0ea3433700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x7f0e9c0203c0 0x7f0e9c020840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.921+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:11 4.a 4:50000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x7f0e9c0203c0 2024-05-08T21:23:01.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.921+0000 7f0e9bfff700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x7f0e9c0203c0 0x7f0e9c020840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.922+0000 7f0e9bfff700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x7f0e9c0203c0 0x7f0e9c020840 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:01.924 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2024-05-08T21:23:01.925 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2024-05-08T21:23:01.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.924+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.6 v2:172.21.15.186:6808/47351 1 ==== osd_op_reply(11 [pgnls start_epoch 35 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e9c0203c0 2024-05-08T21:23:01.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.924+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:12 4.1a 4:58000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x7f0e94028030 2024-05-08T21:23:01.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.925+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.4 v2:172.21.15.186:6810/47344 3 ==== osd_op_reply(12 [pgnls start_epoch 35 out=49b] v33'1 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e94028030 2024-05-08T21:23:01.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.925+0000 7f0e9bfff700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x7f0e8c039920 0x7f0e8c026370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:01.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.925+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:13 4.6 4:60000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x7f0e8c039920 2024-05-08T21:23:01.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.925+0000 7f0ea3433700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x7f0e8c039920 0x7f0e8c026370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:01.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.925+0000 7f0ea3433700 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x7f0e8c039920 0x7f0e8c026370 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:01.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.929+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.0 v2:172.21.15.3:6824/47120 1 ==== osd_op_reply(13 [pgnls start_epoch 35 out=77b] v34'2 uv2 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x7f0e9c02b040 con 0x7f0e8c039920 2024-05-08T21:23:01.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.929+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:14 4.16 4:68000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x7f0e8c039920 2024-05-08T21:23:01.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.929+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.0 v2:172.21.15.3:6824/47120 2 ==== osd_op_reply(14 [pgnls start_epoch 35 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e9c02b040 con 0x7f0e8c039920 2024-05-08T21:23:01.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.929+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:15 4.e 4:70000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x7f0e94028030 2024-05-08T21:23:01.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.929+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.4 v2:172.21.15.186:6810/47344 4 ==== osd_op_reply(15 [pgnls start_epoch 35 out=73b] v33'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e94028030 2024-05-08T21:23:01.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.929+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:16 4.1e 4:78000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x7f0e8c039920 2024-05-08T21:23:01.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.930+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.0 v2:172.21.15.3:6824/47120 3 ==== osd_op_reply(16 [pgnls start_epoch 35 out=49b] v33'1 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e9c02b040 con 0x7f0e8c039920 2024-05-08T21:23:01.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.930+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:17 4.1 4:80000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x7f0e94028030 2024-05-08T21:23:01.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.930+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.4 v2:172.21.15.186:6810/47344 5 ==== osd_op_reply(17 [pgnls start_epoch 35 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e94028030 2024-05-08T21:23:01.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.930+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:18 4.11 4:88000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x5587792ee690 2024-05-08T21:23:01.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.931+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 4 ==== osd_op_reply(18 [pgnls start_epoch 35 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e9c02b040 con 0x5587792ee690 2024-05-08T21:23:01.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.931+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:19 4.9 4:90000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x7f0e94028030 2024-05-08T21:23:01.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.931+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.4 v2:172.21.15.186:6810/47344 6 ==== osd_op_reply(19 [pgnls start_epoch 35 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e94028030 2024-05-08T21:23:01.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.931+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:20 4.19 4:98000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x5587792ee690 2024-05-08T21:23:01.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.932+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 5 ==== osd_op_reply(20 [pgnls start_epoch 35 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e9c02b040 con 0x5587792ee690 2024-05-08T21:23:01.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.932+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:21 4.5 4:a0000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x558779396010 con 0x7f0e9c0203c0 2024-05-08T21:23:01.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.932+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.6 v2:172.21.15.186:6808/47351 2 ==== osd_op_reply(21 [pgnls start_epoch 35 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e9c0203c0 2024-05-08T21:23:01.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.932+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:22 4.15 4:a8000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x7f0e8c046040 con 0x7f0e9c001630 2024-05-08T21:23:01.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.932+0000 7f0ea2c32700 1 -- 172.21.15.3:0/4157376603 <== osd.5 v2:172.21.15.186:6800/47340 2 ==== osd_op_reply(22 [pgnls start_epoch 35 out=97b] v33'3 uv3 ondisk = 1) v8 ==== 144+0+97 (crc 0 0 0) 0x7f0e94025070 con 0x7f0e9c001630 2024-05-08T21:23:01.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.932+0000 7f0ea2c32700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:23 4.d 4:b0000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x7f0e8c046040 con 0x7f0e94028030 2024-05-08T21:23:01.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.933+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.4 v2:172.21.15.186:6810/47344 7 ==== osd_op_reply(23 [pgnls start_epoch 35 out=74b] v33'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e94028030 2024-05-08T21:23:01.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.933+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:24 4.1d 4:b8000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x7f0e8c046040 con 0x5587792ee690 2024-05-08T21:23:01.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.933+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 6 ==== osd_op_reply(24 [pgnls start_epoch 35 out=73b] v33'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x7f0e9c02b040 con 0x5587792ee690 2024-05-08T21:23:01.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.933+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:25 4.3 4:c0000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x7f0e8c046040 con 0x7f0e8c039920 2024-05-08T21:23:01.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.934+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.0 v2:172.21.15.3:6824/47120 4 ==== osd_op_reply(25 [pgnls start_epoch 35 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e9c02b040 con 0x7f0e8c039920 2024-05-08T21:23:01.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.934+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:26 4.13 4:c8000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x7f0e8c046040 con 0x7f0e94028030 2024-05-08T21:23:01.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.934+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.4 v2:172.21.15.186:6810/47344 8 ==== osd_op_reply(26 [pgnls start_epoch 35 out=98b] v34'3 uv3 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e94028030 2024-05-08T21:23:01.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.934+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:27 4.b 4:d0000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x7f0e8c046040 con 0x7f0e8c039920 2024-05-08T21:23:01.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.934+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.0 v2:172.21.15.3:6824/47120 5 ==== osd_op_reply(27 [pgnls start_epoch 35 out=76b] v34'3 uv3 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x7f0e9c02b040 con 0x7f0e8c039920 2024-05-08T21:23:01.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.934+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:28 4.1b 4:d8000000:%01:::head [pgnls start_epoch 35 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e35) v8 -- 0x7f0e8c046040 con 0x7f0e94028030 2024-05-08T21:23:01.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.935+0000 7f0ea0c2e700 1 -- 172.21.15.3:0/4157376603 <== osd.4 v2:172.21.15.186:6810/47344 9 ==== osd_map(36..36 src has 1..36) v4 ==== 288+0+0 (crc 0 0 0) 0x7f0e8c010750 con 0x7f0e94028030 2024-05-08T21:23:01.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.935+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.4 v2:172.21.15.186:6810/47344 10 ==== osd_op_reply(28 [pgnls start_epoch 35 out=49b] v33'1 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e94028030 2024-05-08T21:23:01.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.935+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:29 4.7 4:e0000000:%01:::head [pgnls start_epoch 36 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e36) v8 -- 0x7f0e8c046040 con 0x7f0e8c010bd0 2024-05-08T21:23:01.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.938+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.1 v2:172.21.15.3:6816/47119 4 ==== osd_op_reply(29 [pgnls start_epoch 36 out=73b] v33'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x7f0e9c02b040 con 0x7f0e8c010bd0 2024-05-08T21:23:01.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.938+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:30 4.17 4:e8000000:%01:::head [pgnls start_epoch 36 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e36) v8 -- 0x7f0e8c046040 con 0x5587792ee690 2024-05-08T21:23:01.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.938+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 7 ==== osd_op_reply(30 [pgnls start_epoch 36 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0e9c02b040 con 0x5587792ee690 2024-05-08T21:23:01.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.938+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:31 4.f 4:f0000000:%01:::head [pgnls start_epoch 36 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e36) v8 -- 0x7f0e8c046040 con 0x7f0e8c010bd0 2024-05-08T21:23:01.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.939+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.1 v2:172.21.15.3:6816/47119 5 ==== osd_op_reply(31 [pgnls start_epoch 36 out=73b] v33'1 uv1 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x7f0e9c02b040 con 0x7f0e8c010bd0 2024-05-08T21:23:01.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.939+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:32 4.1f 4:f8000000:%01:::head [pgnls start_epoch 36 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e36) v8 -- 0x7f0e8c046040 con 0x7f0e9c0203c0 2024-05-08T21:23:01.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.6 v2:172.21.15.186:6808/47351 3 ==== osd_op_reply(32 [pgnls start_epoch 36 out=101b] v34'6 uv6 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e9c0203c0 2024-05-08T21:23:01.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:33 4.10 4:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x55877938d540 con 0x5587792ee690 2024-05-08T21:23:01.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:34 4.10 4:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x55877938cee0 con 0x5587792ee690 2024-05-08T21:23:01.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:35 4.10 4:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x558779306a90 con 0x5587792ee690 2024-05-08T21:23:01.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:36 4.10 4:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x558779307380 con 0x5587792ee690 2024-05-08T21:23:01.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:37 4.10 4:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x558779307c90 con 0x5587792ee690 2024-05-08T21:23:01.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:38 4.14 4:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x5587794366d0 con 0x5587792ee690 2024-05-08T21:23:01.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:39 4.14 4:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x558779436f70 con 0x5587792ee690 2024-05-08T21:23:01.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:40 4.14 4:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x558779437810 con 0x5587792ee690 2024-05-08T21:23:01.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:41 4.14 4:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x5587794380b0 con 0x5587792ee690 2024-05-08T21:23:01.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:42 4.1c 4:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x558779438950 con 0x7f0e8c029910 2024-05-08T21:23:01.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:43 4.6 4:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x558779439300 con 0x7f0e8c039920 2024-05-08T21:23:01.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:44 4.e 4:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x558779439ba0 con 0x7f0e94028030 2024-05-08T21:23:01.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:45 4.15 4:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x55877943a440 con 0x7f0e9c001630 2024-05-08T21:23:01.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:46 4.15 4:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x55877943ad40 con 0x7f0e9c001630 2024-05-08T21:23:01.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:47 4.d 4:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x55877943b5e0 con 0x7f0e94028030 2024-05-08T21:23:01.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.942+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:48 4.1d 4:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x55877943c0d0 con 0x5587792ee690 2024-05-08T21:23:01.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.944+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.4 v2:172.21.15.186:6810/47344 11 ==== osd_op_reply(44 608.00000000 [delete] v36'3 uv3 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e94028030 2024-05-08T21:23:01.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.944+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:49 4.13 4:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x55877943c4d0 con 0x7f0e94028030 2024-05-08T21:23:01.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.946+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.4 v2:172.21.15.186:6810/47344 12 ==== osd_op_reply(49 mds0_inotable [delete] v36'4 uv4 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e94028030 2024-05-08T21:23:01.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.946+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:50 4.13 4:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x55877943c880 con 0x7f0e94028030 2024-05-08T21:23:01.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.947+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 8 ==== osd_op_reply(38 1.00000000.inode [delete] v36'19 uv19 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f0e9c02b040 con 0x5587792ee690 2024-05-08T21:23:01.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.947+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:51 4.b 4:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x55877943cc30 con 0x7f0e8c039920 2024-05-08T21:23:01.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.947+0000 7f0ea2c32700 1 -- 172.21.15.3:0/4157376603 <== osd.5 v2:172.21.15.186:6800/47340 3 ==== osd_op_reply(45 604.00000000 [delete] v36'4 uv4 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f0e94025070 con 0x7f0e9c001630 2024-05-08T21:23:01.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.947+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:52 4.7 4:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x55877943cc30 con 0x7f0e8c010bd0 2024-05-08T21:23:01.949 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2024-05-08T21:23:01.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.947+0000 7f0ea2c32700 1 -- 172.21.15.3:0/4157376603 <== osd.5 v2:172.21.15.186:6800/47340 4 ==== osd_op_reply(46 500.00000000 [delete] v36'5 uv5 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f0e94025070 con 0x7f0e9c001630 2024-05-08T21:23:01.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.947+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:53 4.f 4:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x55877943cc30 con 0x7f0e8c010bd0 2024-05-08T21:23:01.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.949+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.4 v2:172.21.15.186:6810/47344 13 ==== osd_op_reply(47 mds_snaptable [delete] v36'2 uv2 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e94028030 2024-05-08T21:23:01.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.949+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:54 4.1f 4:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e36) v8 -- 0x55877943ccd0 con 0x7f0e9c0203c0 2024-05-08T21:23:01.950 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2024-05-08T21:23:01.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.949+0000 7f0ea0c2e700 1 -- 172.21.15.3:0/4157376603 <== osd.4 v2:172.21.15.186:6810/47344 14 ==== osd_map(37..37 src has 1..37) v4 ==== 288+0+0 (crc 0 0 0) 0x7f0e8c00fa60 con 0x7f0e94028030 2024-05-08T21:23:01.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.950+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.1 v2:172.21.15.3:6816/47119 6 ==== osd_op_reply(52 609.00000000 [delete] v36'3 uv3 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f0e9c02b040 con 0x7f0e8c010bd0 2024-05-08T21:23:01.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.950+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:55 4.1f 4:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e37) v8 -- 0x55877943d150 con 0x7f0e9c0203c0 2024-05-08T21:23:01.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.951+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 9 ==== osd_op_reply(48 605.00000000 [delete] v36'3 uv3 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f0e9c02b040 con 0x5587792ee690 2024-05-08T21:23:01.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.952+0000 7f0ea2c32700 1 -- 172.21.15.3:0/4157376603 <== osd.2 v2:172.21.15.3:6808/47117 2 ==== osd_op_reply(42 607.00000000 [delete] v36'3 uv3 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f0e94025070 con 0x7f0e8c029910 2024-05-08T21:23:01.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.952+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 10 ==== osd_op_reply(33 601.00000000 [delete] v36'11 uv11 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f0e9c02b040 con 0x5587792ee690 2024-05-08T21:23:01.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.952+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 11 ==== osd_op_reply(34 602.00000000 [delete] v36'12 uv12 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f0e9c02b040 con 0x5587792ee690 2024-05-08T21:23:01.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.952+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 12 ==== osd_op_reply(35 10000000000.00000000 [delete] v36'13 uv13 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f0e9c02b040 con 0x5587792ee690 2024-05-08T21:23:01.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.952+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 13 ==== osd_op_reply(36 600.00000000 [delete] v36'14 uv14 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f0e9c02b040 con 0x5587792ee690 2024-05-08T21:23:01.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.952+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 14 ==== osd_op_reply(37 603.00000000 [delete] v36'15 uv15 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f0e9c02b040 con 0x5587792ee690 2024-05-08T21:23:01.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.952+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 15 ==== osd_op_reply(39 200.00000000 [delete] v36'20 uv20 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f0e9c02b040 con 0x5587792ee690 2024-05-08T21:23:01.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.952+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 16 ==== osd_op_reply(40 200.00000001 [delete] v36'21 uv21 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f0e9c02b040 con 0x5587792ee690 2024-05-08T21:23:01.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.952+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.3 v2:172.21.15.3:6800/47118 17 ==== osd_op_reply(41 606.00000000 [delete] v36'22 uv22 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f0e9c02b040 con 0x5587792ee690 2024-05-08T21:23:01.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.953+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.1 v2:172.21.15.3:6816/47119 7 ==== osd_op_reply(53 400.00000000 [delete] v36'2 uv2 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f0e9c02b040 con 0x7f0e8c010bd0 2024-05-08T21:23:01.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.954+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.6 v2:172.21.15.186:6808/47351 4 ==== osd_op_reply(54 100.00000000.inode [delete] v36'7 uv7 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e9c0203c0 2024-05-08T21:23:01.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.954+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.4 v2:172.21.15.186:6810/47344 15 ==== osd_op_reply(50 100.00000000 [delete] v37'5 uv5 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e94028030 2024-05-08T21:23:01.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.955+0000 7f0e9bfff700 1 -- 172.21.15.3:0/4157376603 <== osd.6 v2:172.21.15.186:6808/47351 5 ==== osd_op_reply(55 1.00000000 [delete] v37'8 uv8 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7f0e8c018070 con 0x7f0e9c0203c0 2024-05-08T21:23:01.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.955+0000 7f0ea0c2e700 1 -- 172.21.15.3:0/4157376603 <== osd.0 v2:172.21.15.3:6824/47120 6 ==== osd_map(37..37 src has 1..37) v4 ==== 288+0+0 (crc 0 0 0) 0x7f0e9c0235f0 con 0x7f0e8c039920 2024-05-08T21:23:01.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.957+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.0 v2:172.21.15.3:6824/47120 7 ==== osd_op_reply(51 mds0_sessionmap [delete] v37'4 uv4 ondisk = 0) v8 ==== 159+0+0 (crc 0 0 0) 0x7f0e9c02b040 con 0x7f0e8c039920 2024-05-08T21:23:01.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.958+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 <== osd.0 v2:172.21.15.3:6824/47120 8 ==== osd_op_reply(43 mds0_openfiles.0 [delete] v37'3 uv3 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f0e9c02b040 con 0x7f0e8c039920 2024-05-08T21:23:01.960 INFO:teuthology.orchestra.run.smithi003.stdout:Removed 23 objects 2024-05-08T21:23:01.960 INFO:teuthology.orchestra.run.smithi003.stdout:successfully purged pool cephfs_metadata 2024-05-08T21:23:01.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.958+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x7f0e8c039920 msgr2=0x7f0e8c026370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.958+0000 7f0eacb60080 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x7f0e8c039920 0x7f0e8c026370 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.958+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x7f0e8c010bd0 msgr2=0x7f0e8c01e920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.958+0000 7f0eacb60080 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x7f0e8c010bd0 0x7f0e8c01e920 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.958+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x7f0e8c029910 msgr2=0x7f0e8c00f3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.958+0000 7f0eacb60080 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x7f0e8c029910 0x7f0e8c00f3a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.958+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x5587792ee690 msgr2=0x55877938c9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.958+0000 7f0eacb60080 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x5587792ee690 0x55877938c9a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.958+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x7f0e94028030 msgr2=0x7f0e94057a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.958+0000 7f0eacb60080 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x7f0e94028030 0x7f0e94057a50 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x7f0e9c001630 msgr2=0x7f0e9c001ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0eacb60080 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x7f0e9c001630 0x7f0e9c001ab0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x7f0e9c0203c0 msgr2=0x7f0e9c020840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0eacb60080 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x7f0e9c0203c0 0x7f0e9c020840 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 reap_dead start 2024-05-08T21:23:01.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 reap_dead start 2024-05-08T21:23:01.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0ea3433700 1 -- 172.21.15.3:0/4157376603 reap_dead start 2024-05-08T21:23:01.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0e840695e0 msgr2=0x7f0e8406baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0eacb60080 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0e840695e0 0x7f0e8406baa0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x5587792c1240 tx=0x7f0e8c003000 comp rx=0 tx=0).stop 2024-05-08T21:23:01.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587792bf780 msgr2=0x55877941fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:01.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0eacb60080 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587792bf780 0x55877941fa30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0e94002000 tx=0x7f0e94002fb0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 shutdown_connections 2024-05-08T21:23:01.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0eacb60080 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5587792bf780 0x55877941fa30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:01.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0eacb60080 1 --2- 172.21.15.3:0/4157376603 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f0e840695e0 0x7f0e8406baa0 secure :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0x5587792c1240 tx=0x7f0e8c003000 comp rx=0 tx=0).stop 2024-05-08T21:23:01.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 >> 172.21.15.3:0/4157376603 conn(0x5587792e92d0 msgr2=0x5587792becb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:01.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 shutdown_connections 2024-05-08T21:23:01.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:01.959+0000 7f0eacb60080 1 -- 172.21.15.3:0/4157376603 wait complete. 2024-05-08T21:23:01.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:23:02.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.099+0000 7f17f6563700 1 Processor -- start 2024-05-08T21:23:02.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.099+0000 7f17f6563700 1 -- start start 2024-05-08T21:23:02.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.099+0000 7f17f6563700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17f00cd4e0 0x7f17f006f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:02.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.099+0000 7f17f6563700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f17f00cecf0 0x7f17f015c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:02.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.099+0000 7f17f6563700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f015c280 con 0x7f17f006f130 2024-05-08T21:23:02.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.099+0000 7f17f6563700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17f006e780 con 0x7f17f00cecf0 2024-05-08T21:23:02.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.099+0000 7f17f6563700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f006ea60 con 0x7f17f00cd4e0 2024-05-08T21:23:02.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.100+0000 7f17f4b00700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f17f00cecf0 0x7f17f015c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:02.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.100+0000 7f17effff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17f00cd4e0 0x7f17f006f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:02.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.100+0000 7f17f4b00700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f17f00cecf0 0x7f17f015c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:38998/0 (socket says 172.21.15.3:38998) 2024-05-08T21:23:02.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.100+0000 7f17ef7fe700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f17f006f130 0x7f17f00ce5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:40728/0 (socket says 172.21.15.3:40728) 2024-05-08T21:23:02.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.100+0000 7f17ef7fe700 1 -- 172.21.15.3:0/2242989173 learned_addr learned my addr 172.21.15.3:0/2242989173 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:02.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.100+0000 7f17effff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17f00cd4e0 0x7f17f006f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.3:56346/0 (socket says 172.21.15.3:56346) 2024-05-08T21:23:02.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.100+0000 7f17effff700 1 -- 172.21.15.3:0/2242989173 >> v2:172.21.15.186:3300/0 conn(0x7f17f00cecf0 msgr2=0x7f17f015c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:02.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.100+0000 7f17effff700 1 --2- 172.21.15.3:0/2242989173 >> v2:172.21.15.186:3300/0 conn(0x7f17f00cecf0 0x7f17f015c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.100+0000 7f17effff700 1 -- 172.21.15.3:0/2242989173 >> v1:172.21.15.186:6789/0 conn(0x7f17f006f130 legacy=0x7f17f00ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.100+0000 7f17effff700 1 -- 172.21.15.3:0/2242989173 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f006ed40 con 0x7f17f00cd4e0 2024-05-08T21:23:02.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.100+0000 7f17eeffd700 1 -- 172.21.15.3:0/2242989173 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561166863 0 0) 0x7f17f015c280 con 0x7f17f006f130 2024-05-08T21:23:02.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.100+0000 7f17f4b00700 1 --2- 172.21.15.3:0/2242989173 >> v2:172.21.15.186:3300/0 conn(0x7f17f00cecf0 0x7f17f015c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:02.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.101+0000 7f17effff700 1 --2- 172.21.15.3:0/2242989173 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17f00cd4e0 0x7f17f006f910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f17e000bae0 tx=0x7f17e000bdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc8656857d1ec5c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:02.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.101+0000 7f17eeffd700 1 -- 172.21.15.3:0/2242989173 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e000dc70 con 0x7f17f00cd4e0 2024-05-08T21:23:02.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.101+0000 7f17eeffd700 1 -- 172.21.15.3:0/2242989173 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e000ddd0 con 0x7f17f00cd4e0 2024-05-08T21:23:02.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.101+0000 7f17eeffd700 1 -- 172.21.15.3:0/2242989173 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0010690 con 0x7f17f00cd4e0 2024-05-08T21:23:02.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.101+0000 7f17f6563700 1 -- 172.21.15.3:0/2242989173 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17f00cd4e0 msgr2=0x7f17f006f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.101+0000 7f17f6563700 1 --2- 172.21.15.3:0/2242989173 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17f00cd4e0 0x7f17f006f910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f17e000bae0 tx=0x7f17e000bdf0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.101+0000 7f17f6563700 1 -- 172.21.15.3:0/2242989173 shutdown_connections 2024-05-08T21:23:02.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.101+0000 7f17f6563700 1 --2- 172.21.15.3:0/2242989173 >> v2:172.21.15.186:3300/0 conn(0x7f17f00cecf0 0x7f17f015c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.101+0000 7f17f6563700 1 --2- 172.21.15.3:0/2242989173 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17f00cd4e0 0x7f17f006f910 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.101+0000 7f17f6563700 1 -- 172.21.15.3:0/2242989173 >> 172.21.15.3:0/2242989173 conn(0x7f17f00d0320 msgr2=0x7f17f00d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:02.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.101+0000 7f17f6563700 1 -- 172.21.15.3:0/2242989173 shutdown_connections 2024-05-08T21:23:02.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.101+0000 7f17f6563700 1 -- 172.21.15.3:0/2242989173 wait complete. 2024-05-08T21:23:02.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.103+0000 7f17f6563700 1 Processor -- start 2024-05-08T21:23:02.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.103+0000 7f17f6563700 1 -- start start 2024-05-08T21:23:02.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17f6563700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17f006f130 0x7f17f00aef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:02.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17f6563700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17f00cd4e0 0x7f17f014d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:02.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17f6563700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17f00cecf0 0x7f17f014d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:02.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17f6563700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f00c5b90 con 0x7f17f00cecf0 2024-05-08T21:23:02.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17f6563700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f00c5700 con 0x7f17f006f130 2024-05-08T21:23:02.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17f6563700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f00c4af0 con 0x7f17f00cd4e0 2024-05-08T21:23:02.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17f4b00700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17f00cecf0 0x7f17f014d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:02.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17effff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17f006f130 0x7f17f00aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:02.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17ef7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17f00cd4e0 0x7f17f014d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:02.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17f4b00700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17f00cecf0 0x7f17f014d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54034/0 (socket says 172.21.15.3:54034) 2024-05-08T21:23:02.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17f4b00700 1 -- 172.21.15.3:0/3253964814 learned_addr learned my addr 172.21.15.3:0/3253964814 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:02.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17f4b00700 1 -- 172.21.15.3:0/3253964814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17f00cd4e0 msgr2=0x7f17f014d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17f4b00700 1 --2- 172.21.15.3:0/3253964814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17f00cd4e0 0x7f17f014d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17f4b00700 1 -- 172.21.15.3:0/3253964814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17f006f130 msgr2=0x7f17f00aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17f4b00700 1 --2- 172.21.15.3:0/3253964814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17f006f130 0x7f17f00aef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17f4b00700 1 -- 172.21.15.3:0/3253964814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e8009a70 con 0x7f17f00cecf0 2024-05-08T21:23:02.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17ef7fe700 1 --2- 172.21.15.3:0/3253964814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17f00cd4e0 0x7f17f014d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:02.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17f4b00700 1 --2- 172.21.15.3:0/3253964814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17f00cecf0 0x7f17f014d8a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009e60 tx=0x7f17e800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:02.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.104+0000 7f17ecff9700 1 -- 172.21.15.3:0/3253964814 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e8009cd0 con 0x7f17f00cecf0 2024-05-08T21:23:02.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.105+0000 7f17ecff9700 1 -- 172.21.15.3:0/3253964814 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e800f040 con 0x7f17f00cecf0 2024-05-08T21:23:02.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.105+0000 7f17f6563700 1 -- 172.21.15.3:0/3253964814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e000b740 con 0x7f17f00cecf0 2024-05-08T21:23:02.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.105+0000 7f17ecff9700 1 -- 172.21.15.3:0/3253964814 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e800ccb0 con 0x7f17f00cecf0 2024-05-08T21:23:02.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.105+0000 7f17f6563700 1 -- 172.21.15.3:0/3253964814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f00c50b0 con 0x7f17f00cecf0 2024-05-08T21:23:02.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.106+0000 7f17f6563700 1 -- 172.21.15.3:0/3253964814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f00c39b0 con 0x7f17f00cecf0 2024-05-08T21:23:02.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.106+0000 7f17ecff9700 1 -- 172.21.15.3:0/3253964814 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f17e8014480 con 0x7f17f00cecf0 2024-05-08T21:23:02.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.106+0000 7f17ecff9700 1 --2- 172.21.15.3:0/3253964814 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f17dc06ac90 0x7f17dc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:02.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.107+0000 7f17ecff9700 1 -- 172.21.15.3:0/3253964814 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f17e80898c0 con 0x7f17f00cecf0 2024-05-08T21:23:02.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.107+0000 7f17effff700 1 --2- 172.21.15.3:0/3253964814 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f17dc06ac90 0x7f17dc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:02.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.107+0000 7f17effff700 1 --2- 172.21.15.3:0/3253964814 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f17dc06ac90 0x7f17dc06d150 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f17e0000c00 tx=0x7f17e0013000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:02.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.111+0000 7f17ecff9700 1 -- 172.21.15.3:0/3253964814 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f17e8054a40 con 0x7f17f00cecf0 2024-05-08T21:23:02.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.338+0000 7f17f6563700 1 -- 172.21.15.3:0/3253964814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17f0068000 con 0x7f17f00cecf0 2024-05-08T21:23:02.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.339+0000 7f17ecff9700 1 -- 172.21.15.3:0/3253964814 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16180 (secure 0 0 0) 0x7f17e8058060 con 0x7f17f00cecf0 2024-05-08T21:23:02.341 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:02.341 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":39,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:23:01.967890+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:22:47.853786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-08T21:22:49.153759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6834/3599481370":"2024-05-09T21:23:01.563392+0000","172.21.15.3:6832/3599481370":"2024-05-09T21:23:01.563392+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:23:02.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.341+0000 7f17f6563700 1 -- 172.21.15.3:0/3253964814 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f17dc06ac90 msgr2=0x7f17dc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.341+0000 7f17f6563700 1 --2- 172.21.15.3:0/3253964814 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f17dc06ac90 0x7f17dc06d150 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f17e0000c00 tx=0x7f17e0013000 comp rx=0 tx=0).stop 2024-05-08T21:23:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.341+0000 7f17f6563700 1 -- 172.21.15.3:0/3253964814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17f00cecf0 msgr2=0x7f17f014d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.341+0000 7f17f6563700 1 --2- 172.21.15.3:0/3253964814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17f00cecf0 0x7f17f014d8a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009e60 tx=0x7f17e800c3a0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.342+0000 7f17f6563700 1 -- 172.21.15.3:0/3253964814 shutdown_connections 2024-05-08T21:23:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.342+0000 7f17f6563700 1 --2- 172.21.15.3:0/3253964814 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f17dc06ac90 0x7f17dc06d150 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.342+0000 7f17f6563700 1 --2- 172.21.15.3:0/3253964814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17f00cecf0 0x7f17f014d8a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.342+0000 7f17f6563700 1 --2- 172.21.15.3:0/3253964814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17f006f130 0x7f17f00aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.342+0000 7f17f6563700 1 --2- 172.21.15.3:0/3253964814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17f00cd4e0 0x7f17f014d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.342+0000 7f17f6563700 1 -- 172.21.15.3:0/3253964814 >> 172.21.15.3:0/3253964814 conn(0x7f17f00d0320 msgr2=0x7f17f00d0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.342+0000 7f17f6563700 1 -- 172.21.15.3:0/3253964814 shutdown_connections 2024-05-08T21:23:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.342+0000 7f17f6563700 1 -- 172.21.15.3:0/3253964814 wait complete. 2024-05-08T21:23:02.357 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-08T21:22:47.853786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:22:49.153759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:23:02.358 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:23:02.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.495+0000 7f05a5006700 1 Processor -- start 2024-05-08T21:23:02.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.495+0000 7f05a5006700 1 -- start start 2024-05-08T21:23:02.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.496+0000 7f05a5006700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f05a00d3d20 0x7f05a00b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:02.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.496+0000 7f05a5006700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a00cf610 con 0x7f05a00b3fc0 2024-05-08T21:23:02.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.496+0000 7f05a5006700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a00d1fc0 con 0x7f05a00cff30 2024-05-08T21:23:02.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.496+0000 7f05a5006700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05a00d22a0 con 0x7f05a00d3d20 2024-05-08T21:23:02.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.496+0000 7f059f59e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f05a00b3fc0 0x7f05a00b4430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:42824/0 (socket says 172.21.15.3:42824) 2024-05-08T21:23:02.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.496+0000 7f059f59e700 1 -- 172.21.15.3:0/2768638003 learned_addr learned my addr 172.21.15.3:0/2768638003 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:02.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.497+0000 7f059e59c700 1 --2- 172.21.15.3:0/2768638003 >> v2:172.21.15.186:3300/0 conn(0x7f05a00d3d20 0x7f05a00b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:02.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.497+0000 7f059dd9b700 1 -- 172.21.15.3:0/2768638003 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548716445 0 0) 0x7f05a00cf610 con 0x7f05a00b3fc0 2024-05-08T21:23:02.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.497+0000 7f059dd9b700 1 -- 172.21.15.3:0/2768638003 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05880035b0 con 0x7f05a00b3fc0 2024-05-08T21:23:02.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.497+0000 7f059e59c700 1 -- 172.21.15.3:0/2768638003 >> v1:172.21.15.186:6789/0 conn(0x7f05a00cff30 legacy=0x7f05a00d0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.497+0000 7f059e59c700 1 -- 172.21.15.3:0/2768638003 >> v1:172.21.15.3:6789/0 conn(0x7f05a00b3fc0 legacy=0x7f05a00b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.497+0000 7f059e59c700 1 -- 172.21.15.3:0/2768638003 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a00d2580 con 0x7f05a00d3d20 2024-05-08T21:23:02.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.497+0000 7f059e59c700 1 --2- 172.21.15.3:0/2768638003 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05a00d3d20 0x7f05a00b3a80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0594009840 tx=0x7f0594009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35a7653f2b337d31 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:02.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.497+0000 7f059dd9b700 1 -- 172.21.15.3:0/2768638003 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594011830 con 0x7f05a00d3d20 2024-05-08T21:23:02.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.497+0000 7f059dd9b700 1 -- 172.21.15.3:0/2768638003 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0594011990 con 0x7f05a00d3d20 2024-05-08T21:23:02.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.497+0000 7f059dd9b700 1 -- 172.21.15.3:0/2768638003 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594011d20 con 0x7f05a00d3d20 2024-05-08T21:23:02.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.498+0000 7f05a5006700 1 -- 172.21.15.3:0/2768638003 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05a00d3d20 msgr2=0x7f05a00b3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.498+0000 7f05a5006700 1 --2- 172.21.15.3:0/2768638003 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05a00d3d20 0x7f05a00b3a80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0594009840 tx=0x7f0594009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:02.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.498+0000 7f05a5006700 1 -- 172.21.15.3:0/2768638003 shutdown_connections 2024-05-08T21:23:02.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.498+0000 7f05a5006700 1 --2- 172.21.15.3:0/2768638003 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05a00d3d20 0x7f05a00b3a80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.498+0000 7f05a5006700 1 -- 172.21.15.3:0/2768638003 >> 172.21.15.3:0/2768638003 conn(0x7f05a008afe0 msgr2=0x7f05a00d3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:02.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.498+0000 7f05a5006700 1 -- 172.21.15.3:0/2768638003 shutdown_connections 2024-05-08T21:23:02.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.498+0000 7f05a5006700 1 -- 172.21.15.3:0/2768638003 wait complete. 2024-05-08T21:23:02.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.500+0000 7f05a5006700 1 Processor -- start 2024-05-08T21:23:02.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.500+0000 7f05a5006700 1 -- start start 2024-05-08T21:23:02.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.500+0000 7f05a5006700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05a00b3fc0 0x7f05a015b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:02.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.500+0000 7f05a5006700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05a00cff30 0x7f05a00b6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:02.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.500+0000 7f05a5006700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05a00d3d20 0x7f05a00b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:02.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.500+0000 7f05a5006700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a00b78c0 con 0x7f05a00cff30 2024-05-08T21:23:02.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.500+0000 7f05a5006700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a015ba90 con 0x7f05a00d3d20 2024-05-08T21:23:02.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.500+0000 7f05a5006700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a00b6cb0 con 0x7f05a00b3fc0 2024-05-08T21:23:02.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.501+0000 7f059e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05a00cff30 0x7f05a00b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:02.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.501+0000 7f059e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05a00cff30 0x7f05a00b6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54036/0 (socket says 172.21.15.3:54036) 2024-05-08T21:23:02.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.501+0000 7f059e59c700 1 -- 172.21.15.3:0/1690107699 learned_addr learned my addr 172.21.15.3:0/1690107699 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:02.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.501+0000 7f059f59e700 1 --2- 172.21.15.3:0/1690107699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05a00d3d20 0x7f05a00b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:02.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.501+0000 7f059ed9d700 1 --2- 172.21.15.3:0/1690107699 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05a00b3fc0 0x7f05a015b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:02.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.501+0000 7f059e59c700 1 -- 172.21.15.3:0/1690107699 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05a00b3fc0 msgr2=0x7f05a015b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.501+0000 7f059e59c700 1 --2- 172.21.15.3:0/1690107699 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05a00b3fc0 0x7f05a015b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.501+0000 7f059e59c700 1 -- 172.21.15.3:0/1690107699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05a00d3d20 msgr2=0x7f05a00b8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.501+0000 7f059e59c700 1 --2- 172.21.15.3:0/1690107699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05a00d3d20 0x7f05a00b8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.501+0000 7f059e59c700 1 -- 172.21.15.3:0/1690107699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05940094a0 con 0x7f05a00cff30 2024-05-08T21:23:02.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.501+0000 7f059e59c700 1 --2- 172.21.15.3:0/1690107699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05a00cff30 0x7f05a00b6230 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0594009fd0 tx=0x7f0594004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:02.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.501+0000 7f05877fe700 1 -- 172.21.15.3:0/1690107699 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05940038a0 con 0x7f05a00cff30 2024-05-08T21:23:02.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.501+0000 7f05877fe700 1 -- 172.21.15.3:0/1690107699 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059401f070 con 0x7f05a00cff30 2024-05-08T21:23:02.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.501+0000 7f05a5006700 1 -- 172.21.15.3:0/1690107699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a00b6f90 con 0x7f05a00cff30 2024-05-08T21:23:02.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.501+0000 7f05877fe700 1 -- 172.21.15.3:0/1690107699 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594003c40 con 0x7f05a00cff30 2024-05-08T21:23:02.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.501+0000 7f05a5006700 1 -- 172.21.15.3:0/1690107699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a00b7550 con 0x7f05a00cff30 2024-05-08T21:23:02.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.502+0000 7f05a5006700 1 -- 172.21.15.3:0/1690107699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a00b7710 con 0x7f05a00cff30 2024-05-08T21:23:02.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.503+0000 7f05877fe700 1 -- 172.21.15.3:0/1690107699 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0594026480 con 0x7f05a00cff30 2024-05-08T21:23:02.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.503+0000 7f05877fe700 1 --2- 172.21.15.3:0/1690107699 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f058c06acc0 0x7f058c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:02.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.503+0000 7f05877fe700 1 -- 172.21.15.3:0/1690107699 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0594021070 con 0x7f05a00cff30 2024-05-08T21:23:02.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.504+0000 7f059ed9d700 1 --2- 172.21.15.3:0/1690107699 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f058c06acc0 0x7f058c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:02.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.504+0000 7f059ed9d700 1 --2- 172.21.15.3:0/1690107699 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f058c06acc0 0x7f058c06d180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0590000f60 tx=0x7f0590009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:02.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.508+0000 7f05877fe700 1 -- 172.21.15.3:0/1690107699 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0594055d70 con 0x7f05a00cff30 2024-05-08T21:23:02.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.731+0000 7f05a5006700 1 -- 172.21.15.3:0/1690107699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f05a0074bc0 con 0x7f05a00cff30 2024-05-08T21:23:02.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.732+0000 7f05877fe700 1 -- 172.21.15.3:0/1690107699 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) v1 ==== 93+0+10 (secure 0 0 0) 0x7f059402b0a0 con 0x7f05a00cff30 2024-05-08T21:23:02.733 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:23:02.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.734+0000 7f05a5006700 1 -- 172.21.15.3:0/1690107699 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f058c06acc0 msgr2=0x7f058c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.734+0000 7f05a5006700 1 --2- 172.21.15.3:0/1690107699 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f058c06acc0 0x7f058c06d180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0590000f60 tx=0x7f0590009040 comp rx=0 tx=0).stop 2024-05-08T21:23:02.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.734+0000 7f05a5006700 1 -- 172.21.15.3:0/1690107699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05a00cff30 msgr2=0x7f05a00b6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.734+0000 7f05a5006700 1 --2- 172.21.15.3:0/1690107699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05a00cff30 0x7f05a00b6230 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0594009fd0 tx=0x7f0594004750 comp rx=0 tx=0).stop 2024-05-08T21:23:02.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.734+0000 7f05a5006700 1 -- 172.21.15.3:0/1690107699 shutdown_connections 2024-05-08T21:23:02.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.734+0000 7f05a5006700 1 --2- 172.21.15.3:0/1690107699 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05a00b3fc0 0x7f05a015b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.734+0000 7f05a5006700 1 --2- 172.21.15.3:0/1690107699 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f058c06acc0 0x7f058c06d180 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.734+0000 7f05a5006700 1 --2- 172.21.15.3:0/1690107699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05a00d3d20 0x7f05a00b8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.734+0000 7f05a5006700 1 --2- 172.21.15.3:0/1690107699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05a00cff30 0x7f05a00b6230 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.734+0000 7f05a5006700 1 -- 172.21.15.3:0/1690107699 >> 172.21.15.3:0/1690107699 conn(0x7f05a008afe0 msgr2=0x7f05a00b50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:02.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.734+0000 7f05a5006700 1 -- 172.21.15.3:0/1690107699 shutdown_connections 2024-05-08T21:23:02.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.734+0000 7f05a5006700 1 -- 172.21.15.3:0/1690107699 wait complete. 2024-05-08T21:23:02.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-08T21:23:02.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.886+0000 7f6b7b37c700 1 Processor -- start 2024-05-08T21:23:02.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.887+0000 7f6b7b37c700 1 -- start start 2024-05-08T21:23:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.887+0000 7f6b7b37c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b7406edf0 0x7f6b74159f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.888+0000 7f6b7b37c700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b740cc980 con 0x7f6b7415bb20 2024-05-08T21:23:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.888+0000 7f6b7b37c700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b740d00f0 con 0x7f6b740cfd10 2024-05-08T21:23:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.888+0000 7f6b7b37c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b740ccbd0 con 0x7f6b7406edf0 2024-05-08T21:23:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.888+0000 7f6b78917700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6b7415bb20 0x7f6b7406e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:42836/0 (socket says 172.21.15.3:42836) 2024-05-08T21:23:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.888+0000 7f6b78917700 1 -- 172.21.15.3:0/3924315607 learned_addr learned my addr 172.21.15.3:0/3924315607 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.888+0000 7f6b79919700 1 --2- 172.21.15.3:0/3924315607 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b7406edf0 0x7f6b74159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.888+0000 7f6b6bfff700 1 -- 172.21.15.3:0/3924315607 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684833026 0 0) 0x7f6b740cc980 con 0x7f6b7415bb20 2024-05-08T21:23:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.888+0000 7f6b6bfff700 1 -- 172.21.15.3:0/3924315607 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b600035b0 con 0x7f6b7415bb20 2024-05-08T21:23:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.888+0000 7f6b6bfff700 1 -- 172.21.15.3:0/3924315607 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768172033 0 0) 0x7f6b740d00f0 con 0x7f6b740cfd10 2024-05-08T21:23:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.888+0000 7f6b6bfff700 1 -- 172.21.15.3:0/3924315607 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b740cc980 con 0x7f6b740cfd10 2024-05-08T21:23:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.888+0000 7f6b6bfff700 1 -- 172.21.15.3:0/3924315607 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3493462564 0 0) 0x7f6b600035b0 con 0x7f6b7415bb20 2024-05-08T21:23:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.888+0000 7f6b79919700 1 -- 172.21.15.3:0/3924315607 >> v1:172.21.15.186:6789/0 conn(0x7f6b740cfd10 legacy=0x7f6b7406dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.888+0000 7f6b79919700 1 -- 172.21.15.3:0/3924315607 >> v1:172.21.15.3:6789/0 conn(0x7f6b7415bb20 legacy=0x7f6b7406e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.888+0000 7f6b79919700 1 -- 172.21.15.3:0/3924315607 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b740cceb0 con 0x7f6b7406edf0 2024-05-08T21:23:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.889+0000 7f6b79919700 1 --2- 172.21.15.3:0/3924315607 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b7406edf0 0x7f6b74159f80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009dc0 tx=0x7f6b6c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3ff73923fff2c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.889+0000 7f6b6bfff700 1 -- 172.21.15.3:0/3924315607 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c00e070 con 0x7f6b7406edf0 2024-05-08T21:23:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.889+0000 7f6b6bfff700 1 -- 172.21.15.3:0/3924315607 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6c003a20 con 0x7f6b7406edf0 2024-05-08T21:23:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.889+0000 7f6b6bfff700 1 -- 172.21.15.3:0/3924315607 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b64001670 con 0x7f6b7406edf0 2024-05-08T21:23:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.889+0000 7f6b7b37c700 1 -- 172.21.15.3:0/3924315607 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b7406edf0 msgr2=0x7f6b74159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.889+0000 7f6b7b37c700 1 --2- 172.21.15.3:0/3924315607 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b7406edf0 0x7f6b74159f80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009dc0 tx=0x7f6b6c00b330 comp rx=0 tx=0).stop 2024-05-08T21:23:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.889+0000 7f6b7b37c700 1 -- 172.21.15.3:0/3924315607 shutdown_connections 2024-05-08T21:23:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.889+0000 7f6b7b37c700 1 --2- 172.21.15.3:0/3924315607 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b7406edf0 0x7f6b74159f80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.889+0000 7f6b7b37c700 1 -- 172.21.15.3:0/3924315607 >> 172.21.15.3:0/3924315607 conn(0x7f6b74099a50 msgr2=0x7f6b740d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.889+0000 7f6b7b37c700 1 -- 172.21.15.3:0/3924315607 shutdown_connections 2024-05-08T21:23:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.890+0000 7f6b7b37c700 1 -- 172.21.15.3:0/3924315607 wait complete. 2024-05-08T21:23:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.891+0000 7f6b7b37c700 1 Processor -- start 2024-05-08T21:23:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.891+0000 7f6b7b37c700 1 -- start start 2024-05-08T21:23:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.892+0000 7f6b7b37c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b7406edf0 0x7f6b740c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.892+0000 7f6b7b37c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b740cfd10 0x7f6b740b0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.892+0000 7f6b7b37c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b7415bb20 0x7f6b740b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.892+0000 7f6b7b37c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b740b4ce0 con 0x7f6b7415bb20 2024-05-08T21:23:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.892+0000 7f6b7b37c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b740b4160 con 0x7f6b7406edf0 2024-05-08T21:23:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.892+0000 7f6b7b37c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b740b4440 con 0x7f6b740cfd10 2024-05-08T21:23:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.892+0000 7f6b79919700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b7415bb20 0x7f6b740b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.892+0000 7f6b79919700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b7415bb20 0x7f6b740b35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54048/0 (socket says 172.21.15.3:54048) 2024-05-08T21:23:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.892+0000 7f6b79919700 1 -- 172.21.15.3:0/2862585258 learned_addr learned my addr 172.21.15.3:0/2862585258 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.893+0000 7f6b78917700 1 --2- 172.21.15.3:0/2862585258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b740cfd10 0x7f6b740b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.893+0000 7f6b79118700 1 --2- 172.21.15.3:0/2862585258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b7406edf0 0x7f6b740c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.893+0000 7f6b79919700 1 -- 172.21.15.3:0/2862585258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b740cfd10 msgr2=0x7f6b740b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.893+0000 7f6b79919700 1 --2- 172.21.15.3:0/2862585258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b740cfd10 0x7f6b740b0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.893+0000 7f6b79919700 1 -- 172.21.15.3:0/2862585258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b7406edf0 msgr2=0x7f6b740c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.893+0000 7f6b79919700 1 --2- 172.21.15.3:0/2862585258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b7406edf0 0x7f6b740c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.893+0000 7f6b79919700 1 -- 172.21.15.3:0/2862585258 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c009a70 con 0x7f6b7415bb20 2024-05-08T21:23:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.893+0000 7f6b79919700 1 --2- 172.21.15.3:0/2862585258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b7415bb20 0x7f6b740b35b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009d90 tx=0x7f6b6c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.893+0000 7f6b69ffb700 1 -- 172.21.15.3:0/2862585258 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c0046f0 con 0x7f6b7415bb20 2024-05-08T21:23:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.893+0000 7f6b69ffb700 1 -- 172.21.15.3:0/2862585258 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6c004850 con 0x7f6b7415bb20 2024-05-08T21:23:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.893+0000 7f6b7b37c700 1 -- 172.21.15.3:0/2862585258 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b740b4720 con 0x7f6b7415bb20 2024-05-08T21:23:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.893+0000 7f6b69ffb700 1 -- 172.21.15.3:0/2862585258 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c01d670 con 0x7f6b7415bb20 2024-05-08T21:23:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.893+0000 7f6b7b37c700 1 -- 172.21.15.3:0/2862585258 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b740b3090 con 0x7f6b7415bb20 2024-05-08T21:23:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.894+0000 7f6b7b37c700 1 -- 172.21.15.3:0/2862585258 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b74074bc0 con 0x7f6b7415bb20 2024-05-08T21:23:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.895+0000 7f6b69ffb700 1 -- 172.21.15.3:0/2862585258 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6b6c01d7d0 con 0x7f6b7415bb20 2024-05-08T21:23:02.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.895+0000 7f6b69ffb700 1 --2- 172.21.15.3:0/2862585258 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6b5806ac90 0x7f6b5806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:02.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.895+0000 7f6b69ffb700 1 -- 172.21.15.3:0/2862585258 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6b6c08b100 con 0x7f6b7415bb20 2024-05-08T21:23:02.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.896+0000 7f6b79118700 1 --2- 172.21.15.3:0/2862585258 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6b5806ac90 0x7f6b5806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:02.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.896+0000 7f6b79118700 1 --2- 172.21.15.3:0/2862585258 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6b5806ac90 0x7f6b5806d150 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6b700011e0 tx=0x7f6b70009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:02.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:02.900+0000 7f6b69ffb700 1 -- 172.21.15.3:0/2862585258 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6b6c056270 con 0x7f6b7415bb20 2024-05-08T21:23:03.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.118+0000 7f6b7b37c700 1 -- 172.21.15.3:0/2862585258 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6b740ca910 con 0x7f6b7415bb20 2024-05-08T21:23:03.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.118+0000 7f6b69ffb700 1 -- 172.21.15.3:0/2862585258 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6b6c059890 con 0x7f6b7415bb20 2024-05-08T21:23:03.120 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:23:03.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.120+0000 7f6b7b37c700 1 -- 172.21.15.3:0/2862585258 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6b5806ac90 msgr2=0x7f6b5806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:03.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.120+0000 7f6b7b37c700 1 --2- 172.21.15.3:0/2862585258 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6b5806ac90 0x7f6b5806d150 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6b700011e0 tx=0x7f6b70009040 comp rx=0 tx=0).stop 2024-05-08T21:23:03.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.120+0000 7f6b7b37c700 1 -- 172.21.15.3:0/2862585258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b7415bb20 msgr2=0x7f6b740b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:03.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.120+0000 7f6b7b37c700 1 --2- 172.21.15.3:0/2862585258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b7415bb20 0x7f6b740b35b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009d90 tx=0x7f6b6c004ad0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.120+0000 7f6b7b37c700 1 -- 172.21.15.3:0/2862585258 shutdown_connections 2024-05-08T21:23:03.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.120+0000 7f6b7b37c700 1 --2- 172.21.15.3:0/2862585258 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6b5806ac90 0x7f6b5806d150 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.120+0000 7f6b7b37c700 1 --2- 172.21.15.3:0/2862585258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b7415bb20 0x7f6b740b35b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.120+0000 7f6b7b37c700 1 --2- 172.21.15.3:0/2862585258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b7406edf0 0x7f6b740c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.120+0000 7f6b7b37c700 1 --2- 172.21.15.3:0/2862585258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b740cfd10 0x7f6b740b0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.120+0000 7f6b7b37c700 1 -- 172.21.15.3:0/2862585258 >> 172.21.15.3:0/2862585258 conn(0x7f6b74099a50 msgr2=0x7f6b7406a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:03.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.121+0000 7f6b7b37c700 1 -- 172.21.15.3:0/2862585258 shutdown_connections 2024-05-08T21:23:03.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.121+0000 7f6b7b37c700 1 -- 172.21.15.3:0/2862585258 wait complete. 2024-05-08T21:23:03.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:23:03.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.273+0000 7f48caf9b700 1 Processor -- start 2024-05-08T21:23:03.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.273+0000 7f48caf9b700 1 -- start start 2024-05-08T21:23:03.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48caf9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f48c40cbf50 0x7f48c4161d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:03.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48caf9b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c4162240 0x7f48c415e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:03.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48caf9b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f48c40d0f10 0x7f48c415e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:03.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48caf9b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c40cf970 con 0x7f48c40cbf50 2024-05-08T21:23:03.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48caf9b700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c40cfc50 con 0x7f48c40d0f10 2024-05-08T21:23:03.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48caf9b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c40cff30 con 0x7f48c4162240 2024-05-08T21:23:03.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48c8d37700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f48c40cbf50 0x7f48c4161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:03.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48c8d37700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f48c40cbf50 0x7f48c4161d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54064/0 (socket says 172.21.15.3:54064) 2024-05-08T21:23:03.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48c8d37700 1 -- 172.21.15.3:0/2954702069 learned_addr learned my addr 172.21.15.3:0/2954702069 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:03.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48c9538700 1 --2- 172.21.15.3:0/2954702069 >> v2:172.21.15.186:3300/0 conn(0x7f48c40d0f10 0x7f48c415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:03.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48c3fff700 1 --2- 172.21.15.3:0/2954702069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c4162240 0x7f48c415e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:03.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48c8d37700 1 -- 172.21.15.3:0/2954702069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c4162240 msgr2=0x7f48c415e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:03.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48c8d37700 1 --2- 172.21.15.3:0/2954702069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c4162240 0x7f48c415e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48c8d37700 1 -- 172.21.15.3:0/2954702069 >> v2:172.21.15.186:3300/0 conn(0x7f48c40d0f10 msgr2=0x7f48c415e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:03.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48c8d37700 1 --2- 172.21.15.3:0/2954702069 >> v2:172.21.15.186:3300/0 conn(0x7f48c40d0f10 0x7f48c415e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48c8d37700 1 -- 172.21.15.3:0/2954702069 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c41617e0 con 0x7f48c40cbf50 2024-05-08T21:23:03.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48c8d37700 1 --2- 172.21.15.3:0/2954702069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c40cbf50 0x7f48c4161d00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f48c4072c50 tx=0x7f48b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7668438414b1d1db server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:03.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48c37fe700 1 -- 172.21.15.3:0/2954702069 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b400b870 con 0x7f48c40cbf50 2024-05-08T21:23:03.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48c37fe700 1 -- 172.21.15.3:0/2954702069 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b400b9d0 con 0x7f48c40cbf50 2024-05-08T21:23:03.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.274+0000 7f48c37fe700 1 -- 172.21.15.3:0/2954702069 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b400bd60 con 0x7f48c40cbf50 2024-05-08T21:23:03.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.275+0000 7f48caf9b700 1 -- 172.21.15.3:0/2954702069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c40cbf50 msgr2=0x7f48c4161d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:03.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.275+0000 7f48caf9b700 1 --2- 172.21.15.3:0/2954702069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c40cbf50 0x7f48c4161d00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f48c4072c50 tx=0x7f48b4009bd0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.275+0000 7f48caf9b700 1 -- 172.21.15.3:0/2954702069 shutdown_connections 2024-05-08T21:23:03.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.275+0000 7f48caf9b700 1 --2- 172.21.15.3:0/2954702069 >> v2:172.21.15.186:3300/0 conn(0x7f48c40d0f10 0x7f48c415e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.275+0000 7f48caf9b700 1 --2- 172.21.15.3:0/2954702069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c40cbf50 0x7f48c4161d00 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.275+0000 7f48caf9b700 1 --2- 172.21.15.3:0/2954702069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c4162240 0x7f48c415e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.275+0000 7f48caf9b700 1 -- 172.21.15.3:0/2954702069 >> 172.21.15.3:0/2954702069 conn(0x7f48c4099a50 msgr2=0x7f48c40d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:03.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.275+0000 7f48caf9b700 1 -- 172.21.15.3:0/2954702069 shutdown_connections 2024-05-08T21:23:03.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.275+0000 7f48caf9b700 1 -- 172.21.15.3:0/2954702069 wait complete. 2024-05-08T21:23:03.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.276+0000 7f48caf9b700 1 Processor -- start 2024-05-08T21:23:03.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.277+0000 7f48caf9b700 1 -- start start 2024-05-08T21:23:03.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.277+0000 7f48caf9b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c40cbf50 0x7f48c41593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:03.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.277+0000 7f48caf9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c40d0f10 0x7f48c41e7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:03.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.277+0000 7f48caf9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48c4162240 0x7f48c40b2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:03.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.277+0000 7f48caf9b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c40b4200 con 0x7f48c40d0f10 2024-05-08T21:23:03.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.277+0000 7f48caf9b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c40b35f0 con 0x7f48c4162240 2024-05-08T21:23:03.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48caf9b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c40b38d0 con 0x7f48c40cbf50 2024-05-08T21:23:03.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48c3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c40d0f10 0x7f48c41e7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:03.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48c3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c40d0f10 0x7f48c41e7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54070/0 (socket says 172.21.15.3:54070) 2024-05-08T21:23:03.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48c3fff700 1 -- 172.21.15.3:0/2750976118 learned_addr learned my addr 172.21.15.3:0/2750976118 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:03.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48c8d37700 1 --2- 172.21.15.3:0/2750976118 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c40cbf50 0x7f48c41593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:03.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48c9538700 1 --2- 172.21.15.3:0/2750976118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48c4162240 0x7f48c40b2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:03.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48c3fff700 1 -- 172.21.15.3:0/2750976118 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c40cbf50 msgr2=0x7f48c41593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:03.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48c3fff700 1 --2- 172.21.15.3:0/2750976118 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c40cbf50 0x7f48c41593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48c3fff700 1 -- 172.21.15.3:0/2750976118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48c4162240 msgr2=0x7f48c40b2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:03.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48c3fff700 1 --2- 172.21.15.3:0/2750976118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48c4162240 0x7f48c40b2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48c3fff700 1 -- 172.21.15.3:0/2750976118 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b4009550 con 0x7f48c40d0f10 2024-05-08T21:23:03.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48c3fff700 1 --2- 172.21.15.3:0/2750976118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c40d0f10 0x7f48c41e7e60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00c7f0 tx=0x7f48ac00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:03.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48c17fa700 1 -- 172.21.15.3:0/2750976118 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac004220 con 0x7f48c40d0f10 2024-05-08T21:23:03.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48c17fa700 1 -- 172.21.15.3:0/2750976118 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ac004380 con 0x7f48c40d0f10 2024-05-08T21:23:03.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48caf9b700 1 -- 172.21.15.3:0/2750976118 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c406e030 con 0x7f48c40d0f10 2024-05-08T21:23:03.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48c17fa700 1 -- 172.21.15.3:0/2750976118 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac0036a0 con 0x7f48c40d0f10 2024-05-08T21:23:03.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.278+0000 7f48caf9b700 1 -- 172.21.15.3:0/2750976118 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c40b3e90 con 0x7f48c40d0f10 2024-05-08T21:23:03.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.279+0000 7f48caf9b700 1 -- 172.21.15.3:0/2750976118 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c4074bc0 con 0x7f48c40d0f10 2024-05-08T21:23:03.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.281+0000 7f48c17fa700 1 -- 172.21.15.3:0/2750976118 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f48ac003800 con 0x7f48c40d0f10 2024-05-08T21:23:03.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.281+0000 7f48c17fa700 1 --2- 172.21.15.3:0/2750976118 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48a406ac90 0x7f48a406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:03.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.281+0000 7f48c17fa700 1 -- 172.21.15.3:0/2750976118 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f48ac089300 con 0x7f48c40d0f10 2024-05-08T21:23:03.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.281+0000 7f48c8d37700 1 --2- 172.21.15.3:0/2750976118 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48a406ac90 0x7f48a406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:03.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.282+0000 7f48c8d37700 1 --2- 172.21.15.3:0/2750976118 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48a406ac90 0x7f48a406d150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f48b4014040 tx=0x7f48b4008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:03.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.285+0000 7f48c17fa700 1 -- 172.21.15.3:0/2750976118 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f48ac054460 con 0x7f48c40d0f10 2024-05-08T21:23:03.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.501+0000 7f48caf9b700 1 -- 172.21.15.3:0/2750976118 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f48c40b62b0 con 0x7f48c40d0f10 2024-05-08T21:23:03.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.501+0000 7f48c17fa700 1 -- 172.21.15.3:0/2750976118 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) v1 ==== 100+0+11 (secure 0 0 0) 0x7f48ac01e020 con 0x7f48c40d0f10 2024-05-08T21:23:03.503 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:23:03.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.503+0000 7f48caf9b700 1 -- 172.21.15.3:0/2750976118 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48a406ac90 msgr2=0x7f48a406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:03.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.503+0000 7f48caf9b700 1 --2- 172.21.15.3:0/2750976118 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48a406ac90 0x7f48a406d150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f48b4014040 tx=0x7f48b4008030 comp rx=0 tx=0).stop 2024-05-08T21:23:03.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.503+0000 7f48caf9b700 1 -- 172.21.15.3:0/2750976118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c40d0f10 msgr2=0x7f48c41e7e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:03.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.503+0000 7f48caf9b700 1 --2- 172.21.15.3:0/2750976118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c40d0f10 0x7f48c41e7e60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00c7f0 tx=0x7f48ac00cb00 comp rx=0 tx=0).stop 2024-05-08T21:23:03.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.504+0000 7f48caf9b700 1 -- 172.21.15.3:0/2750976118 shutdown_connections 2024-05-08T21:23:03.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.504+0000 7f48caf9b700 1 --2- 172.21.15.3:0/2750976118 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48c40cbf50 0x7f48c41593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.504+0000 7f48caf9b700 1 --2- 172.21.15.3:0/2750976118 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48a406ac90 0x7f48a406d150 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.504+0000 7f48caf9b700 1 --2- 172.21.15.3:0/2750976118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48c4162240 0x7f48c40b2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.504+0000 7f48caf9b700 1 --2- 172.21.15.3:0/2750976118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48c40d0f10 0x7f48c41e7e60 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.504+0000 7f48caf9b700 1 -- 172.21.15.3:0/2750976118 >> 172.21.15.3:0/2750976118 conn(0x7f48c4099a50 msgr2=0x7f48c40ae570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:03.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.504+0000 7f48caf9b700 1 -- 172.21.15.3:0/2750976118 shutdown_connections 2024-05-08T21:23:03.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.504+0000 7f48caf9b700 1 -- 172.21.15.3:0/2750976118 wait complete. 2024-05-08T21:23:03.519 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'recovery_fs' 2024-05-08T21:23:03.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create recovery_fs_metadata --pg_num_min 64 2024-05-08T21:23:03.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.659+0000 7ff16548a700 1 Processor -- start 2024-05-08T21:23:03.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.659+0000 7ff16548a700 1 -- start start 2024-05-08T21:23:03.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.660+0000 7ff16548a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1600cff90 0x7ff1600d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:03.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.660+0000 7ff16548a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff16015dc50 0x7ff1600d11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:03.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.660+0000 7ff16548a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff1600d41b0 0x7ff1601613f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:03.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.660+0000 7ff16548a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff16015eae0 con 0x7ff16015dc50 2024-05-08T21:23:03.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.660+0000 7ff16548a700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1600d08b0 con 0x7ff1600d41b0 2024-05-08T21:23:03.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.660+0000 7ff16548a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff16015e140 con 0x7ff1600cff90 2024-05-08T21:23:03.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.660+0000 7ff15e7fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff16015dc50 0x7ff1600d11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:03.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.660+0000 7ff15e7fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff16015dc50 0x7ff1600d11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54084/0 (socket says 172.21.15.3:54084) 2024-05-08T21:23:03.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.660+0000 7ff15e7fc700 1 -- 172.21.15.3:0/2705512623 learned_addr learned my addr 172.21.15.3:0/2705512623 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:03.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.660+0000 7ff15f7fe700 1 --2- 172.21.15.3:0/2705512623 >> v2:172.21.15.186:3300/0 conn(0x7ff1600d41b0 0x7ff1601613f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:03.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.660+0000 7ff15effd700 1 --2- 172.21.15.3:0/2705512623 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1600cff90 0x7ff1600d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:03.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.660+0000 7ff15e7fc700 1 -- 172.21.15.3:0/2705512623 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1600cff90 msgr2=0x7ff1600d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:03.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.660+0000 7ff15e7fc700 1 --2- 172.21.15.3:0/2705512623 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1600cff90 0x7ff1600d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.660+0000 7ff15e7fc700 1 -- 172.21.15.3:0/2705512623 >> v2:172.21.15.186:3300/0 conn(0x7ff1600d41b0 msgr2=0x7ff1601613f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:03.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.660+0000 7ff15e7fc700 1 --2- 172.21.15.3:0/2705512623 >> v2:172.21.15.186:3300/0 conn(0x7ff1600d41b0 0x7ff1601613f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.660+0000 7ff15e7fc700 1 -- 172.21.15.3:0/2705512623 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16015e420 con 0x7ff16015dc50 2024-05-08T21:23:03.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.661+0000 7ff15e7fc700 1 --2- 172.21.15.3:0/2705512623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff16015dc50 0x7ff1600d11e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff1600cde90 tx=0x7ff150009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881e4f415b5382f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:03.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.661+0000 7ff15dffb700 1 -- 172.21.15.3:0/2705512623 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff150011830 con 0x7ff16015dc50 2024-05-08T21:23:03.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.661+0000 7ff15dffb700 1 -- 172.21.15.3:0/2705512623 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff150011990 con 0x7ff16015dc50 2024-05-08T21:23:03.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.661+0000 7ff15dffb700 1 -- 172.21.15.3:0/2705512623 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff150011d20 con 0x7ff16015dc50 2024-05-08T21:23:03.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.661+0000 7ff16548a700 1 -- 172.21.15.3:0/2705512623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff16015dc50 msgr2=0x7ff1600d11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:03.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.661+0000 7ff16548a700 1 --2- 172.21.15.3:0/2705512623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff16015dc50 0x7ff1600d11e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff1600cde90 tx=0x7ff150009bd0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.661+0000 7ff16548a700 1 -- 172.21.15.3:0/2705512623 shutdown_connections 2024-05-08T21:23:03.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.661+0000 7ff16548a700 1 --2- 172.21.15.3:0/2705512623 >> v2:172.21.15.186:3300/0 conn(0x7ff1600d41b0 0x7ff1601613f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.661+0000 7ff16548a700 1 --2- 172.21.15.3:0/2705512623 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1600cff90 0x7ff1600d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.661+0000 7ff16548a700 1 --2- 172.21.15.3:0/2705512623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff16015dc50 0x7ff1600d11e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.661+0000 7ff16548a700 1 -- 172.21.15.3:0/2705512623 >> 172.21.15.3:0/2705512623 conn(0x7ff16008afe0 msgr2=0x7ff1600d3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:03.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.661+0000 7ff16548a700 1 -- 172.21.15.3:0/2705512623 shutdown_connections 2024-05-08T21:23:03.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.661+0000 7ff16548a700 1 -- 172.21.15.3:0/2705512623 wait complete. 2024-05-08T21:23:03.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.663+0000 7ff16548a700 1 Processor -- start 2024-05-08T21:23:03.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.663+0000 7ff16548a700 1 -- start start 2024-05-08T21:23:03.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff16548a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1600cff90 0x7ff16014b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:03.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff16548a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1600d41b0 0x7ff1600ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:03.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff16548a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff16015dc50 0x7ff1600b1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:03.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff16548a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1600cbdf0 con 0x7ff16015dc50 2024-05-08T21:23:03.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff16548a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1600cb1e0 con 0x7ff1600d41b0 2024-05-08T21:23:03.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff16548a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1600cb4c0 con 0x7ff1600cff90 2024-05-08T21:23:03.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff15f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff16015dc50 0x7ff1600b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:03.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff15f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff16015dc50 0x7ff1600b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54096/0 (socket says 172.21.15.3:54096) 2024-05-08T21:23:03.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff15f7fe700 1 -- 172.21.15.3:0/3540322748 learned_addr learned my addr 172.21.15.3:0/3540322748 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:03.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff15e7fc700 1 --2- 172.21.15.3:0/3540322748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1600d41b0 0x7ff1600ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:03.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff15effd700 1 --2- 172.21.15.3:0/3540322748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1600cff90 0x7ff16014b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:03.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff15f7fe700 1 -- 172.21.15.3:0/3540322748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1600cff90 msgr2=0x7ff16014b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:03.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff15f7fe700 1 --2- 172.21.15.3:0/3540322748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1600cff90 0x7ff16014b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff15f7fe700 1 -- 172.21.15.3:0/3540322748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1600d41b0 msgr2=0x7ff1600ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:03.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff15f7fe700 1 --2- 172.21.15.3:0/3540322748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1600d41b0 0x7ff1600ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:03.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff15f7fe700 1 -- 172.21.15.3:0/3540322748 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff150009550 con 0x7ff16015dc50 2024-05-08T21:23:03.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.664+0000 7ff15f7fe700 1 --2- 172.21.15.3:0/3540322748 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff16015dc50 0x7ff1600b1230 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff15800efc0 tx=0x7ff15800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:03.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.665+0000 7ff1577fe700 1 -- 172.21.15.3:0/3540322748 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15800cc90 con 0x7ff16015dc50 2024-05-08T21:23:03.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.665+0000 7ff1577fe700 1 -- 172.21.15.3:0/3540322748 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff15800f040 con 0x7ff16015dc50 2024-05-08T21:23:03.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.665+0000 7ff16548a700 1 -- 172.21.15.3:0/3540322748 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1600cb7a0 con 0x7ff16015dc50 2024-05-08T21:23:03.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.665+0000 7ff1577fe700 1 -- 172.21.15.3:0/3540322748 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1580146c0 con 0x7ff16015dc50 2024-05-08T21:23:03.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.665+0000 7ff16548a700 1 -- 172.21.15.3:0/3540322748 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1600cc280 con 0x7ff16015dc50 2024-05-08T21:23:03.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.666+0000 7ff16548a700 1 -- 172.21.15.3:0/3540322748 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1600cbf60 con 0x7ff16015dc50 2024-05-08T21:23:03.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.666+0000 7ff1577fe700 1 -- 172.21.15.3:0/3540322748 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff158014820 con 0x7ff16015dc50 2024-05-08T21:23:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.666+0000 7ff1577fe700 1 --2- 172.21.15.3:0/3540322748 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff14006ac90 0x7ff14006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.667+0000 7ff1577fe700 1 -- 172.21.15.3:0/3540322748 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff158089f40 con 0x7ff16015dc50 2024-05-08T21:23:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.667+0000 7ff15effd700 1 --2- 172.21.15.3:0/3540322748 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff14006ac90 0x7ff14006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.667+0000 7ff15effd700 1 --2- 172.21.15.3:0/3540322748 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff14006ac90 0x7ff14006d150 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff148009b60 tx=0x7ff148009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:03.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.671+0000 7ff1577fe700 1 -- 172.21.15.3:0/3540322748 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff158055030 con 0x7ff16015dc50 2024-05-08T21:23:03.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:03.889+0000 7ff16548a700 1 -- 172.21.15.3:0/3540322748 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "recovery_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff160074bc0 con 0x7ff16015dc50 2024-05-08T21:23:04.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.631+0000 7ff1577fe700 1 -- 172.21.15.3:0/3540322748 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "recovery_fs_metadata", "pg_num_min": 64}]=0 pool 'recovery_fs_metadata' created v40) v1 ==== 148+0+0 (secure 0 0 0) 0x7ff158018070 con 0x7ff16015dc50 2024-05-08T21:23:04.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.637+0000 7ff16548a700 1 -- 172.21.15.3:0/3540322748 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff14006ac90 msgr2=0x7ff14006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:04.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.637+0000 7ff16548a700 1 --2- 172.21.15.3:0/3540322748 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff14006ac90 0x7ff14006d150 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff148009b60 tx=0x7ff148009210 comp rx=0 tx=0).stop 2024-05-08T21:23:04.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.637+0000 7ff16548a700 1 -- 172.21.15.3:0/3540322748 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff16015dc50 msgr2=0x7ff1600b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:04.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.637+0000 7ff16548a700 1 --2- 172.21.15.3:0/3540322748 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff16015dc50 0x7ff1600b1230 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff15800efc0 tx=0x7ff15800c3a0 comp rx=0 tx=0).stop 2024-05-08T21:23:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.638+0000 7ff16548a700 1 -- 172.21.15.3:0/3540322748 shutdown_connections 2024-05-08T21:23:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.638+0000 7ff16548a700 1 --2- 172.21.15.3:0/3540322748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1600cff90 0x7ff16014b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.638+0000 7ff16548a700 1 --2- 172.21.15.3:0/3540322748 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff14006ac90 0x7ff14006d150 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.638+0000 7ff16548a700 1 --2- 172.21.15.3:0/3540322748 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff16015dc50 0x7ff1600b1230 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.638+0000 7ff16548a700 1 --2- 172.21.15.3:0/3540322748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1600d41b0 0x7ff1600ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.638+0000 7ff16548a700 1 -- 172.21.15.3:0/3540322748 >> 172.21.15.3:0/3540322748 conn(0x7ff16008afe0 msgr2=0x7ff16015f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.638+0000 7ff16548a700 1 -- 172.21.15.3:0/3540322748 shutdown_connections 2024-05-08T21:23:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.638+0000 7ff16548a700 1 -- 172.21.15.3:0/3540322748 wait complete. 2024-05-08T21:23:04.641 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'recovery_fs_metadata' created 2024-05-08T21:23:04.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-08T21:23:04.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.802+0000 7fdc113e8700 1 Processor -- start 2024-05-08T21:23:04.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.802+0000 7fdc113e8700 1 -- start start 2024-05-08T21:23:04.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.803+0000 7fdc113e8700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdc0c0d0a60 0x7fdc0c0b5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:04.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.803+0000 7fdc113e8700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0c0af6e0 con 0x7fdc0c15fbb0 2024-05-08T21:23:04.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.803+0000 7fdc113e8700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0c0aeb50 con 0x7fdc0c145040 2024-05-08T21:23:04.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.803+0000 7fdc113e8700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0c0b6460 con 0x7fdc0c0d0a60 2024-05-08T21:23:04.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.803+0000 7fdc0b7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fdc0c15fbb0 0x7fdc0c0afb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:42838/0 (socket says 172.21.15.3:42838) 2024-05-08T21:23:04.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.803+0000 7fdc0b7fe700 1 -- 172.21.15.3:0/178408095 learned_addr learned my addr 172.21.15.3:0/178408095 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:04.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.803+0000 7fdc0a7fc700 1 --2- 172.21.15.3:0/178408095 >> v2:172.21.15.186:3300/0 conn(0x7fdc0c0d0a60 0x7fdc0c0b5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:04.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.804+0000 7fdc09ffb700 1 -- 172.21.15.3:0/178408095 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860264832 0 0) 0x7fdc0c0af6e0 con 0x7fdc0c15fbb0 2024-05-08T21:23:04.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.804+0000 7fdc09ffb700 1 -- 172.21.15.3:0/178408095 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf80035b0 con 0x7fdc0c15fbb0 2024-05-08T21:23:04.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.804+0000 7fdc0a7fc700 1 -- 172.21.15.3:0/178408095 >> v1:172.21.15.186:6789/0 conn(0x7fdc0c145040 legacy=0x7fdc0c0d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:04.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.804+0000 7fdc0a7fc700 1 -- 172.21.15.3:0/178408095 >> v1:172.21.15.3:6789/0 conn(0x7fdc0c15fbb0 legacy=0x7fdc0c0afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:04.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.804+0000 7fdc0a7fc700 1 -- 172.21.15.3:0/178408095 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c0aedd0 con 0x7fdc0c0d0a60 2024-05-08T21:23:04.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.804+0000 7fdc0a7fc700 1 --2- 172.21.15.3:0/178408095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc0c0d0a60 0x7fdc0c0b5f20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009840 tx=0x7fdbfc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f796845d975b0d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:04.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.804+0000 7fdc09ffb700 1 -- 172.21.15.3:0/178408095 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc011830 con 0x7fdc0c0d0a60 2024-05-08T21:23:04.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.804+0000 7fdc09ffb700 1 -- 172.21.15.3:0/178408095 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbfc011990 con 0x7fdc0c0d0a60 2024-05-08T21:23:04.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.804+0000 7fdc09ffb700 1 -- 172.21.15.3:0/178408095 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc011d20 con 0x7fdc0c0d0a60 2024-05-08T21:23:04.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.805+0000 7fdc113e8700 1 -- 172.21.15.3:0/178408095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc0c0d0a60 msgr2=0x7fdc0c0b5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:04.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.805+0000 7fdc113e8700 1 --2- 172.21.15.3:0/178408095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc0c0d0a60 0x7fdc0c0b5f20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009840 tx=0x7fdbfc009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:04.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.805+0000 7fdc113e8700 1 -- 172.21.15.3:0/178408095 shutdown_connections 2024-05-08T21:23:04.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.805+0000 7fdc113e8700 1 --2- 172.21.15.3:0/178408095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc0c0d0a60 0x7fdc0c0b5f20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:04.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.805+0000 7fdc113e8700 1 -- 172.21.15.3:0/178408095 >> 172.21.15.3:0/178408095 conn(0x7fdc0c099a50 msgr2=0x7fdc0c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:04.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.805+0000 7fdc113e8700 1 -- 172.21.15.3:0/178408095 shutdown_connections 2024-05-08T21:23:04.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.805+0000 7fdc113e8700 1 -- 172.21.15.3:0/178408095 wait complete. 2024-05-08T21:23:04.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.807+0000 7fdc113e8700 1 Processor -- start 2024-05-08T21:23:04.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.807+0000 7fdc113e8700 1 -- start start 2024-05-08T21:23:04.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.807+0000 7fdc113e8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc0c0d0a60 0x7fdc0c140300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:04.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.807+0000 7fdc113e8700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc0c145040 0x7fdc0c13d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:04.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.807+0000 7fdc113e8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc0c15fbb0 0x7fdc0c13a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:04.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.807+0000 7fdc113e8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c139610 con 0x7fdc0c0d0a60 2024-05-08T21:23:04.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.807+0000 7fdc113e8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c13f540 con 0x7fdc0c15fbb0 2024-05-08T21:23:04.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.807+0000 7fdc113e8700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c13f820 con 0x7fdc0c145040 2024-05-08T21:23:04.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.807+0000 7fdc0affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc0c0d0a60 0x7fdc0c140300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:04.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.808+0000 7fdc0affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc0c0d0a60 0x7fdc0c140300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54108/0 (socket says 172.21.15.3:54108) 2024-05-08T21:23:04.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.808+0000 7fdc0affd700 1 -- 172.21.15.3:0/2384163641 learned_addr learned my addr 172.21.15.3:0/2384163641 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:04.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.808+0000 7fdc0b7fe700 1 --2- 172.21.15.3:0/2384163641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc0c15fbb0 0x7fdc0c13a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:04.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.808+0000 7fdc0a7fc700 1 --2- 172.21.15.3:0/2384163641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc0c145040 0x7fdc0c13d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:04.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.808+0000 7fdc0affd700 1 -- 172.21.15.3:0/2384163641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc0c145040 msgr2=0x7fdc0c13d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:04.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.808+0000 7fdc0affd700 1 --2- 172.21.15.3:0/2384163641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc0c145040 0x7fdc0c13d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:04.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.808+0000 7fdc0affd700 1 -- 172.21.15.3:0/2384163641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc0c15fbb0 msgr2=0x7fdc0c13a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:04.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.808+0000 7fdc0affd700 1 --2- 172.21.15.3:0/2384163641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc0c15fbb0 0x7fdc0c13a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:04.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.808+0000 7fdc0affd700 1 -- 172.21.15.3:0/2384163641 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc0094a0 con 0x7fdc0c0d0a60 2024-05-08T21:23:04.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.808+0000 7fdc0affd700 1 --2- 172.21.15.3:0/2384163641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc0c0d0a60 0x7fdc0c140300 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400c4e0 tx=0x7fdbf400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:04.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.808+0000 7fdc037fe700 1 -- 172.21.15.3:0/2384163641 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40052d0 con 0x7fdc0c0d0a60 2024-05-08T21:23:04.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.808+0000 7fdc037fe700 1 -- 172.21.15.3:0/2384163641 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf4005430 con 0x7fdc0c0d0a60 2024-05-08T21:23:04.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.808+0000 7fdc037fe700 1 -- 172.21.15.3:0/2384163641 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40038c0 con 0x7fdc0c0d0a60 2024-05-08T21:23:04.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.808+0000 7fdc113e8700 1 -- 172.21.15.3:0/2384163641 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0c13fb00 con 0x7fdc0c0d0a60 2024-05-08T21:23:04.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.808+0000 7fdc113e8700 1 -- 172.21.15.3:0/2384163641 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c13f1f0 con 0x7fdc0c0d0a60 2024-05-08T21:23:04.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.809+0000 7fdc113e8700 1 -- 172.21.15.3:0/2384163641 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0c140020 con 0x7fdc0c0d0a60 2024-05-08T21:23:04.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.810+0000 7fdc037fe700 1 -- 172.21.15.3:0/2384163641 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdbf4003a20 con 0x7fdc0c0d0a60 2024-05-08T21:23:04.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.810+0000 7fdc037fe700 1 --2- 172.21.15.3:0/2384163641 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdbec06ac90 0x7fdbec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:04.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.811+0000 7fdc037fe700 1 -- 172.21.15.3:0/2384163641 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6170+0+0 (secure 0 0 0) 0x7fdbf4088930 con 0x7fdc0c0d0a60 2024-05-08T21:23:04.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.811+0000 7fdc0a7fc700 1 --2- 172.21.15.3:0/2384163641 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdbec06ac90 0x7fdbec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:04.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.811+0000 7fdc0a7fc700 1 --2- 172.21.15.3:0/2384163641 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdbec06ac90 0x7fdbec06d150 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0090b0 tx=0x7fdbfc008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:04.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:04.815+0000 7fdc037fe700 1 -- 172.21.15.3:0/2384163641 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fdc0c140020 con 0x7fdc0c0d0a60 2024-05-08T21:23:05.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.033+0000 7fdc113e8700 1 -- 172.21.15.3:0/2384163641 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fdc0c140020 con 0x7fdc0c0d0a60 2024-05-08T21:23:05.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.034+0000 7fdc037fe700 1 -- 172.21.15.3:0/2384163641 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' already exists v40) v1 ==== 177+0+0 (secure 0 0 0) 0x7fdc0c140020 con 0x7fdc0c0d0a60 2024-05-08T21:23:05.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.036+0000 7fdc113e8700 1 -- 172.21.15.3:0/2384163641 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdbec06ac90 msgr2=0x7fdbec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:05.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.036+0000 7fdc113e8700 1 --2- 172.21.15.3:0/2384163641 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdbec06ac90 0x7fdbec06d150 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0090b0 tx=0x7fdbfc008030 comp rx=0 tx=0).stop 2024-05-08T21:23:05.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.036+0000 7fdc113e8700 1 -- 172.21.15.3:0/2384163641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc0c0d0a60 msgr2=0x7fdc0c140300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:05.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.036+0000 7fdc113e8700 1 --2- 172.21.15.3:0/2384163641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc0c0d0a60 0x7fdc0c140300 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400c4e0 tx=0x7fdbf400c7f0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.036+0000 7fdc113e8700 1 -- 172.21.15.3:0/2384163641 shutdown_connections 2024-05-08T21:23:05.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.036+0000 7fdc113e8700 1 --2- 172.21.15.3:0/2384163641 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdbec06ac90 0x7fdbec06d150 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.036+0000 7fdc113e8700 1 --2- 172.21.15.3:0/2384163641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc0c15fbb0 0x7fdc0c13a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.036+0000 7fdc113e8700 1 --2- 172.21.15.3:0/2384163641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc0c0d0a60 0x7fdc0c140300 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.036+0000 7fdc113e8700 1 --2- 172.21.15.3:0/2384163641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc0c145040 0x7fdc0c13d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.036+0000 7fdc113e8700 1 -- 172.21.15.3:0/2384163641 >> 172.21.15.3:0/2384163641 conn(0x7fdc0c099a50 msgr2=0x7fdc0c06dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:05.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.036+0000 7fdc113e8700 1 -- 172.21.15.3:0/2384163641 shutdown_connections 2024-05-08T21:23:05.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.036+0000 7fdc113e8700 1 -- 172.21.15.3:0/2384163641 wait complete. 2024-05-08T21:23:05.039 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' already exists 2024-05-08T21:23:05.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new recovery_fs recovery_fs_metadata cephfs_data --recover --allow-dangerous-metadata-overlay 2024-05-08T21:23:05.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.192+0000 7fcbb7561700 1 Processor -- start 2024-05-08T21:23:05.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.193+0000 7fcbb7561700 1 -- start start 2024-05-08T21:23:05.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.193+0000 7fcbb7561700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb006bd60 0x7fcbb00cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:05.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.193+0000 7fcbb7561700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb00b1200 con 0x7fcbb00cdc90 2024-05-08T21:23:05.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.194+0000 7fcbb7561700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb00cfcc0 con 0x7fcbb00d13a0 2024-05-08T21:23:05.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.194+0000 7fcbb7561700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb00ce5c0 con 0x7fcbb006bd60 2024-05-08T21:23:05.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.194+0000 7fcbb5afe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fcbb00cdc90 0x7fcbb00b19c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:42844/0 (socket says 172.21.15.3:42844) 2024-05-08T21:23:05.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.194+0000 7fcbb5afe700 1 -- 172.21.15.3:0/1896636031 learned_addr learned my addr 172.21.15.3:0/1896636031 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:05.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.194+0000 7fcbb4afc700 1 --2- 172.21.15.3:0/1896636031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb006bd60 0x7fcbb00cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:05.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.194+0000 7fcba7fff700 1 -- 172.21.15.3:0/1896636031 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350020992 0 0) 0x7fcbb00b1200 con 0x7fcbb00cdc90 2024-05-08T21:23:05.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.194+0000 7fcba7fff700 1 -- 172.21.15.3:0/1896636031 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb9c0035b0 con 0x7fcbb00cdc90 2024-05-08T21:23:05.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.194+0000 7fcba7fff700 1 -- 172.21.15.3:0/1896636031 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388775655 0 0) 0x7fcbb00cfcc0 con 0x7fcbb00d13a0 2024-05-08T21:23:05.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.194+0000 7fcba7fff700 1 -- 172.21.15.3:0/1896636031 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb00b1200 con 0x7fcbb00d13a0 2024-05-08T21:23:05.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.194+0000 7fcba7fff700 1 -- 172.21.15.3:0/1896636031 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2402872020 0 0) 0x7fcb9c0035b0 con 0x7fcbb00cdc90 2024-05-08T21:23:05.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.194+0000 7fcbb4afc700 1 -- 172.21.15.3:0/1896636031 >> v1:172.21.15.186:6789/0 conn(0x7fcbb00d13a0 legacy=0x7fcbb00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:05.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.194+0000 7fcbb4afc700 1 -- 172.21.15.3:0/1896636031 >> v1:172.21.15.3:6789/0 conn(0x7fcbb00cdc90 legacy=0x7fcbb00b19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:05.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.194+0000 7fcbb4afc700 1 -- 172.21.15.3:0/1896636031 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb00ce8a0 con 0x7fcbb006bd60 2024-05-08T21:23:05.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.195+0000 7fcbb4afc700 1 --2- 172.21.15.3:0/1896636031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb006bd60 0x7fcbb00cf780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009870 tx=0x7fcba0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b7a2191707d9395 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:05.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.195+0000 7fcba7fff700 1 -- 172.21.15.3:0/1896636031 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba80019f0 con 0x7fcbb006bd60 2024-05-08T21:23:05.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.195+0000 7fcba7fff700 1 -- 172.21.15.3:0/1896636031 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba0004030 con 0x7fcbb006bd60 2024-05-08T21:23:05.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.195+0000 7fcba7fff700 1 -- 172.21.15.3:0/1896636031 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba00043c0 con 0x7fcbb006bd60 2024-05-08T21:23:05.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.195+0000 7fcbb7561700 1 -- 172.21.15.3:0/1896636031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb006bd60 msgr2=0x7fcbb00cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:05.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.195+0000 7fcbb7561700 1 --2- 172.21.15.3:0/1896636031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb006bd60 0x7fcbb00cf780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009870 tx=0x7fcba0009b80 comp rx=0 tx=0).stop 2024-05-08T21:23:05.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.195+0000 7fcbb7561700 1 -- 172.21.15.3:0/1896636031 shutdown_connections 2024-05-08T21:23:05.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.195+0000 7fcbb7561700 1 --2- 172.21.15.3:0/1896636031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb006bd60 0x7fcbb00cf780 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.195+0000 7fcbb7561700 1 -- 172.21.15.3:0/1896636031 >> 172.21.15.3:0/1896636031 conn(0x7fcbb008afe0 msgr2=0x7fcbb008b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:05.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.196+0000 7fcbb7561700 1 -- 172.21.15.3:0/1896636031 shutdown_connections 2024-05-08T21:23:05.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.196+0000 7fcbb7561700 1 -- 172.21.15.3:0/1896636031 wait complete. 2024-05-08T21:23:05.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.197+0000 7fcbb7561700 1 Processor -- start 2024-05-08T21:23:05.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.197+0000 7fcbb7561700 1 -- start start 2024-05-08T21:23:05.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.198+0000 7fcbb7561700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbb006bd60 0x7fcbb0149a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:05.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.198+0000 7fcbb7561700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb00cdc90 0x7fcbb014efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:05.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.198+0000 7fcbb7561700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb00d13a0 0x7fcbb014f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:05.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.198+0000 7fcbb7561700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb014fe10 con 0x7fcbb006bd60 2024-05-08T21:23:05.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.198+0000 7fcbb7561700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb00aead0 con 0x7fcbb00cdc90 2024-05-08T21:23:05.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.198+0000 7fcbb7561700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb00c8660 con 0x7fcbb00d13a0 2024-05-08T21:23:05.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.198+0000 7fcbb52fd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbb006bd60 0x7fcbb0149a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:05.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.198+0000 7fcbb52fd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbb006bd60 0x7fcbb0149a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54112/0 (socket says 172.21.15.3:54112) 2024-05-08T21:23:05.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.198+0000 7fcbb52fd700 1 -- 172.21.15.3:0/3480518620 learned_addr learned my addr 172.21.15.3:0/3480518620 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:05.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.198+0000 7fcbb4afc700 1 --2- 172.21.15.3:0/3480518620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb00cdc90 0x7fcbb014efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:05.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.198+0000 7fcbb5afe700 1 --2- 172.21.15.3:0/3480518620 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb00d13a0 0x7fcbb014f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:05.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.198+0000 7fcbb52fd700 1 -- 172.21.15.3:0/3480518620 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb00d13a0 msgr2=0x7fcbb014f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:05.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.198+0000 7fcbb52fd700 1 --2- 172.21.15.3:0/3480518620 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb00d13a0 0x7fcbb014f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.198+0000 7fcbb52fd700 1 -- 172.21.15.3:0/3480518620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb00cdc90 msgr2=0x7fcbb014efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:05.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.198+0000 7fcbb52fd700 1 --2- 172.21.15.3:0/3480518620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb00cdc90 0x7fcbb014efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.198+0000 7fcbb52fd700 1 -- 172.21.15.3:0/3480518620 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba00094a0 con 0x7fcbb006bd60 2024-05-08T21:23:05.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.199+0000 7fcbb52fd700 1 --2- 172.21.15.3:0/3480518620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbb006bd60 0x7fcbb0149a90 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcbac00c520 tx=0x7fcbac00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:05.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.199+0000 7fcba5ffb700 1 -- 172.21.15.3:0/3480518620 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbac0052d0 con 0x7fcbb006bd60 2024-05-08T21:23:05.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.199+0000 7fcba5ffb700 1 -- 172.21.15.3:0/3480518620 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbac005430 con 0x7fcbb006bd60 2024-05-08T21:23:05.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.199+0000 7fcbb7561700 1 -- 172.21.15.3:0/3480518620 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb00c8940 con 0x7fcbb006bd60 2024-05-08T21:23:05.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.199+0000 7fcba5ffb700 1 -- 172.21.15.3:0/3480518620 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbac0038c0 con 0x7fcbb006bd60 2024-05-08T21:23:05.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.199+0000 7fcbb7561700 1 -- 172.21.15.3:0/3480518620 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb0149fd0 con 0x7fcbb006bd60 2024-05-08T21:23:05.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.200+0000 7fcbb7561700 1 -- 172.21.15.3:0/3480518620 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbb014a1c0 con 0x7fcbb006bd60 2024-05-08T21:23:05.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.200+0000 7fcba5ffb700 1 -- 172.21.15.3:0/3480518620 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcbac003a20 con 0x7fcbb006bd60 2024-05-08T21:23:05.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.201+0000 7fcba5ffb700 1 --2- 172.21.15.3:0/3480518620 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb9406ac90 0x7fcb9406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:05.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.201+0000 7fcba5ffb700 1 -- 172.21.15.3:0/3480518620 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6170+0+0 (secure 0 0 0) 0x7fcbac088980 con 0x7fcbb006bd60 2024-05-08T21:23:05.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.201+0000 7fcbb4afc700 1 --2- 172.21.15.3:0/3480518620 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb9406ac90 0x7fcb9406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:05.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.202+0000 7fcbb4afc700 1 --2- 172.21.15.3:0/3480518620 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb9406ac90 0x7fcb9406d150 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcba00097c0 tx=0x7fcba0008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:05.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.206+0000 7fcba5ffb700 1 -- 172.21.15.3:0/3480518620 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fcbac053980 con 0x7fcbb006bd60 2024-05-08T21:23:05.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.473+0000 7fcbb7561700 1 -- 172.21.15.3:0/3480518620 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "recovery_fs", "metadata": "recovery_fs_metadata", "data": "cephfs_data", "recover": true, "allow_dangerous_metadata_overlay": true} v 0) v1 -- 0x7fcbb00cfc70 con 0x7fcbb006bd60 2024-05-08T21:23:05.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.479+0000 7fcba5ffb700 1 -- 172.21.15.3:0/3480518620 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "recovery_fs", "metadata": "recovery_fs_metadata", "data": "cephfs_data", "recover": true, "allow_dangerous_metadata_overlay": true}]=0 new fs with metadata pool 6 and data pool 5 v19) v1 ==== 241+0+0 (secure 0 0 0) 0x7fcbac017100 con 0x7fcbb006bd60 2024-05-08T21:23:05.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.482+0000 7fcbb7561700 1 -- 172.21.15.3:0/3480518620 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb9406ac90 msgr2=0x7fcb9406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:05.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.482+0000 7fcbb7561700 1 --2- 172.21.15.3:0/3480518620 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb9406ac90 0x7fcb9406d150 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcba00097c0 tx=0x7fcba0008030 comp rx=0 tx=0).stop 2024-05-08T21:23:05.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.483+0000 7fcbb7561700 1 -- 172.21.15.3:0/3480518620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbb006bd60 msgr2=0x7fcbb0149a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:05.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.483+0000 7fcbb7561700 1 --2- 172.21.15.3:0/3480518620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbb006bd60 0x7fcbb0149a90 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcbac00c520 tx=0x7fcbac00c830 comp rx=0 tx=0).stop 2024-05-08T21:23:05.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.483+0000 7fcbb7561700 1 -- 172.21.15.3:0/3480518620 shutdown_connections 2024-05-08T21:23:05.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.483+0000 7fcbb7561700 1 --2- 172.21.15.3:0/3480518620 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb00d13a0 0x7fcbb014f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.483+0000 7fcbb7561700 1 --2- 172.21.15.3:0/3480518620 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb9406ac90 0x7fcb9406d150 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.483+0000 7fcbb7561700 1 --2- 172.21.15.3:0/3480518620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbb006bd60 0x7fcbb0149a90 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.483+0000 7fcbb7561700 1 --2- 172.21.15.3:0/3480518620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb00cdc90 0x7fcbb014efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.483+0000 7fcbb7561700 1 -- 172.21.15.3:0/3480518620 >> 172.21.15.3:0/3480518620 conn(0x7fcbb008afe0 msgr2=0x7fcbb00b18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:05.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.483+0000 7fcbb7561700 1 -- 172.21.15.3:0/3480518620 shutdown_connections 2024-05-08T21:23:05.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.483+0000 7fcbb7561700 1 -- 172.21.15.3:0/3480518620 wait complete. 2024-05-08T21:23:05.488 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 6 and data pool 5 2024-05-08T21:23:05.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:23:05.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.641+0000 7f22826cb700 1 Processor -- start 2024-05-08T21:23:05.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.641+0000 7f22826cb700 1 -- start start 2024-05-08T21:23:05.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.642+0000 7f22826cb700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f227c06b140 0x7f227c0ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:05.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.642+0000 7f22826cb700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f227c0cf760 0x7f227c0cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:05.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.642+0000 7f22826cb700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f227c0cdbb0 con 0x7f227c06a960 2024-05-08T21:23:05.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.642+0000 7f22826cb700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f227c0cd310 con 0x7f227c06b140 2024-05-08T21:23:05.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.642+0000 7f22826cb700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f227c0cd5f0 con 0x7f227c0cf760 2024-05-08T21:23:05.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.642+0000 7f227bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f227c06b140 0x7f227c0ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:05.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.642+0000 7f227b7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f227c06a960 0x7f227c15bc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:42850/0 (socket says 172.21.15.3:42850) 2024-05-08T21:23:05.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.642+0000 7f227bfff700 1 --2- 172.21.15.3:0/2070839897 >> v2:172.21.15.3:3300/0 conn(0x7f227c06b140 0x7f227c0ccd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54128/0 (socket says 172.21.15.3:54128) 2024-05-08T21:23:05.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.642+0000 7f227b7fe700 1 -- 172.21.15.3:0/2070839897 learned_addr learned my addr 172.21.15.3:0/2070839897 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:05.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.642+0000 7f2280c68700 1 --2- 172.21.15.3:0/2070839897 >> v2:172.21.15.186:3300/0 conn(0x7f227c0cf760 0x7f227c0cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:05.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.642+0000 7f227bfff700 1 -- 172.21.15.3:0/2070839897 >> v2:172.21.15.186:3300/0 conn(0x7f227c0cf760 msgr2=0x7f227c0cfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:05.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.642+0000 7f227bfff700 1 --2- 172.21.15.3:0/2070839897 >> v2:172.21.15.186:3300/0 conn(0x7f227c0cf760 0x7f227c0cfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.642+0000 7f227bfff700 1 -- 172.21.15.3:0/2070839897 >> v1:172.21.15.3:6789/0 conn(0x7f227c06a960 legacy=0x7f227c15bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:05.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.642+0000 7f227bfff700 1 -- 172.21.15.3:0/2070839897 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227c0cd8d0 con 0x7f227c06b140 2024-05-08T21:23:05.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.643+0000 7f2280c68700 1 --2- 172.21.15.3:0/2070839897 >> v2:172.21.15.186:3300/0 conn(0x7f227c0cf760 0x7f227c0cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:05.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.642+0000 7f227affd700 1 -- 172.21.15.3:0/2070839897 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37585002 0 0) 0x7f227c0cdbb0 con 0x7f227c06a960 2024-05-08T21:23:05.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.643+0000 7f227bfff700 1 --2- 172.21.15.3:0/2070839897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f227c06b140 0x7f227c0ccd30 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f227c0cab50 tx=0x7f226c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5456596e90f705f server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:05.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.643+0000 7f227affd700 1 -- 172.21.15.3:0/2070839897 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226c011830 con 0x7f227c06b140 2024-05-08T21:23:05.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.643+0000 7f227affd700 1 -- 172.21.15.3:0/2070839897 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f226c011990 con 0x7f227c06b140 2024-05-08T21:23:05.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.643+0000 7f227affd700 1 -- 172.21.15.3:0/2070839897 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226c011d20 con 0x7f227c06b140 2024-05-08T21:23:05.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.643+0000 7f22826cb700 1 -- 172.21.15.3:0/2070839897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f227c06b140 msgr2=0x7f227c0ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:05.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.643+0000 7f22826cb700 1 --2- 172.21.15.3:0/2070839897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f227c06b140 0x7f227c0ccd30 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f227c0cab50 tx=0x7f226c009b20 comp rx=0 tx=0).stop 2024-05-08T21:23:05.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.644+0000 7f22826cb700 1 -- 172.21.15.3:0/2070839897 shutdown_connections 2024-05-08T21:23:05.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.644+0000 7f22826cb700 1 --2- 172.21.15.3:0/2070839897 >> v2:172.21.15.186:3300/0 conn(0x7f227c0cf760 0x7f227c0cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.644+0000 7f22826cb700 1 --2- 172.21.15.3:0/2070839897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f227c06b140 0x7f227c0ccd30 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.644+0000 7f22826cb700 1 -- 172.21.15.3:0/2070839897 >> 172.21.15.3:0/2070839897 conn(0x7f227c0d0320 msgr2=0x7f227c0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:05.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.644+0000 7f22826cb700 1 -- 172.21.15.3:0/2070839897 shutdown_connections 2024-05-08T21:23:05.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.644+0000 7f22826cb700 1 -- 172.21.15.3:0/2070839897 wait complete. 2024-05-08T21:23:05.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.645+0000 7f22826cb700 1 Processor -- start 2024-05-08T21:23:05.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.646+0000 7f22826cb700 1 -- start start 2024-05-08T21:23:05.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.646+0000 7f22826cb700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c06a960 0x7f227c0c6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:05.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f22826cb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f227c06b140 0x7f227c0c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:05.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f22826cb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f227c0cf760 0x7f227c0aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:05.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f22826cb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227c0c4390 con 0x7f227c0cf760 2024-05-08T21:23:05.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f22826cb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227c0c45c0 con 0x7f227c06b140 2024-05-08T21:23:05.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f22826cb700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f227c0c48a0 con 0x7f227c06a960 2024-05-08T21:23:05.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f2280c68700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f227c0cf760 0x7f227c0aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:05.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f2280c68700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f227c0cf760 0x7f227c0aeae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54136/0 (socket says 172.21.15.3:54136) 2024-05-08T21:23:05.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f2280c68700 1 -- 172.21.15.3:0/114638035 learned_addr learned my addr 172.21.15.3:0/114638035 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:05.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f227b7fe700 1 --2- 172.21.15.3:0/114638035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f227c06b140 0x7f227c0c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:05.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f227bfff700 1 --2- 172.21.15.3:0/114638035 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c06a960 0x7f227c0c6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:05.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f2280c68700 1 -- 172.21.15.3:0/114638035 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c06a960 msgr2=0x7f227c0c6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:05.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f2280c68700 1 --2- 172.21.15.3:0/114638035 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c06a960 0x7f227c0c6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f2280c68700 1 -- 172.21.15.3:0/114638035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f227c06b140 msgr2=0x7f227c0c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:05.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f2280c68700 1 --2- 172.21.15.3:0/114638035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f227c06b140 0x7f227c0c3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f2280c68700 1 -- 172.21.15.3:0/114638035 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c0094a0 con 0x7f227c0cf760 2024-05-08T21:23:05.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f2280c68700 1 --2- 172.21.15.3:0/114638035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f227c0cf760 0x7f227c0aeae0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2274007ef0 tx=0x7f227400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:05.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f2278ff9700 1 -- 172.21.15.3:0/114638035 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227400dd80 con 0x7f227c0cf760 2024-05-08T21:23:05.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f2278ff9700 1 -- 172.21.15.3:0/114638035 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227400f040 con 0x7f227c0cf760 2024-05-08T21:23:05.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f22826cb700 1 -- 172.21.15.3:0/114638035 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f227c0c3cc0 con 0x7f227c0cf760 2024-05-08T21:23:05.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.647+0000 7f2278ff9700 1 -- 172.21.15.3:0/114638035 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2274003c60 con 0x7f227c0cf760 2024-05-08T21:23:05.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.648+0000 7f22826cb700 1 -- 172.21.15.3:0/114638035 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f227c0c4e60 con 0x7f227c0cf760 2024-05-08T21:23:05.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.648+0000 7f22826cb700 1 -- 172.21.15.3:0/114638035 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f227c0c51d0 con 0x7f227c0cf760 2024-05-08T21:23:05.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.649+0000 7f2278ff9700 1 -- 172.21.15.3:0/114638035 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2274004030 con 0x7f227c0cf760 2024-05-08T21:23:05.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.649+0000 7f2278ff9700 1 --2- 172.21.15.3:0/114638035 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f225c06ac90 0x7f225c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:05.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.650+0000 7f2278ff9700 1 -- 172.21.15.3:0/114638035 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f22740895c0 con 0x7f227c0cf760 2024-05-08T21:23:05.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.650+0000 7f227bfff700 1 --2- 172.21.15.3:0/114638035 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f225c06ac90 0x7f225c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:05.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.650+0000 7f227bfff700 1 --2- 172.21.15.3:0/114638035 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f225c06ac90 0x7f225c06d150 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f226c0090b0 tx=0x7f226c008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:05.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.654+0000 7f2278ff9700 1 -- 172.21.15.3:0/114638035 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f227c0c51d0 con 0x7f227c0cf760 2024-05-08T21:23:05.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.873+0000 7f22826cb700 1 -- 172.21.15.3:0/114638035 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f227c0c51d0 con 0x7f227c0cf760 2024-05-08T21:23:05.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.875+0000 7f2278ff9700 1 -- 172.21.15.3:0/114638035 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+18042 (secure 0 0 0) 0x7f227c0c51d0 con 0x7f227c0cf760 2024-05-08T21:23:05.876 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:05.877 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":41,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:23:05.475124+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:22:47.853786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-08T21:22:49.153759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"recovery_fs_metadata","create_time":"2024-05-08T21:23:03.921695+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6834/3599481370":"2024-05-09T21:23:01.563392+0000","172.21.15.3:6832/3599481370":"2024-05-09T21:23:01.563392+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:23:05.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.877+0000 7f22826cb700 1 -- 172.21.15.3:0/114638035 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f225c06ac90 msgr2=0x7f225c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:05.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.877+0000 7f22826cb700 1 --2- 172.21.15.3:0/114638035 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f225c06ac90 0x7f225c06d150 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f226c0090b0 tx=0x7f226c008030 comp rx=0 tx=0).stop 2024-05-08T21:23:05.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.877+0000 7f22826cb700 1 -- 172.21.15.3:0/114638035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f227c0cf760 msgr2=0x7f227c0aeae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:05.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.877+0000 7f22826cb700 1 --2- 172.21.15.3:0/114638035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f227c0cf760 0x7f227c0aeae0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2274007ef0 tx=0x7f227400d3b0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.877+0000 7f22826cb700 1 -- 172.21.15.3:0/114638035 shutdown_connections 2024-05-08T21:23:05.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.877+0000 7f22826cb700 1 --2- 172.21.15.3:0/114638035 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c06a960 0x7f227c0c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.877+0000 7f22826cb700 1 --2- 172.21.15.3:0/114638035 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f225c06ac90 0x7f225c06d150 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.877+0000 7f22826cb700 1 --2- 172.21.15.3:0/114638035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f227c0cf760 0x7f227c0aeae0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.877+0000 7f22826cb700 1 --2- 172.21.15.3:0/114638035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f227c06b140 0x7f227c0c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:05.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.877+0000 7f22826cb700 1 -- 172.21.15.3:0/114638035 >> 172.21.15.3:0/114638035 conn(0x7f227c0d0320 msgr2=0x7f227c0d0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:05.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.877+0000 7f22826cb700 1 -- 172.21.15.3:0/114638035 shutdown_connections 2024-05-08T21:23:05.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:05.877+0000 7f22826cb700 1 -- 172.21.15.3:0/114638035 wait complete. 2024-05-08T21:23:05.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:23:06.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.028+0000 7f361ed73700 1 Processor -- start 2024-05-08T21:23:06.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.028+0000 7f361ed73700 1 -- start start 2024-05-08T21:23:06.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.029+0000 7f361ed73700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f36180ccbd0 0x7f3618168580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:06.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.029+0000 7f361ed73700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361809a870 con 0x7f3618161030 2024-05-08T21:23:06.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.029+0000 7f361ed73700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36180d0840 con 0x7f36180b2480 2024-05-08T21:23:06.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.029+0000 7f361ed73700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361806ccc0 con 0x7f36180ccbd0 2024-05-08T21:23:06.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.029+0000 7f361cb0f700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3618161030 0x7f3618099a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:42856/0 (socket says 172.21.15.3:42856) 2024-05-08T21:23:06.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.029+0000 7f361cb0f700 1 -- 172.21.15.3:0/2013888631 learned_addr learned my addr 172.21.15.3:0/2013888631 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:06.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.029+0000 7f361d310700 1 --2- 172.21.15.3:0/2013888631 >> v2:172.21.15.186:3300/0 conn(0x7f36180ccbd0 0x7f3618168580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:06.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.029+0000 7f36177fe700 1 -- 172.21.15.3:0/2013888631 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709938378 0 0) 0x7f361809a870 con 0x7f3618161030 2024-05-08T21:23:06.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.029+0000 7f36177fe700 1 -- 172.21.15.3:0/2013888631 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36080035b0 con 0x7f3618161030 2024-05-08T21:23:06.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.029+0000 7f36177fe700 1 -- 172.21.15.3:0/2013888631 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3428147815 0 0) 0x7f36080035b0 con 0x7f3618161030 2024-05-08T21:23:06.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.029+0000 7f36177fe700 1 -- 172.21.15.3:0/2013888631 >> v2:172.21.15.186:3300/0 conn(0x7f36180ccbd0 msgr2=0x7f3618168580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:06.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.029+0000 7f36177fe700 1 --2- 172.21.15.3:0/2013888631 >> v2:172.21.15.186:3300/0 conn(0x7f36180ccbd0 0x7f3618168580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:06.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.029+0000 7f36177fe700 1 -- 172.21.15.3:0/2013888631 >> v1:172.21.15.186:6789/0 conn(0x7f36180b2480 legacy=0x7f361809a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:06.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.029+0000 7f36177fe700 1 -- 172.21.15.3:0/2013888631 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361806c600 con 0x7f3618161030 2024-05-08T21:23:06.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.029+0000 7f36177fe700 1 -- 172.21.15.3:0/2013888631 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f360c001670 con 0x7f3618161030 2024-05-08T21:23:06.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f36177fe700 1 -- 172.21.15.3:0/2013888631 >> v1:172.21.15.3:6789/0 conn(0x7f3618161030 legacy=0x7f3618099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:06.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f36177fe700 1 --2- 172.21.15.3:0/2013888631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3608003a00 0x7f3608003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:06.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f36177fe700 1 --2- 172.21.15.3:0/2013888631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3608004850 0x7f3608004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:06.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f36177fe700 1 --2- 172.21.15.3:0/2013888631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3608005210 0x7f360800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:06.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f36177fe700 1 -- 172.21.15.3:0/2013888631 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361806ccc0 con 0x7f3608004850 2024-05-08T21:23:06.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f36177fe700 1 -- 172.21.15.3:0/2013888631 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360800c0b0 con 0x7f3608005210 2024-05-08T21:23:06.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f36177fe700 1 -- 172.21.15.3:0/2013888631 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f360800c270 con 0x7f3608003a00 2024-05-08T21:23:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f3617fff700 1 --2- 172.21.15.3:0/2013888631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3608004850 0x7f3608004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f361cb0f700 1 --2- 172.21.15.3:0/2013888631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3608003a00 0x7f3608003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f361d310700 1 --2- 172.21.15.3:0/2013888631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3608005210 0x7f360800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f3617fff700 1 -- 172.21.15.3:0/2013888631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3608003a00 msgr2=0x7f3608003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f3617fff700 1 --2- 172.21.15.3:0/2013888631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3608003a00 0x7f3608003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f3617fff700 1 -- 172.21.15.3:0/2013888631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3608005210 msgr2=0x7f360800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f3617fff700 1 --2- 172.21.15.3:0/2013888631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3608005210 0x7f360800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f3617fff700 1 -- 172.21.15.3:0/2013888631 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36080035b0 con 0x7f3608004850 2024-05-08T21:23:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f361d310700 1 -- 172.21.15.3:0/2013888631 reap_dead start 2024-05-08T21:23:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f3617fff700 1 --2- 172.21.15.3:0/2013888631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3608004850 0x7f3608004cd0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3604001800 tx=0x7f3608005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79083e26b431b0ab server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f36177fe700 1 -- 172.21.15.3:0/2013888631 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36040012f0 con 0x7f3608004850 2024-05-08T21:23:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f36177fe700 1 -- 172.21.15.3:0/2013888631 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3604005310 con 0x7f3608004850 2024-05-08T21:23:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.030+0000 7f36177fe700 1 -- 172.21.15.3:0/2013888631 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3604006860 con 0x7f3608004850 2024-05-08T21:23:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.031+0000 7f361ed73700 1 -- 172.21.15.3:0/2013888631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3608004850 msgr2=0x7f3608004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.031+0000 7f361ed73700 1 --2- 172.21.15.3:0/2013888631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3608004850 0x7f3608004cd0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3604001800 tx=0x7f3608005880 comp rx=0 tx=0).stop 2024-05-08T21:23:06.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.031+0000 7f361ed73700 1 -- 172.21.15.3:0/2013888631 shutdown_connections 2024-05-08T21:23:06.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.031+0000 7f361ed73700 1 --2- 172.21.15.3:0/2013888631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3608004850 0x7f3608004cd0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:06.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.031+0000 7f361ed73700 1 -- 172.21.15.3:0/2013888631 >> 172.21.15.3:0/2013888631 conn(0x7f36180b2f20 msgr2=0x7f36180b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:06.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.031+0000 7f361ed73700 1 -- 172.21.15.3:0/2013888631 shutdown_connections 2024-05-08T21:23:06.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.031+0000 7f361ed73700 1 -- 172.21.15.3:0/2013888631 wait complete. 2024-05-08T21:23:06.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.032+0000 7f361ed73700 1 Processor -- start 2024-05-08T21:23:06.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.032+0000 7f361ed73700 1 -- start start 2024-05-08T21:23:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.033+0000 7f361ed73700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3608004850 0x7f3618153dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.033+0000 7f361ed73700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36180cae70 0x7f36180c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.033+0000 7f361ed73700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36180c9540 0x7f361815ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.033+0000 7f361ed73700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36180d0660 con 0x7f36180cae70 2024-05-08T21:23:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.033+0000 7f361ed73700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36180caa50 con 0x7f36180c9540 2024-05-08T21:23:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.033+0000 7f361ed73700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3618154310 con 0x7f3608004850 2024-05-08T21:23:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.034+0000 7f3617fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36180cae70 0x7f36180c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.034+0000 7f3617fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36180cae70 0x7f36180c9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54162/0 (socket says 172.21.15.3:54162) 2024-05-08T21:23:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.034+0000 7f3617fff700 1 -- 172.21.15.3:0/817028920 learned_addr learned my addr 172.21.15.3:0/817028920 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.034+0000 7f361cb0f700 1 --2- 172.21.15.3:0/817028920 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3608004850 0x7f3618153dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.034+0000 7f361d310700 1 --2- 172.21.15.3:0/817028920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36180c9540 0x7f361815ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.034+0000 7f3617fff700 1 -- 172.21.15.3:0/817028920 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3608004850 msgr2=0x7f3618153dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.034+0000 7f3617fff700 1 --2- 172.21.15.3:0/817028920 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3608004850 0x7f3618153dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.034+0000 7f3617fff700 1 -- 172.21.15.3:0/817028920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36180c9540 msgr2=0x7f361815ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.034+0000 7f3617fff700 1 --2- 172.21.15.3:0/817028920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36180c9540 0x7f361815ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:06.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.034+0000 7f3617fff700 1 -- 172.21.15.3:0/817028920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36180c7b70 con 0x7f36180cae70 2024-05-08T21:23:06.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.034+0000 7f3617fff700 1 --2- 172.21.15.3:0/817028920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36180cae70 0x7f36180c9000 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3604014040 tx=0x7f3604001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:06.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.034+0000 7f36157fa700 1 -- 172.21.15.3:0/817028920 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3604001290 con 0x7f36180cae70 2024-05-08T21:23:06.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.034+0000 7f36157fa700 1 -- 172.21.15.3:0/817028920 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3604006e20 con 0x7f36180cae70 2024-05-08T21:23:06.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.034+0000 7f361ed73700 1 -- 172.21.15.3:0/817028920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36180c9dd0 con 0x7f36180cae70 2024-05-08T21:23:06.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.034+0000 7f36157fa700 1 -- 172.21.15.3:0/817028920 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3604007040 con 0x7f36180cae70 2024-05-08T21:23:06.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.034+0000 7f361ed73700 1 -- 172.21.15.3:0/817028920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36180ca390 con 0x7f36180cae70 2024-05-08T21:23:06.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.035+0000 7f361ed73700 1 -- 172.21.15.3:0/817028920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3618068000 con 0x7f36180cae70 2024-05-08T21:23:06.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.036+0000 7f36157fa700 1 -- 172.21.15.3:0/817028920 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f36040071a0 con 0x7f36180cae70 2024-05-08T21:23:06.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.036+0000 7f36157fa700 1 --2- 172.21.15.3:0/817028920 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f35fc06ac90 0x7f35fc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:06.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.036+0000 7f36157fa700 1 -- 172.21.15.3:0/817028920 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f360408a0a0 con 0x7f36180cae70 2024-05-08T21:23:06.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.037+0000 7f361cb0f700 1 --2- 172.21.15.3:0/817028920 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f35fc06ac90 0x7f35fc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:06.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.037+0000 7f361cb0f700 1 --2- 172.21.15.3:0/817028920 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f35fc06ac90 0x7f35fc06d150 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3618069190 tx=0x7f360c000bc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:06.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.041+0000 7f36157fa700 1 -- 172.21.15.3:0/817028920 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f3604055210 con 0x7f36180cae70 2024-05-08T21:23:06.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.263+0000 7f361ed73700 1 -- 172.21.15.3:0/817028920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36180d0cf0 con 0x7f36180cae70 2024-05-08T21:23:06.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.265+0000 7f36157fa700 1 -- 172.21.15.3:0/817028920 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+18042 (secure 0 0 0) 0x7f3604058830 con 0x7f36180cae70 2024-05-08T21:23:06.267 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:06.267 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":41,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:23:05.475124+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:22:47.853786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-08T21:22:49.153759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"recovery_fs_metadata","create_time":"2024-05-08T21:23:03.921695+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6834/3599481370":"2024-05-09T21:23:01.563392+0000","172.21.15.3:6832/3599481370":"2024-05-09T21:23:01.563392+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:23:06.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.267+0000 7f361ed73700 1 -- 172.21.15.3:0/817028920 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f35fc06ac90 msgr2=0x7f35fc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:06.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.267+0000 7f361ed73700 1 --2- 172.21.15.3:0/817028920 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f35fc06ac90 0x7f35fc06d150 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3618069190 tx=0x7f360c000bc0 comp rx=0 tx=0).stop 2024-05-08T21:23:06.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.267+0000 7f361ed73700 1 -- 172.21.15.3:0/817028920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36180cae70 msgr2=0x7f36180c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:06.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.267+0000 7f361ed73700 1 --2- 172.21.15.3:0/817028920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36180cae70 0x7f36180c9000 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3604014040 tx=0x7f3604001120 comp rx=0 tx=0).stop 2024-05-08T21:23:06.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.267+0000 7f361ed73700 1 -- 172.21.15.3:0/817028920 shutdown_connections 2024-05-08T21:23:06.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.267+0000 7f361ed73700 1 --2- 172.21.15.3:0/817028920 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3608004850 0x7f3618153dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:06.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.267+0000 7f361ed73700 1 --2- 172.21.15.3:0/817028920 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f35fc06ac90 0x7f35fc06d150 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:06.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.267+0000 7f361ed73700 1 --2- 172.21.15.3:0/817028920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36180c9540 0x7f361815ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:06.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.267+0000 7f361ed73700 1 --2- 172.21.15.3:0/817028920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36180cae70 0x7f36180c9000 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:06.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.267+0000 7f361ed73700 1 -- 172.21.15.3:0/817028920 >> 172.21.15.3:0/817028920 conn(0x7f36180b2f20 msgr2=0x7f361806f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:06.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.267+0000 7f361ed73700 1 -- 172.21.15.3:0/817028920 shutdown_connections 2024-05-08T21:23:06.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.267+0000 7f361ed73700 1 -- 172.21.15.3:0/817028920 wait complete. 2024-05-08T21:23:06.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set recovery_fs standby_count_wanted 0 2024-05-08T21:23:06.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.420+0000 7f150b837700 1 Processor -- start 2024-05-08T21:23:06.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.421+0000 7f150b837700 1 -- start start 2024-05-08T21:23:06.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.422+0000 7f150b837700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f15040c6c50 0x7f15040ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:06.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.422+0000 7f150b837700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15040b3dd0 con 0x7f15040d0d30 2024-05-08T21:23:06.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.422+0000 7f150b837700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15040d2010 con 0x7f15040d0240 2024-05-08T21:23:06.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.422+0000 7f150b837700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15040cf7f0 con 0x7f15040c6c50 2024-05-08T21:23:06.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.422+0000 7f1509dd4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f15040c6c50 0x7f15040ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:06.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.422+0000 7f1508dd2700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f15040d0d30 0x7f15040ce260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:42864/0 (socket says 172.21.15.3:42864) 2024-05-08T21:23:06.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.422+0000 7f1508dd2700 1 -- 172.21.15.3:0/3216481374 learned_addr learned my addr 172.21.15.3:0/3216481374 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:06.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.422+0000 7f1509dd4700 1 -- 172.21.15.3:0/3216481374 >> v1:172.21.15.186:6789/0 conn(0x7f15040d0240 legacy=0x7f15040d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:06.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.422+0000 7f1509dd4700 1 -- 172.21.15.3:0/3216481374 >> v1:172.21.15.3:6789/0 conn(0x7f15040d0d30 legacy=0x7f15040ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:06.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.422+0000 7f14fbfff700 1 -- 172.21.15.3:0/3216481374 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617825049 0 0) 0x7f15040b3dd0 con 0x7f15040d0d30 2024-05-08T21:23:06.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.422+0000 7f1509dd4700 1 -- 172.21.15.3:0/3216481374 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15040d22f0 con 0x7f15040c6c50 2024-05-08T21:23:06.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.422+0000 7f1509dd4700 1 --2- 172.21.15.3:0/3216481374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040c6c50 0x7f15040ce970 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f14fc009d90 tx=0x7f14fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57cc99c393d4ddab server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:06.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.422+0000 7f14fbfff700 1 -- 172.21.15.3:0/3216481374 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc00e070 con 0x7f15040c6c50 2024-05-08T21:23:06.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.423+0000 7f14fbfff700 1 -- 172.21.15.3:0/3216481374 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14fc00bb90 con 0x7f15040c6c50 2024-05-08T21:23:06.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.423+0000 7f14fbfff700 1 -- 172.21.15.3:0/3216481374 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc003a20 con 0x7f15040c6c50 2024-05-08T21:23:06.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.423+0000 7f150b837700 1 -- 172.21.15.3:0/3216481374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040c6c50 msgr2=0x7f15040ce970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:06.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.423+0000 7f150b837700 1 --2- 172.21.15.3:0/3216481374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040c6c50 0x7f15040ce970 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f14fc009d90 tx=0x7f14fc00b330 comp rx=0 tx=0).stop 2024-05-08T21:23:06.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.423+0000 7f150b837700 1 -- 172.21.15.3:0/3216481374 shutdown_connections 2024-05-08T21:23:06.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.423+0000 7f150b837700 1 --2- 172.21.15.3:0/3216481374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040c6c50 0x7f15040ce970 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:06.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.423+0000 7f150b837700 1 -- 172.21.15.3:0/3216481374 >> 172.21.15.3:0/3216481374 conn(0x7f150408afe0 msgr2=0x7f15040d3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:06.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.423+0000 7f150b837700 1 -- 172.21.15.3:0/3216481374 shutdown_connections 2024-05-08T21:23:06.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.423+0000 7f150b837700 1 -- 172.21.15.3:0/3216481374 wait complete. 2024-05-08T21:23:06.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.425+0000 7f150b837700 1 Processor -- start 2024-05-08T21:23:06.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.425+0000 7f150b837700 1 -- start start 2024-05-08T21:23:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.425+0000 7f150b837700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15040c6c50 0x7f15040b8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.425+0000 7f150b837700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15040d0240 0x7f15040c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.425+0000 7f150b837700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040d0d30 0x7f15040b1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.425+0000 7f150b837700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15040c8070 con 0x7f15040d0d30 2024-05-08T21:23:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.425+0000 7f150b837700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15040c82f0 con 0x7f15040d0240 2024-05-08T21:23:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.425+0000 7f150b837700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15040b63d0 con 0x7f15040c6c50 2024-05-08T21:23:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.425+0000 7f1509dd4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040d0d30 0x7f15040b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.426+0000 7f1509dd4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040d0d30 0x7f15040b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54180/0 (socket says 172.21.15.3:54180) 2024-05-08T21:23:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.426+0000 7f1509dd4700 1 -- 172.21.15.3:0/1339889414 learned_addr learned my addr 172.21.15.3:0/1339889414 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.426+0000 7f1508dd2700 1 --2- 172.21.15.3:0/1339889414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15040d0240 0x7f15040c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.426+0000 7f15095d3700 1 --2- 172.21.15.3:0/1339889414 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15040c6c50 0x7f15040b8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.426+0000 7f1509dd4700 1 -- 172.21.15.3:0/1339889414 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15040c6c50 msgr2=0x7f15040b8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:06.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.426+0000 7f1509dd4700 1 --2- 172.21.15.3:0/1339889414 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15040c6c50 0x7f15040b8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:06.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.426+0000 7f1509dd4700 1 -- 172.21.15.3:0/1339889414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15040d0240 msgr2=0x7f15040c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:06.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.426+0000 7f1509dd4700 1 --2- 172.21.15.3:0/1339889414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15040d0240 0x7f15040c7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:06.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.426+0000 7f1509dd4700 1 -- 172.21.15.3:0/1339889414 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14fc009a70 con 0x7f15040d0d30 2024-05-08T21:23:06.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.426+0000 7f1508dd2700 1 --2- 172.21.15.3:0/1339889414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15040d0240 0x7f15040c7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:06.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.426+0000 7f1509dd4700 1 --2- 172.21.15.3:0/1339889414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040d0d30 0x7f15040b1230 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f150408bb00 tx=0x7f14fc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:06.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.426+0000 7f14f9ffb700 1 -- 172.21.15.3:0/1339889414 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc012630 con 0x7f15040d0d30 2024-05-08T21:23:06.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.426+0000 7f14f9ffb700 1 -- 172.21.15.3:0/1339889414 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14fc012790 con 0x7f15040d0d30 2024-05-08T21:23:06.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.426+0000 7f150b837700 1 -- 172.21.15.3:0/1339889414 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15040b66b0 con 0x7f15040d0d30 2024-05-08T21:23:06.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.426+0000 7f14f9ffb700 1 -- 172.21.15.3:0/1339889414 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc012b40 con 0x7f15040d0d30 2024-05-08T21:23:06.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.427+0000 7f150b837700 1 -- 172.21.15.3:0/1339889414 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15040b6c70 con 0x7f15040d0d30 2024-05-08T21:23:06.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.428+0000 7f150b837700 1 -- 172.21.15.3:0/1339889414 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15040b6e90 con 0x7f15040d0d30 2024-05-08T21:23:06.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.429+0000 7f14f9ffb700 1 -- 172.21.15.3:0/1339889414 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f14fc02f480 con 0x7f15040d0d30 2024-05-08T21:23:06.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.429+0000 7f14f9ffb700 1 --2- 172.21.15.3:0/1339889414 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f14e806ac90 0x7f14e806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:06.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.429+0000 7f14f9ffb700 1 -- 172.21.15.3:0/1339889414 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f14fc08ba80 con 0x7f15040d0d30 2024-05-08T21:23:06.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.429+0000 7f15095d3700 1 --2- 172.21.15.3:0/1339889414 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f14e806ac90 0x7f14e806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:06.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.430+0000 7f15095d3700 1 --2- 172.21.15.3:0/1339889414 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f14e806ac90 0x7f14e806d150 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1500007fd0 tx=0x7f1500007ec0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:06.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.433+0000 7f14f9ffb700 1 -- 172.21.15.3:0/1339889414 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f14fc056bb0 con 0x7f15040d0d30 2024-05-08T21:23:06.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:06.702+0000 7f150b837700 1 -- 172.21.15.3:0/1339889414 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "recovery_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1504074bc0 con 0x7f15040d0d30 2024-05-08T21:23:07.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.633+0000 7f14f9ffb700 1 -- 172.21.15.3:0/1339889414 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "recovery_fs", "var": "standby_count_wanted", "val": "0"}]=0 v20) v1 ==== 123+0+0 (secure 0 0 0) 0x7f14fc05a1d0 con 0x7f15040d0d30 2024-05-08T21:23:07.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.636+0000 7f150b837700 1 -- 172.21.15.3:0/1339889414 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f14e806ac90 msgr2=0x7f14e806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:07.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.636+0000 7f150b837700 1 --2- 172.21.15.3:0/1339889414 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f14e806ac90 0x7f14e806d150 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1500007fd0 tx=0x7f1500007ec0 comp rx=0 tx=0).stop 2024-05-08T21:23:07.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.636+0000 7f150b837700 1 -- 172.21.15.3:0/1339889414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040d0d30 msgr2=0x7f15040b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:07.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.636+0000 7f150b837700 1 --2- 172.21.15.3:0/1339889414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040d0d30 0x7f15040b1230 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f150408bb00 tx=0x7f14fc004ad0 comp rx=0 tx=0).stop 2024-05-08T21:23:07.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.636+0000 7f150b837700 1 -- 172.21.15.3:0/1339889414 shutdown_connections 2024-05-08T21:23:07.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.636+0000 7f150b837700 1 --2- 172.21.15.3:0/1339889414 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15040c6c50 0x7f15040b8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:07.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.636+0000 7f150b837700 1 --2- 172.21.15.3:0/1339889414 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f14e806ac90 0x7f14e806d150 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:07.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.636+0000 7f150b837700 1 --2- 172.21.15.3:0/1339889414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040d0d30 0x7f15040b1230 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:07.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.636+0000 7f150b837700 1 --2- 172.21.15.3:0/1339889414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15040d0240 0x7f15040c7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:07.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.636+0000 7f150b837700 1 -- 172.21.15.3:0/1339889414 >> 172.21.15.3:0/1339889414 conn(0x7f150408afe0 msgr2=0x7f15040ce220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:07.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.636+0000 7f150b837700 1 -- 172.21.15.3:0/1339889414 shutdown_connections 2024-05-08T21:23:07.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.636+0000 7f150b837700 1 -- 172.21.15.3:0/1339889414 wait complete. 2024-05-08T21:23:07.654 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-08T21:23:07.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set recovery_fs allow_standby_replay false 2024-05-08T21:23:07.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.796+0000 7f9461699700 1 Processor -- start 2024-05-08T21:23:07.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.796+0000 7f9461699700 1 -- start start 2024-05-08T21:23:07.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.797+0000 7f9461699700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f945c161320 0x7f945c0ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:07.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.797+0000 7f9461699700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f945c0d2230 0x7f945c06bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:07.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.797+0000 7f9461699700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945c0cf560 con 0x7f945c0d1e50 2024-05-08T21:23:07.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.797+0000 7f9461699700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c0ce950 con 0x7f945c161320 2024-05-08T21:23:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.797+0000 7f9461699700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c0cec30 con 0x7f945c0d2230 2024-05-08T21:23:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.797+0000 7f945a7fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f945c161320 0x7f945c0ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.797+0000 7f945affd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f945c0d1e50 0x7f945c160c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:42880/0 (socket says 172.21.15.3:42880) 2024-05-08T21:23:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.797+0000 7f945affd700 1 -- 172.21.15.3:0/4052481472 learned_addr learned my addr 172.21.15.3:0/4052481472 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.797+0000 7f945b7fe700 1 --2- 172.21.15.3:0/4052481472 >> v2:172.21.15.186:3300/0 conn(0x7f945c0d2230 0x7f945c06bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.797+0000 7f945a7fc700 1 -- 172.21.15.3:0/4052481472 >> v2:172.21.15.186:3300/0 conn(0x7f945c0d2230 msgr2=0x7f945c06bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.797+0000 7f945a7fc700 1 --2- 172.21.15.3:0/4052481472 >> v2:172.21.15.186:3300/0 conn(0x7f945c0d2230 0x7f945c06bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.797+0000 7f945a7fc700 1 -- 172.21.15.3:0/4052481472 >> v1:172.21.15.3:6789/0 conn(0x7f945c0d1e50 legacy=0x7f945c160c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.797+0000 7f9459ffb700 1 -- 172.21.15.3:0/4052481472 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738446367 0 0) 0x7f945c0cf560 con 0x7f945c0d1e50 2024-05-08T21:23:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.797+0000 7f945a7fc700 1 -- 172.21.15.3:0/4052481472 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c0cef10 con 0x7f945c161320 2024-05-08T21:23:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.797+0000 7f945b7fe700 1 --2- 172.21.15.3:0/4052481472 >> v2:172.21.15.186:3300/0 conn(0x7f945c0d2230 0x7f945c06bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.798+0000 7f945a7fc700 1 --2- 172.21.15.3:0/4052481472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f945c161320 0x7f945c0ce2b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f945c0d3470 tx=0x7f9450009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55a9adba16f3c43b server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:07.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.798+0000 7f9459ffb700 1 -- 172.21.15.3:0/4052481472 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9450011830 con 0x7f945c161320 2024-05-08T21:23:07.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.798+0000 7f9459ffb700 1 -- 172.21.15.3:0/4052481472 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9450011990 con 0x7f945c161320 2024-05-08T21:23:07.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.798+0000 7f9459ffb700 1 -- 172.21.15.3:0/4052481472 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9450011d20 con 0x7f945c161320 2024-05-08T21:23:07.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.798+0000 7f9461699700 1 -- 172.21.15.3:0/4052481472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f945c161320 msgr2=0x7f945c0ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:07.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.798+0000 7f9461699700 1 --2- 172.21.15.3:0/4052481472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f945c161320 0x7f945c0ce2b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f945c0d3470 tx=0x7f9450009b20 comp rx=0 tx=0).stop 2024-05-08T21:23:07.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.798+0000 7f9461699700 1 -- 172.21.15.3:0/4052481472 shutdown_connections 2024-05-08T21:23:07.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.798+0000 7f9461699700 1 --2- 172.21.15.3:0/4052481472 >> v2:172.21.15.186:3300/0 conn(0x7f945c0d2230 0x7f945c06bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:07.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.798+0000 7f9461699700 1 --2- 172.21.15.3:0/4052481472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f945c161320 0x7f945c0ce2b0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:07.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.798+0000 7f9461699700 1 -- 172.21.15.3:0/4052481472 >> 172.21.15.3:0/4052481472 conn(0x7f945c08afe0 msgr2=0x7f945c0d35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:07.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.799+0000 7f9461699700 1 -- 172.21.15.3:0/4052481472 shutdown_connections 2024-05-08T21:23:07.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.799+0000 7f9461699700 1 -- 172.21.15.3:0/4052481472 wait complete. 2024-05-08T21:23:07.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.800+0000 7f9461699700 1 Processor -- start 2024-05-08T21:23:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.800+0000 7f9461699700 1 -- start start 2024-05-08T21:23:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f9461699700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f945c0d1e50 0x7f945c0ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f9461699700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f945c0d2230 0x7f945c14f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f9461699700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f945c161320 0x7f945c14fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f9461699700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c14d640 con 0x7f945c0d1e50 2024-05-08T21:23:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f9461699700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c0c9060 con 0x7f945c161320 2024-05-08T21:23:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f9461699700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c0c9340 con 0x7f945c0d2230 2024-05-08T21:23:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f945affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f945c0d1e50 0x7f945c0ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f945affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f945c0d1e50 0x7f945c0ccf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54204/0 (socket says 172.21.15.3:54204) 2024-05-08T21:23:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f945affd700 1 -- 172.21.15.3:0/3242555214 learned_addr learned my addr 172.21.15.3:0/3242555214 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f945b7fe700 1 --2- 172.21.15.3:0/3242555214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f945c161320 0x7f945c14fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f945affd700 1 -- 172.21.15.3:0/3242555214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f945c0d2230 msgr2=0x7f945c14f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f945a7fc700 1 --2- 172.21.15.3:0/3242555214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f945c0d2230 0x7f945c14f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f945affd700 1 --2- 172.21.15.3:0/3242555214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f945c0d2230 0x7f945c14f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f945affd700 1 -- 172.21.15.3:0/3242555214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f945c161320 msgr2=0x7f945c14fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f945affd700 1 --2- 172.21.15.3:0/3242555214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f945c161320 0x7f945c14fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f945affd700 1 -- 172.21.15.3:0/3242555214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94500094a0 con 0x7f945c0d1e50 2024-05-08T21:23:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f945b7fe700 1 --2- 172.21.15.3:0/3242555214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f945c161320 0x7f945c14fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.801+0000 7f945affd700 1 --2- 172.21.15.3:0/3242555214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f945c0d1e50 0x7f945c0ccf70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f944c0011e0 tx=0x7f944c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.802+0000 7f94437fe700 1 -- 172.21.15.3:0/3242555214 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c004c80 con 0x7f945c0d1e50 2024-05-08T21:23:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.802+0000 7f94437fe700 1 -- 172.21.15.3:0/3242555214 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944c004e00 con 0x7f945c0d1e50 2024-05-08T21:23:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.802+0000 7f9461699700 1 -- 172.21.15.3:0/3242555214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945c0c9620 con 0x7f945c0d1e50 2024-05-08T21:23:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.802+0000 7f94437fe700 1 -- 172.21.15.3:0/3242555214 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c007310 con 0x7f945c0d1e50 2024-05-08T21:23:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.802+0000 7f9461699700 1 -- 172.21.15.3:0/3242555214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945c0cd4b0 con 0x7f945c0d1e50 2024-05-08T21:23:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.803+0000 7f9461699700 1 -- 172.21.15.3:0/3242555214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945c0cf9d0 con 0x7f945c0d1e50 2024-05-08T21:23:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.803+0000 7f94437fe700 1 -- 172.21.15.3:0/3242555214 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f944c007470 con 0x7f945c0d1e50 2024-05-08T21:23:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.804+0000 7f94437fe700 1 --2- 172.21.15.3:0/3242555214 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f944806ac90 0x7f944806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:07.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.804+0000 7f94437fe700 1 -- 172.21.15.3:0/3242555214 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f944c089370 con 0x7f945c0d1e50 2024-05-08T21:23:07.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.804+0000 7f945a7fc700 1 --2- 172.21.15.3:0/3242555214 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f944806ac90 0x7f944806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:07.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.805+0000 7f945a7fc700 1 --2- 172.21.15.3:0/3242555214 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f944806ac90 0x7f944806d150 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f945c161720 tx=0x7f9450008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:07.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:07.809+0000 7f94437fe700 1 -- 172.21.15.3:0/3242555214 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f945c0cf9d0 con 0x7f945c0d1e50 2024-05-08T21:23:08.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.081+0000 7f9461699700 1 -- 172.21.15.3:0/3242555214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "recovery_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f945c0cf9d0 con 0x7f945c0d1e50 2024-05-08T21:23:08.261 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-08T21:23:08.260+0000 7f74f9e57700 -1 log_channel(cluster) log [ERR] : Health check update: 2 filesystems are offline (MDS_ALL_DOWN) 2024-05-08T21:23:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.649+0000 7f94437fe700 1 -- 172.21.15.3:0/3242555214 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "recovery_fs", "var": "allow_standby_replay", "val": "false"}]=0 v21) v1 ==== 127+0+0 (secure 0 0 0) 0x7f945c0cf9d0 con 0x7f945c0d1e50 2024-05-08T21:23:08.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.652+0000 7f9461699700 1 -- 172.21.15.3:0/3242555214 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f944806ac90 msgr2=0x7f944806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:08.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.652+0000 7f9461699700 1 --2- 172.21.15.3:0/3242555214 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f944806ac90 0x7f944806d150 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f945c161720 tx=0x7f9450008030 comp rx=0 tx=0).stop 2024-05-08T21:23:08.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.652+0000 7f9461699700 1 -- 172.21.15.3:0/3242555214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f945c0d1e50 msgr2=0x7f945c0ccf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:08.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.652+0000 7f9461699700 1 --2- 172.21.15.3:0/3242555214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f945c0d1e50 0x7f945c0ccf70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f944c0011e0 tx=0x7f944c00cae0 comp rx=0 tx=0).stop 2024-05-08T21:23:08.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.652+0000 7f9461699700 1 -- 172.21.15.3:0/3242555214 shutdown_connections 2024-05-08T21:23:08.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.652+0000 7f9461699700 1 --2- 172.21.15.3:0/3242555214 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f944806ac90 0x7f944806d150 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:08.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.652+0000 7f9461699700 1 --2- 172.21.15.3:0/3242555214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f945c161320 0x7f945c14fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:08.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.652+0000 7f9461699700 1 --2- 172.21.15.3:0/3242555214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f945c0d1e50 0x7f945c0ccf70 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:08.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.652+0000 7f9461699700 1 --2- 172.21.15.3:0/3242555214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f945c0d2230 0x7f945c14f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:08.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.652+0000 7f9461699700 1 -- 172.21.15.3:0/3242555214 >> 172.21.15.3:0/3242555214 conn(0x7f945c08afe0 msgr2=0x7f945c15fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:08.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.652+0000 7f9461699700 1 -- 172.21.15.3:0/3242555214 shutdown_connections 2024-05-08T21:23:08.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.652+0000 7f9461699700 1 -- 172.21.15.3:0/3242555214 wait complete. 2024-05-08T21:23:08.670 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'recovery_fs' 2024-05-08T21:23:08.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:23:08.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.809+0000 7fb45b6db700 1 Processor -- start 2024-05-08T21:23:08.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.809+0000 7fb45b6db700 1 -- start start 2024-05-08T21:23:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.810+0000 7fb45b6db700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb45406e4d0 0x7fb45415e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.810+0000 7fb45b6db700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4540b3990 0x7fb45406a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.810+0000 7fb45b6db700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4540b4a80 con 0x7fb4540af030 2024-05-08T21:23:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.810+0000 7fb45b6db700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb45415ede0 con 0x7fb45406e4d0 2024-05-08T21:23:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.810+0000 7fb45b6db700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb45406d800 con 0x7fb4540b3990 2024-05-08T21:23:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.810+0000 7fb458c76700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb45406e4d0 0x7fb45415e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.810+0000 7fb459c78700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4540b3990 0x7fb45406a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.810+0000 7fb459c78700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4540b3990 0x7fb45406a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.3:56474/0 (socket says 172.21.15.3:56474) 2024-05-08T21:23:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.810+0000 7fb458c76700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb45406e4d0 0x7fb45415e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:39138/0 (socket says 172.21.15.3:39138) 2024-05-08T21:23:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.810+0000 7fb459c78700 1 -- 172.21.15.3:0/925961425 learned_addr learned my addr 172.21.15.3:0/925961425 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:08.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.811+0000 7fb459c78700 1 -- 172.21.15.3:0/925961425 >> v2:172.21.15.186:3300/0 conn(0x7fb45406e4d0 msgr2=0x7fb45415e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:08.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.811+0000 7fb459c78700 1 --2- 172.21.15.3:0/925961425 >> v2:172.21.15.186:3300/0 conn(0x7fb45406e4d0 0x7fb45415e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:08.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.811+0000 7fb459c78700 1 -- 172.21.15.3:0/925961425 >> v1:172.21.15.186:6789/0 conn(0x7fb4540af030 legacy=0x7fb45406ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:08.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.811+0000 7fb459c78700 1 -- 172.21.15.3:0/925961425 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4540b3e70 con 0x7fb4540b3990 2024-05-08T21:23:08.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.811+0000 7fb458c76700 1 --2- 172.21.15.3:0/925961425 >> v2:172.21.15.186:3300/0 conn(0x7fb45406e4d0 0x7fb45415e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:08.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.811+0000 7fb459c78700 1 --2- 172.21.15.3:0/925961425 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4540b3990 0x7fb45406a960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb44c009e50 tx=0x7fb44c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c37367e58d89d1d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:08.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.811+0000 7fb447fff700 1 -- 172.21.15.3:0/925961425 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c00e070 con 0x7fb4540b3990 2024-05-08T21:23:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.811+0000 7fb447fff700 1 -- 172.21.15.3:0/925961425 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44c00b910 con 0x7fb4540b3990 2024-05-08T21:23:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.811+0000 7fb447fff700 1 -- 172.21.15.3:0/925961425 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c00bcc0 con 0x7fb4540b3990 2024-05-08T21:23:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.812+0000 7fb45b6db700 1 -- 172.21.15.3:0/925961425 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4540b3990 msgr2=0x7fb45406a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.812+0000 7fb45b6db700 1 --2- 172.21.15.3:0/925961425 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4540b3990 0x7fb45406a960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb44c009e50 tx=0x7fb44c00b330 comp rx=0 tx=0).stop 2024-05-08T21:23:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.812+0000 7fb45b6db700 1 -- 172.21.15.3:0/925961425 shutdown_connections 2024-05-08T21:23:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.812+0000 7fb45b6db700 1 --2- 172.21.15.3:0/925961425 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4540b3990 0x7fb45406a960 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.812+0000 7fb45b6db700 1 --2- 172.21.15.3:0/925961425 >> v2:172.21.15.186:3300/0 conn(0x7fb45406e4d0 0x7fb45415e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.812+0000 7fb45b6db700 1 -- 172.21.15.3:0/925961425 >> 172.21.15.3:0/925961425 conn(0x7fb4540cfce0 msgr2=0x7fb4540d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.812+0000 7fb45b6db700 1 -- 172.21.15.3:0/925961425 shutdown_connections 2024-05-08T21:23:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.812+0000 7fb45b6db700 1 -- 172.21.15.3:0/925961425 wait complete. 2024-05-08T21:23:08.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.813+0000 7fb45b6db700 1 Processor -- start 2024-05-08T21:23:08.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.813+0000 7fb45b6db700 1 -- start start 2024-05-08T21:23:08.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.814+0000 7fb45b6db700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb45406e4d0 0x7fb454154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:08.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.814+0000 7fb45b6db700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4540af030 0x7fb4540cde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:08.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.814+0000 7fb45b6db700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4540b3990 0x7fb4541508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:08.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.814+0000 7fb45b6db700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb454155fa0 con 0x7fb4540b3990 2024-05-08T21:23:08.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.814+0000 7fb45b6db700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb454155080 con 0x7fb4540af030 2024-05-08T21:23:08.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.814+0000 7fb45b6db700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb454155320 con 0x7fb45406e4d0 2024-05-08T21:23:08.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.814+0000 7fb459c78700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4540b3990 0x7fb4541508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:08.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.814+0000 7fb459c78700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4540b3990 0x7fb4541508a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54220/0 (socket says 172.21.15.3:54220) 2024-05-08T21:23:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.814+0000 7fb459c78700 1 -- 172.21.15.3:0/3305474193 learned_addr learned my addr 172.21.15.3:0/3305474193 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.814+0000 7fb458c76700 1 --2- 172.21.15.3:0/3305474193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4540af030 0x7fb4540cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.814+0000 7fb459477700 1 --2- 172.21.15.3:0/3305474193 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb45406e4d0 0x7fb454154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.815+0000 7fb459c78700 1 -- 172.21.15.3:0/3305474193 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb45406e4d0 msgr2=0x7fb454154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.815+0000 7fb459c78700 1 --2- 172.21.15.3:0/3305474193 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb45406e4d0 0x7fb454154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.815+0000 7fb459c78700 1 -- 172.21.15.3:0/3305474193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4540af030 msgr2=0x7fb4540cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.815+0000 7fb459c78700 1 --2- 172.21.15.3:0/3305474193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4540af030 0x7fb4540cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.815+0000 7fb459c78700 1 -- 172.21.15.3:0/3305474193 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4500094a0 con 0x7fb4540b3990 2024-05-08T21:23:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.815+0000 7fb459477700 1 --2- 172.21.15.3:0/3305474193 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb45406e4d0 0x7fb454154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.815+0000 7fb459c78700 1 --2- 172.21.15.3:0/3305474193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4540b3990 0x7fb4541508a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb44c012000 tx=0x7fb44c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.815+0000 7fb445ffb700 1 -- 172.21.15.3:0/3305474193 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c0048b0 con 0x7fb4540b3990 2024-05-08T21:23:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.815+0000 7fb445ffb700 1 -- 172.21.15.3:0/3305474193 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44c0092e0 con 0x7fb4540b3990 2024-05-08T21:23:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.815+0000 7fb45b6db700 1 -- 172.21.15.3:0/3305474193 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb44c009b30 con 0x7fb4540b3990 2024-05-08T21:23:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.815+0000 7fb445ffb700 1 -- 172.21.15.3:0/3305474193 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c01c660 con 0x7fb4540b3990 2024-05-08T21:23:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.815+0000 7fb45b6db700 1 -- 172.21.15.3:0/3305474193 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4541558e0 con 0x7fb4540b3990 2024-05-08T21:23:08.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.816+0000 7fb45b6db700 1 -- 172.21.15.3:0/3305474193 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4541560e0 con 0x7fb4540b3990 2024-05-08T21:23:08.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.817+0000 7fb445ffb700 1 -- 172.21.15.3:0/3305474193 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb44c01c7c0 con 0x7fb4540b3990 2024-05-08T21:23:08.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.817+0000 7fb445ffb700 1 --2- 172.21.15.3:0/3305474193 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb43806ac90 0x7fb43806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:08.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.818+0000 7fb445ffb700 1 -- 172.21.15.3:0/3305474193 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7fb44c08ab00 con 0x7fb4540b3990 2024-05-08T21:23:08.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.818+0000 7fb459477700 1 --2- 172.21.15.3:0/3305474193 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb43806ac90 0x7fb43806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:08.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.818+0000 7fb459477700 1 --2- 172.21.15.3:0/3305474193 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb43806ac90 0x7fb43806d150 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb448000f60 tx=0x7fb448009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:08.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:08.822+0000 7fb445ffb700 1 -- 172.21.15.3:0/3305474193 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb4541560e0 con 0x7fb4540b3990 2024-05-08T21:23:09.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.087+0000 7fb45b6db700 1 -- 172.21.15.3:0/3305474193 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4541560e0 con 0x7fb4540b3990 2024-05-08T21:23:09.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.088+0000 7fb445ffb700 1 -- 172.21.15.3:0/3305474193 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5928 (secure 0 0 0) 0x7fb4541560e0 con 0x7fb4540b3990 2024-05-08T21:23:09.089 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:09.089 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6833/2104267343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2104267343},{"type":"v1","addr":"172.21.15.3:6833","nonce":2104267343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/208321708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":208321708},{"type":"v1","addr":"172.21.15.3:6835","nonce":208321708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/1591386131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1591386131},{"type":"v1","addr":"172.21.15.186:6836","nonce":1591386131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5191,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/2000072997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":2000072997},{"type":"v1","addr":"172.21.15.186:6837","nonce":2000072997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:22:50.184245+0000","modified":"2024-05-08T21:23:01.563810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2},{"mdsmap":{"epoch":21,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:23:05.475355+0000","modified":"2024-05-08T21:23:08.637923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[5],"metadata_pool":6,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-08T21:23:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.090+0000 7fb45b6db700 1 -- 172.21.15.3:0/3305474193 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb43806ac90 msgr2=0x7fb43806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.090+0000 7fb45b6db700 1 --2- 172.21.15.3:0/3305474193 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb43806ac90 0x7fb43806d150 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb448000f60 tx=0x7fb448009040 comp rx=0 tx=0).stop 2024-05-08T21:23:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.090+0000 7fb45b6db700 1 -- 172.21.15.3:0/3305474193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4540b3990 msgr2=0x7fb4541508a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.090+0000 7fb45b6db700 1 --2- 172.21.15.3:0/3305474193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4540b3990 0x7fb4541508a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb44c012000 tx=0x7fb44c004ad0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.090+0000 7fb45b6db700 1 -- 172.21.15.3:0/3305474193 shutdown_connections 2024-05-08T21:23:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.090+0000 7fb45b6db700 1 --2- 172.21.15.3:0/3305474193 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb45406e4d0 0x7fb454154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.090+0000 7fb45b6db700 1 --2- 172.21.15.3:0/3305474193 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb43806ac90 0x7fb43806d150 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.090+0000 7fb45b6db700 1 --2- 172.21.15.3:0/3305474193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4540b3990 0x7fb4541508a0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.090+0000 7fb45b6db700 1 --2- 172.21.15.3:0/3305474193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4540af030 0x7fb4540cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.090+0000 7fb45b6db700 1 -- 172.21.15.3:0/3305474193 >> 172.21.15.3:0/3305474193 conn(0x7fb4540cfce0 msgr2=0x7fb4540b03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.090+0000 7fb45b6db700 1 -- 172.21.15.3:0/3305474193 shutdown_connections 2024-05-08T21:23:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.090+0000 7fb45b6db700 1 -- 172.21.15.3:0/3305474193 wait complete. 2024-05-08T21:23:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 21 2024-05-08T21:23:09.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:23:09.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.239+0000 7fb481062700 1 Processor -- start 2024-05-08T21:23:09.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.239+0000 7fb481062700 1 -- start start 2024-05-08T21:23:09.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.240+0000 7fb481062700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cf840 0x7fb47c0af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:09.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.240+0000 7fb481062700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb47c06a960 0x7fb47c15d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:09.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.240+0000 7fb481062700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47c0cd980 con 0x7fb47c0d0eb0 2024-05-08T21:23:09.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.240+0000 7fb481062700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c0ccdb0 con 0x7fb47c06a960 2024-05-08T21:23:09.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.240+0000 7fb481062700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0cd050 con 0x7fb47c0cf840 2024-05-08T21:23:09.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.240+0000 7fb47a59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb47c06a960 0x7fb47c15d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:09.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.240+0000 7fb47b59e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb47c0d0eb0 0x7fb47c06dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:42882/0 (socket says 172.21.15.3:42882) 2024-05-08T21:23:09.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.240+0000 7fb47b59e700 1 -- 172.21.15.3:0/1501383293 learned_addr learned my addr 172.21.15.3:0/1501383293 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.240+0000 7fb47ad9d700 1 --2- 172.21.15.3:0/1501383293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cf840 0x7fb47c0af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.240+0000 7fb47a59c700 1 -- 172.21.15.3:0/1501383293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cf840 msgr2=0x7fb47c0af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.240+0000 7fb47a59c700 1 --2- 172.21.15.3:0/1501383293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cf840 0x7fb47c0af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.240+0000 7fb47a59c700 1 -- 172.21.15.3:0/1501383293 >> v1:172.21.15.3:6789/0 conn(0x7fb47c0d0eb0 legacy=0x7fb47c06dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.240+0000 7fb47a59c700 1 -- 172.21.15.3:0/1501383293 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c0cd330 con 0x7fb47c06a960 2024-05-08T21:23:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.240+0000 7fb47ad9d700 1 --2- 172.21.15.3:0/1501383293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cf840 0x7fb47c0af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.240+0000 7fb479d9b700 1 -- 172.21.15.3:0/1501383293 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749936540 0 0) 0x7fb47c0cd980 con 0x7fb47c0d0eb0 2024-05-08T21:23:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.241+0000 7fb47a59c700 1 --2- 172.21.15.3:0/1501383293 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb47c06a960 0x7fb47c15d4f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb470009870 tx=0x7fb470009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=353965c741ad3c10 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.241+0000 7fb479d9b700 1 -- 172.21.15.3:0/1501383293 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470011830 con 0x7fb47c06a960 2024-05-08T21:23:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.241+0000 7fb479d9b700 1 -- 172.21.15.3:0/1501383293 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb470011990 con 0x7fb47c06a960 2024-05-08T21:23:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.241+0000 7fb479d9b700 1 -- 172.21.15.3:0/1501383293 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470011d20 con 0x7fb47c06a960 2024-05-08T21:23:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.241+0000 7fb481062700 1 -- 172.21.15.3:0/1501383293 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb47c06a960 msgr2=0x7fb47c15d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.241+0000 7fb481062700 1 --2- 172.21.15.3:0/1501383293 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb47c06a960 0x7fb47c15d4f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb470009870 tx=0x7fb470009b80 comp rx=0 tx=0).stop 2024-05-08T21:23:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.241+0000 7fb481062700 1 -- 172.21.15.3:0/1501383293 shutdown_connections 2024-05-08T21:23:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.241+0000 7fb481062700 1 --2- 172.21.15.3:0/1501383293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cf840 0x7fb47c0af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.241+0000 7fb481062700 1 --2- 172.21.15.3:0/1501383293 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb47c06a960 0x7fb47c15d4f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.241+0000 7fb481062700 1 -- 172.21.15.3:0/1501383293 >> 172.21.15.3:0/1501383293 conn(0x7fb47c0d0320 msgr2=0x7fb47c0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:09.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.241+0000 7fb481062700 1 -- 172.21.15.3:0/1501383293 shutdown_connections 2024-05-08T21:23:09.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.241+0000 7fb481062700 1 -- 172.21.15.3:0/1501383293 wait complete. 2024-05-08T21:23:09.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.243+0000 7fb481062700 1 Processor -- start 2024-05-08T21:23:09.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.243+0000 7fb481062700 1 -- start start 2024-05-08T21:23:09.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.244+0000 7fb481062700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb47c06a960 0x7fb47c0b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:09.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.244+0000 7fb481062700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cf840 0x7fb47c0c5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:09.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.244+0000 7fb481062700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb47c0d0eb0 0x7fb47c0c3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:09.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.244+0000 7fb481062700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c156080 con 0x7fb47c06a960 2024-05-08T21:23:09.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.244+0000 7fb481062700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0b4440 con 0x7fb47c0d0eb0 2024-05-08T21:23:09.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.244+0000 7fb481062700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0c5860 con 0x7fb47c0cf840 2024-05-08T21:23:09.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.244+0000 7fb47ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb47c06a960 0x7fb47c0b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:09.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.244+0000 7fb47ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb47c06a960 0x7fb47c0b31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54248/0 (socket says 172.21.15.3:54248) 2024-05-08T21:23:09.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.244+0000 7fb47ad9d700 1 -- 172.21.15.3:0/669742267 learned_addr learned my addr 172.21.15.3:0/669742267 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:09.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.244+0000 7fb47a59c700 1 --2- 172.21.15.3:0/669742267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cf840 0x7fb47c0c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:09.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.244+0000 7fb47b59e700 1 --2- 172.21.15.3:0/669742267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb47c0d0eb0 0x7fb47c0c3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:09.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.244+0000 7fb47ad9d700 1 -- 172.21.15.3:0/669742267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cf840 msgr2=0x7fb47c0c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:09.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.244+0000 7fb47ad9d700 1 --2- 172.21.15.3:0/669742267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cf840 0x7fb47c0c5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.244+0000 7fb47ad9d700 1 -- 172.21.15.3:0/669742267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb47c0d0eb0 msgr2=0x7fb47c0c3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:09.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.244+0000 7fb47ad9d700 1 --2- 172.21.15.3:0/669742267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb47c0d0eb0 0x7fb47c0c3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.244+0000 7fb47ad9d700 1 -- 172.21.15.3:0/669742267 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4700094a0 con 0x7fb47c06a960 2024-05-08T21:23:09.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.245+0000 7fb47ad9d700 1 --2- 172.21.15.3:0/669742267 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb47c06a960 0x7fb47c0b31c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00e810 tx=0x7fb46c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:09.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.245+0000 7fb4637fe700 1 -- 172.21.15.3:0/669742267 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c004230 con 0x7fb47c06a960 2024-05-08T21:23:09.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.245+0000 7fb4637fe700 1 -- 172.21.15.3:0/669742267 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46c004390 con 0x7fb47c06a960 2024-05-08T21:23:09.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.245+0000 7fb481062700 1 -- 172.21.15.3:0/669742267 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c0b3830 con 0x7fb47c06a960 2024-05-08T21:23:09.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.245+0000 7fb4637fe700 1 -- 172.21.15.3:0/669742267 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c0036a0 con 0x7fb47c06a960 2024-05-08T21:23:09.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.245+0000 7fb481062700 1 -- 172.21.15.3:0/669742267 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c0b3df0 con 0x7fb47c06a960 2024-05-08T21:23:09.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.246+0000 7fb481062700 1 -- 172.21.15.3:0/669742267 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c0b4720 con 0x7fb47c06a960 2024-05-08T21:23:09.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.246+0000 7fb4637fe700 1 -- 172.21.15.3:0/669742267 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb46c003800 con 0x7fb47c06a960 2024-05-08T21:23:09.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.247+0000 7fb4637fe700 1 --2- 172.21.15.3:0/669742267 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb46806ac90 0x7fb46806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:09.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.247+0000 7fb4637fe700 1 -- 172.21.15.3:0/669742267 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7fb46c088f40 con 0x7fb47c06a960 2024-05-08T21:23:09.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.247+0000 7fb47a59c700 1 --2- 172.21.15.3:0/669742267 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb46806ac90 0x7fb46806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:09.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.248+0000 7fb47a59c700 1 --2- 172.21.15.3:0/669742267 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb46806ac90 0x7fb46806d150 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb470009870 tx=0x7fb470008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:09.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.252+0000 7fb4637fe700 1 -- 172.21.15.3:0/669742267 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb46c053f90 con 0x7fb47c06a960 2024-05-08T21:23:09.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.470+0000 7fb481062700 1 -- 172.21.15.3:0/669742267 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb47c0cde70 con 0x7fb47c06a960 2024-05-08T21:23:09.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.472+0000 7fb4637fe700 1 -- 172.21.15.3:0/669742267 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+18029 (secure 0 0 0) 0x7fb46c0575b0 con 0x7fb47c06a960 2024-05-08T21:23:09.474 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:09.474 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":42,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:23:06.476868+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:22:47.853786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-08T21:22:49.153759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"recovery_fs_metadata","create_time":"2024-05-08T21:23:03.921695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6834/3599481370":"2024-05-09T21:23:01.563392+0000","172.21.15.3:6832/3599481370":"2024-05-09T21:23:01.563392+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:23:09.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.474+0000 7fb481062700 1 -- 172.21.15.3:0/669742267 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb46806ac90 msgr2=0x7fb46806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:09.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.474+0000 7fb481062700 1 --2- 172.21.15.3:0/669742267 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb46806ac90 0x7fb46806d150 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb470009870 tx=0x7fb470008030 comp rx=0 tx=0).stop 2024-05-08T21:23:09.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.474+0000 7fb481062700 1 -- 172.21.15.3:0/669742267 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb47c06a960 msgr2=0x7fb47c0b31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:09.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.474+0000 7fb481062700 1 --2- 172.21.15.3:0/669742267 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb47c06a960 0x7fb47c0b31c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00e810 tx=0x7fb46c00ebd0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.474+0000 7fb481062700 1 -- 172.21.15.3:0/669742267 shutdown_connections 2024-05-08T21:23:09.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.474+0000 7fb481062700 1 --2- 172.21.15.3:0/669742267 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb46806ac90 0x7fb46806d150 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.474+0000 7fb481062700 1 --2- 172.21.15.3:0/669742267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb47c0d0eb0 0x7fb47c0c3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.474+0000 7fb481062700 1 --2- 172.21.15.3:0/669742267 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb47c06a960 0x7fb47c0b31c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.474+0000 7fb481062700 1 --2- 172.21.15.3:0/669742267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cf840 0x7fb47c0c5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.474+0000 7fb481062700 1 -- 172.21.15.3:0/669742267 >> 172.21.15.3:0/669742267 conn(0x7fb47c0d0320 msgr2=0x7fb47c15c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:09.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.474+0000 7fb481062700 1 -- 172.21.15.3:0/669742267 shutdown_connections 2024-05-08T21:23:09.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.474+0000 7fb481062700 1 -- 172.21.15.3:0/669742267 wait complete. 2024-05-08T21:23:09.490 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-08T21:23:09.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T21:23:09.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.624+0000 7faa41827700 1 Processor -- start 2024-05-08T21:23:09.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.624+0000 7faa41827700 1 -- start start 2024-05-08T21:23:09.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.625+0000 7faa41827700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa3c0cd4e0 0x7faa3c06f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:09.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.625+0000 7faa41827700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa3c15c280 con 0x7faa3c0cecf0 2024-05-08T21:23:09.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.625+0000 7faa41827700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa3c06e780 con 0x7faa3c06f130 2024-05-08T21:23:09.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.625+0000 7faa41827700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c06ea60 con 0x7faa3c0cd4e0 2024-05-08T21:23:09.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.625+0000 7faa3b7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7faa3c0cecf0 0x7faa3c15c710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:42884/0 (socket says 172.21.15.3:42884) 2024-05-08T21:23:09.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.625+0000 7faa3b7fe700 1 -- 172.21.15.3:0/3675358940 learned_addr learned my addr 172.21.15.3:0/3675358940 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:09.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa3affd700 1 --2- 172.21.15.3:0/3675358940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa3c0cd4e0 0x7faa3c06f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:09.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa39ffb700 1 -- 172.21.15.3:0/3675358940 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808808 0 0) 0x7faa3c15c280 con 0x7faa3c0cecf0 2024-05-08T21:23:09.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa39ffb700 1 -- 172.21.15.3:0/3675358940 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa240035b0 con 0x7faa3c0cecf0 2024-05-08T21:23:09.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa39ffb700 1 -- 172.21.15.3:0/3675358940 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3834432077 0 0) 0x7faa240035b0 con 0x7faa3c0cecf0 2024-05-08T21:23:09.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa39ffb700 1 -- 172.21.15.3:0/3675358940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa3c0cd4e0 msgr2=0x7faa3c06f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:09.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa39ffb700 1 --2- 172.21.15.3:0/3675358940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa3c0cd4e0 0x7faa3c06f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa39ffb700 1 -- 172.21.15.3:0/3675358940 >> v1:172.21.15.186:6789/0 conn(0x7faa3c06f130 legacy=0x7faa3c0ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:09.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa39ffb700 1 -- 172.21.15.3:0/3675358940 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c06ed40 con 0x7faa3c0cecf0 2024-05-08T21:23:09.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa39ffb700 1 -- 172.21.15.3:0/3675358940 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7faa340019f0 con 0x7faa3c0cecf0 2024-05-08T21:23:09.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa39ffb700 1 -- 172.21.15.3:0/3675358940 >> v1:172.21.15.3:6789/0 conn(0x7faa3c0cecf0 legacy=0x7faa3c15c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:09.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa39ffb700 1 --2- 172.21.15.3:0/3675358940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa24003a30 0x7faa24003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:09.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa39ffb700 1 --2- 172.21.15.3:0/3675358940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa3c0cd4e0 0x7faa24004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:09.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa39ffb700 1 --2- 172.21.15.3:0/3675358940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa24004d90 0x7faa24005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:09.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa39ffb700 1 -- 172.21.15.3:0/3675358940 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c06ea60 con 0x7faa24004d90 2024-05-08T21:23:09.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa39ffb700 1 -- 172.21.15.3:0/3675358940 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2400bac0 con 0x7faa24003a30 2024-05-08T21:23:09.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa39ffb700 1 -- 172.21.15.3:0/3675358940 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2400bc30 con 0x7faa3c0cd4e0 2024-05-08T21:23:09.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa3b7fe700 1 --2- 172.21.15.3:0/3675358940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa24004d90 0x7faa24005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:09.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa3affd700 1 --2- 172.21.15.3:0/3675358940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa24003a30 0x7faa24003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:09.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.626+0000 7faa3a7fc700 1 --2- 172.21.15.3:0/3675358940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa3c0cd4e0 0x7faa24004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:09.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.627+0000 7faa3b7fe700 1 -- 172.21.15.3:0/3675358940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa3c0cd4e0 msgr2=0x7faa24004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:09.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.627+0000 7faa3b7fe700 1 --2- 172.21.15.3:0/3675358940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa3c0cd4e0 0x7faa24004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.627+0000 7faa3b7fe700 1 -- 172.21.15.3:0/3675358940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa24003a30 msgr2=0x7faa24003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:09.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.627+0000 7faa3b7fe700 1 --2- 172.21.15.3:0/3675358940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa24003a30 0x7faa24003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.627+0000 7faa3b7fe700 1 -- 172.21.15.3:0/3675358940 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa240035b0 con 0x7faa24004d90 2024-05-08T21:23:09.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.627+0000 7faa3b7fe700 1 --2- 172.21.15.3:0/3675358940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa24004d90 0x7faa24005210 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7faa3400a810 tx=0x7faa24005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6aaf93e2b2f16a server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:09.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.627+0000 7faa39ffb700 1 -- 172.21.15.3:0/3675358940 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3400a250 con 0x7faa24004d90 2024-05-08T21:23:09.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.627+0000 7faa39ffb700 1 -- 172.21.15.3:0/3675358940 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa34012600 con 0x7faa24004d90 2024-05-08T21:23:09.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.627+0000 7faa39ffb700 1 -- 172.21.15.3:0/3675358940 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa340129b0 con 0x7faa24004d90 2024-05-08T21:23:09.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.628+0000 7faa41827700 1 -- 172.21.15.3:0/3675358940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa24004d90 msgr2=0x7faa24005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:09.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.628+0000 7faa41827700 1 --2- 172.21.15.3:0/3675358940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa24004d90 0x7faa24005210 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7faa3400a810 tx=0x7faa24005880 comp rx=0 tx=0).stop 2024-05-08T21:23:09.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.628+0000 7faa3b7fe700 1 -- 172.21.15.3:0/3675358940 reap_dead start 2024-05-08T21:23:09.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.628+0000 7faa41827700 1 -- 172.21.15.3:0/3675358940 shutdown_connections 2024-05-08T21:23:09.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.628+0000 7faa41827700 1 -- 172.21.15.3:0/3675358940 >> 172.21.15.3:0/3675358940 conn(0x7faa3c0d0320 msgr2=0x7faa3c0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:09.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.628+0000 7faa41827700 1 -- 172.21.15.3:0/3675358940 shutdown_connections 2024-05-08T21:23:09.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.628+0000 7faa41827700 1 -- 172.21.15.3:0/3675358940 wait complete. 2024-05-08T21:23:09.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.629+0000 7faa41827700 1 Processor -- start 2024-05-08T21:23:09.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.629+0000 7faa41827700 1 -- start start 2024-05-08T21:23:09.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.630+0000 7faa41827700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa3c1508c0 0x7faa3c0c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:09.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.630+0000 7faa41827700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa3c150ca0 0x7faa3c0b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:09.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.630+0000 7faa41827700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa3c0b1b00 0x7faa3c1f21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:09.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.630+0000 7faa41827700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c0b1f90 con 0x7faa3c150ca0 2024-05-08T21:23:09.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.630+0000 7faa41827700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c0b4670 con 0x7faa3c0b1b00 2024-05-08T21:23:09.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.630+0000 7faa41827700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c0c4d50 con 0x7faa3c1508c0 2024-05-08T21:23:09.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.630+0000 7faa3a7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa3c150ca0 0x7faa3c0b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:09.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.630+0000 7faa3a7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa3c150ca0 0x7faa3c0b15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54260/0 (socket says 172.21.15.3:54260) 2024-05-08T21:23:09.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.630+0000 7faa3a7fc700 1 -- 172.21.15.3:0/1518305232 learned_addr learned my addr 172.21.15.3:0/1518305232 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:09.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.630+0000 7faa3affd700 1 --2- 172.21.15.3:0/1518305232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa3c1508c0 0x7faa3c0c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:09.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.630+0000 7faa3b7fe700 1 --2- 172.21.15.3:0/1518305232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa3c0b1b00 0x7faa3c1f21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:09.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.630+0000 7faa3a7fc700 1 -- 172.21.15.3:0/1518305232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa3c1508c0 msgr2=0x7faa3c0c4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:09.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.630+0000 7faa3a7fc700 1 --2- 172.21.15.3:0/1518305232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa3c1508c0 0x7faa3c0c4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.630+0000 7faa3a7fc700 1 -- 172.21.15.3:0/1518305232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa3c0b1b00 msgr2=0x7faa3c1f21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:09.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.630+0000 7faa3a7fc700 1 --2- 172.21.15.3:0/1518305232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa3c0b1b00 0x7faa3c1f21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.630+0000 7faa3a7fc700 1 -- 172.21.15.3:0/1518305232 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c0b3a60 con 0x7faa3c150ca0 2024-05-08T21:23:09.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.631+0000 7faa3a7fc700 1 --2- 172.21.15.3:0/1518305232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa3c150ca0 0x7faa3c0b15c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7faa3000f7f0 tx=0x7faa3000fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:09.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.631+0000 7faa23fff700 1 -- 172.21.15.3:0/1518305232 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa300052e0 con 0x7faa3c150ca0 2024-05-08T21:23:09.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.631+0000 7faa23fff700 1 -- 172.21.15.3:0/1518305232 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa30005460 con 0x7faa3c150ca0 2024-05-08T21:23:09.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.631+0000 7faa41827700 1 -- 172.21.15.3:0/1518305232 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa3c0b3d40 con 0x7faa3c150ca0 2024-05-08T21:23:09.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.631+0000 7faa23fff700 1 -- 172.21.15.3:0/1518305232 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa300038c0 con 0x7faa3c150ca0 2024-05-08T21:23:09.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.631+0000 7faa41827700 1 -- 172.21.15.3:0/1518305232 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa3c0b4300 con 0x7faa3c150ca0 2024-05-08T21:23:09.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.632+0000 7faa41827700 1 -- 172.21.15.3:0/1518305232 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3c0b44c0 con 0x7faa3c150ca0 2024-05-08T21:23:09.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.632+0000 7faa23fff700 1 -- 172.21.15.3:0/1518305232 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faa30003a20 con 0x7faa3c150ca0 2024-05-08T21:23:09.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.633+0000 7faa23fff700 1 --2- 172.21.15.3:0/1518305232 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faa2806ac90 0x7faa2806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:09.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.633+0000 7faa23fff700 1 -- 172.21.15.3:0/1518305232 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7faa30089480 con 0x7faa3c150ca0 2024-05-08T21:23:09.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.633+0000 7faa3affd700 1 --2- 172.21.15.3:0/1518305232 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faa2806ac90 0x7faa2806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:09.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.633+0000 7faa3affd700 1 --2- 172.21.15.3:0/1518305232 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faa2806ac90 0x7faa2806d150 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7faa2c009ba0 tx=0x7faa2c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:09.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.637+0000 7faa23fff700 1 -- 172.21.15.3:0/1518305232 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7faa30054440 con 0x7faa3c150ca0 2024-05-08T21:23:09.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.850+0000 7faa41827700 1 -- 172.21.15.3:0/1518305232 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faa3c0d0e40 con 0x7faa2806ac90 2024-05-08T21:23:09.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.861+0000 7faa23fff700 1 -- 172.21.15.3:0/1518305232 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+369888 (secure 0 0 0) 0x7faa2c0119f0 con 0x7faa2806ac90 2024-05-08T21:23:09.867 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:09.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.865+0000 7faa41827700 1 -- 172.21.15.3:0/1518305232 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faa2806ac90 msgr2=0x7faa2806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:09.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.865+0000 7faa41827700 1 --2- 172.21.15.3:0/1518305232 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faa2806ac90 0x7faa2806d150 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7faa2c009ba0 tx=0x7faa2c008040 comp rx=0 tx=0).stop 2024-05-08T21:23:09.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.865+0000 7faa41827700 1 -- 172.21.15.3:0/1518305232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa3c150ca0 msgr2=0x7faa3c0b15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:09.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.865+0000 7faa41827700 1 --2- 172.21.15.3:0/1518305232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa3c150ca0 0x7faa3c0b15c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7faa3000f7f0 tx=0x7faa3000fbb0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.865+0000 7faa41827700 1 -- 172.21.15.3:0/1518305232 shutdown_connections 2024-05-08T21:23:09.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.865+0000 7faa41827700 1 --2- 172.21.15.3:0/1518305232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa3c1508c0 0x7faa3c0c4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.865+0000 7faa41827700 1 --2- 172.21.15.3:0/1518305232 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faa2806ac90 0x7faa2806d150 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.865+0000 7faa41827700 1 --2- 172.21.15.3:0/1518305232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa3c0b1b00 0x7faa3c1f21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.865+0000 7faa41827700 1 --2- 172.21.15.3:0/1518305232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa3c150ca0 0x7faa3c0b15c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:09.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.865+0000 7faa41827700 1 -- 172.21.15.3:0/1518305232 >> 172.21.15.3:0/1518305232 conn(0x7faa3c0d0320 msgr2=0x7faa3c15bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:09.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.865+0000 7faa41827700 1 -- 172.21.15.3:0/1518305232 shutdown_connections 2024-05-08T21:23:09.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:09.865+0000 7faa41827700 1 -- 172.21.15.3:0/1518305232 wait complete. 2024-05-08T21:23:09.871 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-08T21:23:09.880 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-05-08T21:23:08.549080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6881824,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":117,"num_read_kb":100,"num_write":255,"num_write_kb":8602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":702,"ondisk_log_size":702,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233224,"kb_used_data":17056,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754741496,"statfs":{"total":773094113280,"available":772855291904,"internally_reserved":0,"allocated":17465344,"data_stored":15288048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17437,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":42,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001561"},"pg_stats":[{"pgid":"5.3f","version":"34'8","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638017+0000","last_change":"2024-05-08T21:22:50.209349+0000","last_active":"2024-05-08T21:23:04.638017+0000","last_peered":"2024-05-08T21:23:04.638017+0000","last_clean":"2024-05-08T21:23:04.638017+0000","last_became_active":"2024-05-08T21:22:50.209092+0000","last_became_peered":"2024-05-08T21:22:50.209092+0000","last_unstale":"2024-05-08T21:23:04.638017+0000","last_undegraded":"2024-05-08T21:23:04.638017+0000","last_fullsized":"2024-05-08T21:23:04.638017+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:52:47.510546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"34'4","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635118+0000","last_change":"2024-05-08T21:22:50.205276+0000","last_active":"2024-05-08T21:23:04.635118+0000","last_peered":"2024-05-08T21:23:04.635118+0000","last_clean":"2024-05-08T21:23:04.635118+0000","last_became_active":"2024-05-08T21:22:50.205028+0000","last_became_peered":"2024-05-08T21:22:50.205028+0000","last_unstale":"2024-05-08T21:23:04.635118+0000","last_undegraded":"2024-05-08T21:23:04.635118+0000","last_fullsized":"2024-05-08T21:23:04.635118+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:24:34.674332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"34'7","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638887+0000","last_change":"2024-05-08T21:22:50.205185+0000","last_active":"2024-05-08T21:23:04.638887+0000","last_peered":"2024-05-08T21:23:04.638887+0000","last_clean":"2024-05-08T21:23:04.638887+0000","last_became_active":"2024-05-08T21:22:50.204826+0000","last_became_peered":"2024-05-08T21:22:50.204826+0000","last_unstale":"2024-05-08T21:23:04.638887+0000","last_undegraded":"2024-05-08T21:23:04.638887+0000","last_fullsized":"2024-05-08T21:23:04.638887+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:05:25.952745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"34'8","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.640890+0000","last_change":"2024-05-08T21:22:50.210536+0000","last_active":"2024-05-08T21:23:04.640890+0000","last_peered":"2024-05-08T21:23:04.640890+0000","last_clean":"2024-05-08T21:23:04.640890+0000","last_became_active":"2024-05-08T21:22:50.210248+0000","last_became_peered":"2024-05-08T21:22:50.210248+0000","last_unstale":"2024-05-08T21:23:04.640890+0000","last_undegraded":"2024-05-08T21:23:04.640890+0000","last_fullsized":"2024-05-08T21:23:04.640890+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:27:40.074356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"34'4","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634253+0000","last_change":"2024-05-08T21:22:50.208946+0000","last_active":"2024-05-08T21:23:04.634253+0000","last_peered":"2024-05-08T21:23:04.634253+0000","last_clean":"2024-05-08T21:23:04.634253+0000","last_became_active":"2024-05-08T21:22:50.208626+0000","last_became_peered":"2024-05-08T21:22:50.208626+0000","last_unstale":"2024-05-08T21:23:04.634253+0000","last_undegraded":"2024-05-08T21:23:04.634253+0000","last_fullsized":"2024-05-08T21:23:04.634253+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:09.165560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"34'6","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634170+0000","last_change":"2024-05-08T21:22:50.207734+0000","last_active":"2024-05-08T21:23:04.634170+0000","last_peered":"2024-05-08T21:23:04.634170+0000","last_clean":"2024-05-08T21:23:04.634170+0000","last_became_active":"2024-05-08T21:22:50.207392+0000","last_became_peered":"2024-05-08T21:22:50.207392+0000","last_unstale":"2024-05-08T21:23:04.634170+0000","last_undegraded":"2024-05-08T21:23:04.634170+0000","last_fullsized":"2024-05-08T21:23:04.634170+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:23:48.163955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"34'5","reported_seq":21,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635074+0000","last_change":"2024-05-08T21:22:50.210352+0000","last_active":"2024-05-08T21:23:04.635074+0000","last_peered":"2024-05-08T21:23:04.635074+0000","last_clean":"2024-05-08T21:23:04.635074+0000","last_became_active":"2024-05-08T21:22:50.210060+0000","last_became_peered":"2024-05-08T21:22:50.210060+0000","last_unstale":"2024-05-08T21:23:04.635074+0000","last_undegraded":"2024-05-08T21:23:04.635074+0000","last_fullsized":"2024-05-08T21:23:04.635074+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:58:46.704338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"34'9","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634963+0000","last_change":"2024-05-08T21:22:50.203181+0000","last_active":"2024-05-08T21:23:04.634963+0000","last_peered":"2024-05-08T21:23:04.634963+0000","last_clean":"2024-05-08T21:23:04.634963+0000","last_became_active":"2024-05-08T21:22:50.202914+0000","last_became_peered":"2024-05-08T21:22:50.202914+0000","last_unstale":"2024-05-08T21:23:04.634963+0000","last_undegraded":"2024-05-08T21:23:04.634963+0000","last_fullsized":"2024-05-08T21:23:04.634963+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:24:39.295215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"34'11","reported_seq":28,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.633163+0000","last_change":"2024-05-08T21:22:50.207050+0000","last_active":"2024-05-08T21:23:04.633163+0000","last_peered":"2024-05-08T21:23:04.633163+0000","last_clean":"2024-05-08T21:23:04.633163+0000","last_became_active":"2024-05-08T21:22:50.206737+0000","last_became_peered":"2024-05-08T21:22:50.206737+0000","last_unstale":"2024-05-08T21:23:04.633163+0000","last_undegraded":"2024-05-08T21:23:04.633163+0000","last_fullsized":"2024-05-08T21:23:04.633163+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:52:47.510546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"34'15","reported_seq":31,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641182+0000","last_change":"2024-05-08T21:22:50.212260+0000","last_active":"2024-05-08T21:23:04.641182+0000","last_peered":"2024-05-08T21:23:04.641182+0000","last_clean":"2024-05-08T21:23:04.641182+0000","last_became_active":"2024-05-08T21:22:50.211928+0000","last_became_peered":"2024-05-08T21:22:50.211928+0000","last_unstale":"2024-05-08T21:23:04.641182+0000","last_undegraded":"2024-05-08T21:23:04.641182+0000","last_fullsized":"2024-05-08T21:23:04.641182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:43:00.051225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"34'6","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635427+0000","last_change":"2024-05-08T21:22:50.205779+0000","last_active":"2024-05-08T21:23:04.635427+0000","last_peered":"2024-05-08T21:23:04.635427+0000","last_clean":"2024-05-08T21:23:04.635427+0000","last_became_active":"2024-05-08T21:22:50.205533+0000","last_became_peered":"2024-05-08T21:22:50.205533+0000","last_unstale":"2024-05-08T21:23:04.635427+0000","last_undegraded":"2024-05-08T21:23:04.635427+0000","last_fullsized":"2024-05-08T21:23:04.635427+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:24:34.674332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"34'7","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638783+0000","last_change":"2024-05-08T21:22:50.207359+0000","last_active":"2024-05-08T21:23:04.638783+0000","last_peered":"2024-05-08T21:23:04.638783+0000","last_clean":"2024-05-08T21:23:04.638783+0000","last_became_active":"2024-05-08T21:22:50.207045+0000","last_became_peered":"2024-05-08T21:22:50.207045+0000","last_unstale":"2024-05-08T21:23:04.638783+0000","last_undegraded":"2024-05-08T21:23:04.638783+0000","last_fullsized":"2024-05-08T21:23:04.638783+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:24:34.674332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"34'10","reported_seq":26,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634509+0000","last_change":"2024-05-08T21:22:50.207671+0000","last_active":"2024-05-08T21:23:04.634509+0000","last_peered":"2024-05-08T21:23:04.634509+0000","last_clean":"2024-05-08T21:23:04.634509+0000","last_became_active":"2024-05-08T21:22:50.207119+0000","last_became_peered":"2024-05-08T21:22:50.207119+0000","last_unstale":"2024-05-08T21:23:04.634509+0000","last_undegraded":"2024-05-08T21:23:04.634509+0000","last_fullsized":"2024-05-08T21:23:04.634509+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:59:37.805505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"34'8","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634288+0000","last_change":"2024-05-08T21:22:50.216066+0000","last_active":"2024-05-08T21:23:04.634288+0000","last_peered":"2024-05-08T21:23:04.634288+0000","last_clean":"2024-05-08T21:23:04.634288+0000","last_became_active":"2024-05-08T21:22:50.215796+0000","last_became_peered":"2024-05-08T21:22:50.215796+0000","last_unstale":"2024-05-08T21:23:04.634288+0000","last_undegraded":"2024-05-08T21:23:04.634288+0000","last_fullsized":"2024-05-08T21:23:04.634288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:24:39.295215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"34'7","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638525+0000","last_change":"2024-05-08T21:22:50.211332+0000","last_active":"2024-05-08T21:23:04.638525+0000","last_peered":"2024-05-08T21:23:04.638525+0000","last_clean":"2024-05-08T21:23:04.638525+0000","last_became_active":"2024-05-08T21:22:50.210865+0000","last_became_peered":"2024-05-08T21:22:50.210865+0000","last_unstale":"2024-05-08T21:23:04.638525+0000","last_undegraded":"2024-05-08T21:23:04.638525+0000","last_fullsized":"2024-05-08T21:23:04.638525+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:24:39.295215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"34'9","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635045+0000","last_change":"2024-05-08T21:22:50.215021+0000","last_active":"2024-05-08T21:23:04.635045+0000","last_peered":"2024-05-08T21:23:04.635045+0000","last_clean":"2024-05-08T21:23:04.635045+0000","last_became_active":"2024-05-08T21:22:50.214821+0000","last_became_peered":"2024-05-08T21:22:50.214821+0000","last_unstale":"2024-05-08T21:23:04.635045+0000","last_undegraded":"2024-05-08T21:23:04.635045+0000","last_fullsized":"2024-05-08T21:23:04.635045+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:14.396616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"34'9","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635686+0000","last_change":"2024-05-08T21:22:50.205302+0000","last_active":"2024-05-08T21:23:04.635686+0000","last_peered":"2024-05-08T21:23:04.635686+0000","last_clean":"2024-05-08T21:23:04.635686+0000","last_became_active":"2024-05-08T21:22:50.204962+0000","last_became_peered":"2024-05-08T21:22:50.204962+0000","last_unstale":"2024-05-08T21:23:04.635686+0000","last_undegraded":"2024-05-08T21:23:04.635686+0000","last_fullsized":"2024-05-08T21:23:04.635686+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:09:24.655227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097152,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"34'10","reported_seq":26,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635013+0000","last_change":"2024-05-08T21:22:50.208448+0000","last_active":"2024-05-08T21:23:04.635013+0000","last_peered":"2024-05-08T21:23:04.635013+0000","last_clean":"2024-05-08T21:23:04.635013+0000","last_became_active":"2024-05-08T21:22:50.208210+0000","last_became_peered":"2024-05-08T21:22:50.208210+0000","last_unstale":"2024-05-08T21:23:04.635013+0000","last_undegraded":"2024-05-08T21:23:04.635013+0000","last_fullsized":"2024-05-08T21:23:04.635013+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:38:13.012930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"34'7","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634646+0000","last_change":"2024-05-08T21:22:50.206974+0000","last_active":"2024-05-08T21:23:04.634646+0000","last_peered":"2024-05-08T21:23:04.634646+0000","last_clean":"2024-05-08T21:23:04.634646+0000","last_became_active":"2024-05-08T21:22:50.206688+0000","last_became_peered":"2024-05-08T21:22:50.206688+0000","last_unstale":"2024-05-08T21:23:04.634646+0000","last_undegraded":"2024-05-08T21:23:04.634646+0000","last_fullsized":"2024-05-08T21:23:04.634646+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:41:28.819967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"34'8","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635292+0000","last_change":"2024-05-08T21:22:50.207398+0000","last_active":"2024-05-08T21:23:04.635292+0000","last_peered":"2024-05-08T21:23:04.635292+0000","last_clean":"2024-05-08T21:23:04.635292+0000","last_became_active":"2024-05-08T21:22:50.207040+0000","last_became_peered":"2024-05-08T21:22:50.207040+0000","last_unstale":"2024-05-08T21:23:04.635292+0000","last_undegraded":"2024-05-08T21:23:04.635292+0000","last_fullsized":"2024-05-08T21:23:04.635292+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:28.358029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"34'11","reported_seq":28,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632605+0000","last_change":"2024-05-08T21:22:50.199407+0000","last_active":"2024-05-08T21:23:04.632605+0000","last_peered":"2024-05-08T21:23:04.632605+0000","last_clean":"2024-05-08T21:23:04.632605+0000","last_became_active":"2024-05-08T21:22:50.199133+0000","last_became_peered":"2024-05-08T21:22:50.199133+0000","last_unstale":"2024-05-08T21:23:04.632605+0000","last_undegraded":"2024-05-08T21:23:04.632605+0000","last_fullsized":"2024-05-08T21:23:04.632605+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:40.200175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"34'9","reported_seq":26,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632780+0000","last_change":"2024-05-08T21:22:50.207042+0000","last_active":"2024-05-08T21:23:04.632780+0000","last_peered":"2024-05-08T21:23:04.632780+0000","last_clean":"2024-05-08T21:23:04.632780+0000","last_became_active":"2024-05-08T21:22:50.206554+0000","last_became_peered":"2024-05-08T21:22:50.206554+0000","last_unstale":"2024-05-08T21:23:04.632780+0000","last_undegraded":"2024-05-08T21:23:04.632780+0000","last_fullsized":"2024-05-08T21:23:04.632780+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:15:17.946198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"34'8","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.636567+0000","last_change":"2024-05-08T21:22:50.200284+0000","last_active":"2024-05-08T21:23:04.636567+0000","last_peered":"2024-05-08T21:23:04.636567+0000","last_clean":"2024-05-08T21:23:04.636567+0000","last_became_active":"2024-05-08T21:22:50.200004+0000","last_became_peered":"2024-05-08T21:22:50.200004+0000","last_unstale":"2024-05-08T21:23:04.636567+0000","last_undegraded":"2024-05-08T21:23:04.636567+0000","last_fullsized":"2024-05-08T21:23:04.636567+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:28.358029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"34'6","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638390+0000","last_change":"2024-05-08T21:22:50.208138+0000","last_active":"2024-05-08T21:23:04.638390+0000","last_peered":"2024-05-08T21:23:04.638390+0000","last_clean":"2024-05-08T21:23:04.638390+0000","last_became_active":"2024-05-08T21:22:50.207813+0000","last_became_peered":"2024-05-08T21:22:50.207813+0000","last_unstale":"2024-05-08T21:23:04.638390+0000","last_undegraded":"2024-05-08T21:23:04.638390+0000","last_fullsized":"2024-05-08T21:23:04.638390+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:07:17.470994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"34'2","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641204+0000","last_change":"2024-05-08T21:22:50.202937+0000","last_active":"2024-05-08T21:23:04.641204+0000","last_peered":"2024-05-08T21:23:04.641204+0000","last_clean":"2024-05-08T21:23:04.641204+0000","last_became_active":"2024-05-08T21:22:50.202633+0000","last_became_peered":"2024-05-08T21:22:50.202633+0000","last_unstale":"2024-05-08T21:23:04.641204+0000","last_undegraded":"2024-05-08T21:23:04.641204+0000","last_fullsized":"2024-05-08T21:23:04.641204+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:48:46.138832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"34'15","reported_seq":31,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635221+0000","last_change":"2024-05-08T21:22:50.204581+0000","last_active":"2024-05-08T21:23:04.635221+0000","last_peered":"2024-05-08T21:23:04.635221+0000","last_clean":"2024-05-08T21:23:04.635221+0000","last_became_active":"2024-05-08T21:22:50.204228+0000","last_became_peered":"2024-05-08T21:22:50.204228+0000","last_unstale":"2024-05-08T21:23:04.635221+0000","last_undegraded":"2024-05-08T21:23:04.635221+0000","last_fullsized":"2024-05-08T21:23:04.635221+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:43:00.051225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"34'6","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641363+0000","last_change":"2024-05-08T21:22:50.210823+0000","last_active":"2024-05-08T21:23:04.641363+0000","last_peered":"2024-05-08T21:23:04.641363+0000","last_clean":"2024-05-08T21:23:04.641363+0000","last_became_active":"2024-05-08T21:22:50.210517+0000","last_became_peered":"2024-05-08T21:22:50.210517+0000","last_unstale":"2024-05-08T21:23:04.641363+0000","last_undegraded":"2024-05-08T21:23:04.641363+0000","last_fullsized":"2024-05-08T21:23:04.641363+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:09:24.655227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"34'4","reported_seq":21,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638802+0000","last_change":"2024-05-08T21:22:50.201860+0000","last_active":"2024-05-08T21:23:04.638802+0000","last_peered":"2024-05-08T21:23:04.638802+0000","last_clean":"2024-05-08T21:23:04.638802+0000","last_became_active":"2024-05-08T21:22:50.201204+0000","last_became_peered":"2024-05-08T21:22:50.201204+0000","last_unstale":"2024-05-08T21:23:04.638802+0000","last_undegraded":"2024-05-08T21:23:04.638802+0000","last_fullsized":"2024-05-08T21:23:04.638802+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:23:48.163955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"34'7","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632564+0000","last_change":"2024-05-08T21:22:50.206666+0000","last_active":"2024-05-08T21:23:04.632564+0000","last_peered":"2024-05-08T21:23:04.632564+0000","last_clean":"2024-05-08T21:23:04.632564+0000","last_became_active":"2024-05-08T21:22:50.206283+0000","last_became_peered":"2024-05-08T21:22:50.206283+0000","last_unstale":"2024-05-08T21:23:04.632564+0000","last_undegraded":"2024-05-08T21:23:04.632564+0000","last_fullsized":"2024-05-08T21:23:04.632564+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:53:39.386458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"34'8","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635149+0000","last_change":"2024-05-08T21:22:50.205319+0000","last_active":"2024-05-08T21:23:04.635149+0000","last_peered":"2024-05-08T21:23:04.635149+0000","last_clean":"2024-05-08T21:23:04.635149+0000","last_became_active":"2024-05-08T21:22:50.205014+0000","last_became_peered":"2024-05-08T21:22:50.205014+0000","last_unstale":"2024-05-08T21:23:04.635149+0000","last_undegraded":"2024-05-08T21:23:04.635149+0000","last_fullsized":"2024-05-08T21:23:04.635149+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:10:48.376692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"34'12","reported_seq":28,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637536+0000","last_change":"2024-05-08T21:22:50.210271+0000","last_active":"2024-05-08T21:23:04.637536+0000","last_peered":"2024-05-08T21:23:04.637536+0000","last_clean":"2024-05-08T21:23:04.637536+0000","last_became_active":"2024-05-08T21:22:50.209998+0000","last_became_peered":"2024-05-08T21:22:50.209998+0000","last_unstale":"2024-05-08T21:23:04.637536+0000","last_undegraded":"2024-05-08T21:23:04.637536+0000","last_fullsized":"2024-05-08T21:23:04.637536+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:09.165560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"34'9","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635308+0000","last_change":"2024-05-08T21:22:50.211741+0000","last_active":"2024-05-08T21:23:04.635308+0000","last_peered":"2024-05-08T21:23:04.635308+0000","last_clean":"2024-05-08T21:23:04.635308+0000","last_became_active":"2024-05-08T21:22:50.211430+0000","last_became_peered":"2024-05-08T21:22:50.211430+0000","last_unstale":"2024-05-08T21:23:04.635308+0000","last_undegraded":"2024-05-08T21:23:04.635308+0000","last_fullsized":"2024-05-08T21:23:04.635308+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:23:48.163955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493199+0000","last_change":"2024-05-08T21:23:05.490190+0000","last_active":"2024-05-08T21:23:05.493199+0000","last_peered":"2024-05-08T21:23:05.493199+0000","last_clean":"2024-05-08T21:23:05.493199+0000","last_became_active":"2024-05-08T21:23:05.489780+0000","last_became_peered":"2024-05-08T21:23:05.489780+0000","last_unstale":"2024-05-08T21:23:05.493199+0000","last_undegraded":"2024-05-08T21:23:05.493199+0000","last_fullsized":"2024-05-08T21:23:05.493199+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:13:01.896572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"34'5","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637599+0000","last_change":"2024-05-08T21:22:50.201116+0000","last_active":"2024-05-08T21:23:04.637599+0000","last_peered":"2024-05-08T21:23:04.637599+0000","last_clean":"2024-05-08T21:23:04.637599+0000","last_became_active":"2024-05-08T21:22:50.200817+0000","last_became_peered":"2024-05-08T21:22:50.200817+0000","last_unstale":"2024-05-08T21:23:04.637599+0000","last_undegraded":"2024-05-08T21:23:04.637599+0000","last_fullsized":"2024-05-08T21:23:04.637599+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:09.165560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"37'8","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638089+0000","last_change":"2024-05-08T21:22:49.740529+0000","last_active":"2024-05-08T21:23:04.638089+0000","last_peered":"2024-05-08T21:23:04.638089+0000","last_clean":"2024-05-08T21:23:04.638089+0000","last_became_active":"2024-05-08T21:22:49.740247+0000","last_became_peered":"2024-05-08T21:22:49.740247+0000","last_unstale":"2024-05-08T21:23:04.638089+0000","last_undegraded":"2024-05-08T21:23:04.638089+0000","last_fullsized":"2024-05-08T21:23:04.638089+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:53:44.252529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.492276+0000","last_change":"2024-05-08T21:23:05.490147+0000","last_active":"2024-05-08T21:23:05.492276+0000","last_peered":"2024-05-08T21:23:05.492276+0000","last_clean":"2024-05-08T21:23:05.492276+0000","last_became_active":"2024-05-08T21:23:05.489698+0000","last_became_peered":"2024-05-08T21:23:05.489698+0000","last_unstale":"2024-05-08T21:23:05.492276+0000","last_undegraded":"2024-05-08T21:23:05.492276+0000","last_fullsized":"2024-05-08T21:23:05.492276+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:52:01.457943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"34'6","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638029+0000","last_change":"2024-05-08T21:22:50.208077+0000","last_active":"2024-05-08T21:23:04.638029+0000","last_peered":"2024-05-08T21:23:04.638029+0000","last_clean":"2024-05-08T21:23:04.638029+0000","last_became_active":"2024-05-08T21:22:50.207729+0000","last_became_peered":"2024-05-08T21:22:50.207729+0000","last_unstale":"2024-05-08T21:23:04.638029+0000","last_undegraded":"2024-05-08T21:23:04.638029+0000","last_fullsized":"2024-05-08T21:23:04.638029+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:40.200175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637700+0000","last_change":"2024-05-08T21:22:49.743083+0000","last_active":"2024-05-08T21:23:04.637700+0000","last_peered":"2024-05-08T21:23:04.637700+0000","last_clean":"2024-05-08T21:23:04.637700+0000","last_became_active":"2024-05-08T21:22:49.742790+0000","last_became_peered":"2024-05-08T21:22:49.742790+0000","last_unstale":"2024-05-08T21:23:04.637700+0000","last_undegraded":"2024-05-08T21:23:04.637700+0000","last_fullsized":"2024-05-08T21:23:04.637700+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:06:42.743465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.519256+0000","last_change":"2024-05-08T21:23:05.517988+0000","last_active":"2024-05-08T21:23:05.519256+0000","last_peered":"2024-05-08T21:23:05.519256+0000","last_clean":"2024-05-08T21:23:05.519256+0000","last_became_active":"2024-05-08T21:23:05.517596+0000","last_became_peered":"2024-05-08T21:23:05.517596+0000","last_unstale":"2024-05-08T21:23:05.519256+0000","last_undegraded":"2024-05-08T21:23:05.519256+0000","last_fullsized":"2024-05-08T21:23:05.519256+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:28:47.005653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"34'9","reported_seq":26,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632761+0000","last_change":"2024-05-08T21:22:50.207141+0000","last_active":"2024-05-08T21:23:04.632761+0000","last_peered":"2024-05-08T21:23:04.632761+0000","last_clean":"2024-05-08T21:23:04.632761+0000","last_became_active":"2024-05-08T21:22:50.206838+0000","last_became_peered":"2024-05-08T21:22:50.206838+0000","last_unstale":"2024-05-08T21:23:04.632761+0000","last_undegraded":"2024-05-08T21:23:04.632761+0000","last_fullsized":"2024-05-08T21:23:04.632761+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:16:58.048181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"36'3","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634757+0000","last_change":"2024-05-08T21:22:49.747636+0000","last_active":"2024-05-08T21:23:04.634757+0000","last_peered":"2024-05-08T21:23:04.634757+0000","last_clean":"2024-05-08T21:23:04.634757+0000","last_became_active":"2024-05-08T21:22:49.747345+0000","last_became_peered":"2024-05-08T21:22:49.747345+0000","last_unstale":"2024-05-08T21:23:04.634757+0000","last_undegraded":"2024-05-08T21:23:04.634757+0000","last_fullsized":"2024-05-08T21:23:04.634757+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:29:03.389411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.521132+0000","last_change":"2024-05-08T21:23:05.518010+0000","last_active":"2024-05-08T21:23:05.521132+0000","last_peered":"2024-05-08T21:23:05.521132+0000","last_clean":"2024-05-08T21:23:05.521132+0000","last_became_active":"2024-05-08T21:23:05.517594+0000","last_became_peered":"2024-05-08T21:23:05.517594+0000","last_unstale":"2024-05-08T21:23:05.521132+0000","last_undegraded":"2024-05-08T21:23:05.521132+0000","last_fullsized":"2024-05-08T21:23:05.521132+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:18:53.566436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"34'15","reported_seq":31,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641448+0000","last_change":"2024-05-08T21:22:50.209316+0000","last_active":"2024-05-08T21:23:04.641448+0000","last_peered":"2024-05-08T21:23:04.641448+0000","last_clean":"2024-05-08T21:23:04.641448+0000","last_became_active":"2024-05-08T21:22:50.209044+0000","last_became_peered":"2024-05-08T21:22:50.209044+0000","last_unstale":"2024-05-08T21:23:04.641448+0000","last_undegraded":"2024-05-08T21:23:04.641448+0000","last_fullsized":"2024-05-08T21:23:04.641448+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:25:52.293163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"36'3","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635395+0000","last_change":"2024-05-08T21:22:49.744152+0000","last_active":"2024-05-08T21:23:04.635395+0000","last_peered":"2024-05-08T21:23:04.635395+0000","last_clean":"2024-05-08T21:23:04.635395+0000","last_became_active":"2024-05-08T21:22:49.743867+0000","last_became_peered":"2024-05-08T21:22:49.743867+0000","last_unstale":"2024-05-08T21:23:04.635395+0000","last_undegraded":"2024-05-08T21:23:04.635395+0000","last_fullsized":"2024-05-08T21:23:04.635395+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:55:00.385106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.494808+0000","last_change":"2024-05-08T21:23:05.490761+0000","last_active":"2024-05-08T21:23:05.494808+0000","last_peered":"2024-05-08T21:23:05.494808+0000","last_clean":"2024-05-08T21:23:05.494808+0000","last_became_active":"2024-05-08T21:23:05.490093+0000","last_became_peered":"2024-05-08T21:23:05.490093+0000","last_unstale":"2024-05-08T21:23:05.494808+0000","last_undegraded":"2024-05-08T21:23:05.494808+0000","last_fullsized":"2024-05-08T21:23:05.494808+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:12:43.572456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"34'7","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635214+0000","last_change":"2024-05-08T21:22:50.204033+0000","last_active":"2024-05-08T21:23:04.635214+0000","last_peered":"2024-05-08T21:23:04.635214+0000","last_clean":"2024-05-08T21:23:04.635214+0000","last_became_active":"2024-05-08T21:22:50.203766+0000","last_became_peered":"2024-05-08T21:22:50.203766+0000","last_unstale":"2024-05-08T21:23:04.635214+0000","last_undegraded":"2024-05-08T21:23:04.635214+0000","last_fullsized":"2024-05-08T21:23:04.635214+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:48:46.138832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":19,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632687+0000","last_change":"2024-05-08T21:22:49.740399+0000","last_active":"2024-05-08T21:23:04.632687+0000","last_peered":"2024-05-08T21:23:04.632687+0000","last_clean":"2024-05-08T21:23:04.632687+0000","last_became_active":"2024-05-08T21:22:49.740125+0000","last_became_peered":"2024-05-08T21:22:49.740125+0000","last_unstale":"2024-05-08T21:23:04.632687+0000","last_undegraded":"2024-05-08T21:23:04.632687+0000","last_fullsized":"2024-05-08T21:23:04.632687+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:01:42.646117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493235+0000","last_change":"2024-05-08T21:23:05.489369+0000","last_active":"2024-05-08T21:23:05.493235+0000","last_peered":"2024-05-08T21:23:05.493235+0000","last_clean":"2024-05-08T21:23:05.493235+0000","last_became_active":"2024-05-08T21:23:05.489036+0000","last_became_peered":"2024-05-08T21:23:05.489036+0000","last_unstale":"2024-05-08T21:23:05.493235+0000","last_undegraded":"2024-05-08T21:23:05.493235+0000","last_fullsized":"2024-05-08T21:23:05.493235+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:28:20.539506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"34'8","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638573+0000","last_change":"2024-05-08T21:22:50.205784+0000","last_active":"2024-05-08T21:23:04.638573+0000","last_peered":"2024-05-08T21:23:04.638573+0000","last_clean":"2024-05-08T21:23:04.638573+0000","last_became_active":"2024-05-08T21:22:50.205525+0000","last_became_peered":"2024-05-08T21:22:50.205525+0000","last_unstale":"2024-05-08T21:23:04.638573+0000","last_undegraded":"2024-05-08T21:23:04.638573+0000","last_fullsized":"2024-05-08T21:23:04.638573+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:23.732223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":19,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632771+0000","last_change":"2024-05-08T21:22:49.739604+0000","last_active":"2024-05-08T21:23:04.632771+0000","last_peered":"2024-05-08T21:23:04.632771+0000","last_clean":"2024-05-08T21:23:04.632771+0000","last_became_active":"2024-05-08T21:22:49.739343+0000","last_became_peered":"2024-05-08T21:22:49.739343+0000","last_unstale":"2024-05-08T21:23:04.632771+0000","last_undegraded":"2024-05-08T21:23:04.632771+0000","last_fullsized":"2024-05-08T21:23:04.632771+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:55:43.121728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493154+0000","last_change":"2024-05-08T21:23:05.490982+0000","last_active":"2024-05-08T21:23:05.493154+0000","last_peered":"2024-05-08T21:23:05.493154+0000","last_clean":"2024-05-08T21:23:05.493154+0000","last_became_active":"2024-05-08T21:23:05.490662+0000","last_became_peered":"2024-05-08T21:23:05.490662+0000","last_unstale":"2024-05-08T21:23:05.493154+0000","last_undegraded":"2024-05-08T21:23:05.493154+0000","last_fullsized":"2024-05-08T21:23:05.493154+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:11:23.240353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"34'8","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632420+0000","last_change":"2024-05-08T21:22:50.206820+0000","last_active":"2024-05-08T21:23:04.632420+0000","last_peered":"2024-05-08T21:23:04.632420+0000","last_clean":"2024-05-08T21:23:04.632420+0000","last_became_active":"2024-05-08T21:22:50.206477+0000","last_became_peered":"2024-05-08T21:22:50.206477+0000","last_unstale":"2024-05-08T21:23:04.632420+0000","last_undegraded":"2024-05-08T21:23:04.632420+0000","last_fullsized":"2024-05-08T21:23:04.632420+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:28.358029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634004+0000","last_change":"2024-05-08T21:22:49.737447+0000","last_active":"2024-05-08T21:23:04.634004+0000","last_peered":"2024-05-08T21:23:04.634004+0000","last_clean":"2024-05-08T21:23:04.634004+0000","last_became_active":"2024-05-08T21:22:49.736975+0000","last_became_peered":"2024-05-08T21:22:49.736975+0000","last_unstale":"2024-05-08T21:23:04.634004+0000","last_undegraded":"2024-05-08T21:23:04.634004+0000","last_fullsized":"2024-05-08T21:23:04.634004+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:39.192702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.494913+0000","last_change":"2024-05-08T21:23:05.489998+0000","last_active":"2024-05-08T21:23:05.494913+0000","last_peered":"2024-05-08T21:23:05.494913+0000","last_clean":"2024-05-08T21:23:05.494913+0000","last_became_active":"2024-05-08T21:23:05.489630+0000","last_became_peered":"2024-05-08T21:23:05.489630+0000","last_unstale":"2024-05-08T21:23:05.494913+0000","last_undegraded":"2024-05-08T21:23:05.494913+0000","last_fullsized":"2024-05-08T21:23:05.494913+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:20:48.767679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"34'10","reported_seq":26,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641137+0000","last_change":"2024-05-08T21:22:50.205263+0000","last_active":"2024-05-08T21:23:04.641137+0000","last_peered":"2024-05-08T21:23:04.641137+0000","last_clean":"2024-05-08T21:23:04.641137+0000","last_became_active":"2024-05-08T21:22:50.204963+0000","last_became_peered":"2024-05-08T21:22:50.204963+0000","last_unstale":"2024-05-08T21:23:04.641137+0000","last_undegraded":"2024-05-08T21:23:04.641137+0000","last_fullsized":"2024-05-08T21:23:04.641137+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:44:18.055232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632382+0000","last_change":"2024-05-08T21:22:49.733981+0000","last_active":"2024-05-08T21:23:04.632382+0000","last_peered":"2024-05-08T21:23:04.632382+0000","last_clean":"2024-05-08T21:23:04.632382+0000","last_became_active":"2024-05-08T21:22:49.733714+0000","last_became_peered":"2024-05-08T21:22:49.733714+0000","last_unstale":"2024-05-08T21:23:04.632382+0000","last_undegraded":"2024-05-08T21:23:04.632382+0000","last_fullsized":"2024-05-08T21:23:04.632382+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:47.931452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.519462+0000","last_change":"2024-05-08T21:23:05.518321+0000","last_active":"2024-05-08T21:23:05.519462+0000","last_peered":"2024-05-08T21:23:05.519462+0000","last_clean":"2024-05-08T21:23:05.519462+0000","last_became_active":"2024-05-08T21:23:05.518059+0000","last_became_peered":"2024-05-08T21:23:05.518059+0000","last_unstale":"2024-05-08T21:23:05.519462+0000","last_undegraded":"2024-05-08T21:23:05.519462+0000","last_fullsized":"2024-05-08T21:23:05.519462+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:52:59.903852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"34'10","reported_seq":27,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.639115+0000","last_change":"2024-05-08T21:22:50.209646+0000","last_active":"2024-05-08T21:23:04.639115+0000","last_peered":"2024-05-08T21:23:04.639115+0000","last_clean":"2024-05-08T21:23:04.639115+0000","last_became_active":"2024-05-08T21:22:50.209335+0000","last_became_peered":"2024-05-08T21:22:50.209335+0000","last_unstale":"2024-05-08T21:23:04.639115+0000","last_undegraded":"2024-05-08T21:23:04.639115+0000","last_fullsized":"2024-05-08T21:23:04.639115+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:21:31.919852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.636375+0000","last_change":"2024-05-08T21:22:49.738039+0000","last_active":"2024-05-08T21:23:04.636375+0000","last_peered":"2024-05-08T21:23:04.636375+0000","last_clean":"2024-05-08T21:23:04.636375+0000","last_became_active":"2024-05-08T21:22:49.737785+0000","last_became_peered":"2024-05-08T21:22:49.737785+0000","last_unstale":"2024-05-08T21:23:04.636375+0000","last_undegraded":"2024-05-08T21:23:04.636375+0000","last_fullsized":"2024-05-08T21:23:04.636375+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:27.350557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.492900+0000","last_change":"2024-05-08T21:23:05.490857+0000","last_active":"2024-05-08T21:23:05.492900+0000","last_peered":"2024-05-08T21:23:05.492900+0000","last_clean":"2024-05-08T21:23:05.492900+0000","last_became_active":"2024-05-08T21:23:05.490603+0000","last_became_peered":"2024-05-08T21:23:05.490603+0000","last_unstale":"2024-05-08T21:23:05.492900+0000","last_undegraded":"2024-05-08T21:23:05.492900+0000","last_fullsized":"2024-05-08T21:23:05.492900+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:05:40.853105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"34'9","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.636726+0000","last_change":"2024-05-08T21:22:50.206695+0000","last_active":"2024-05-08T21:23:04.636726+0000","last_peered":"2024-05-08T21:23:04.636726+0000","last_clean":"2024-05-08T21:23:04.636726+0000","last_became_active":"2024-05-08T21:22:50.206398+0000","last_became_peered":"2024-05-08T21:22:50.206398+0000","last_unstale":"2024-05-08T21:23:04.636726+0000","last_undegraded":"2024-05-08T21:23:04.636726+0000","last_fullsized":"2024-05-08T21:23:04.636726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:07:17.470994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637794+0000","last_change":"2024-05-08T21:22:49.740160+0000","last_active":"2024-05-08T21:23:04.637794+0000","last_peered":"2024-05-08T21:23:04.637794+0000","last_clean":"2024-05-08T21:23:04.637794+0000","last_became_active":"2024-05-08T21:22:49.739818+0000","last_became_peered":"2024-05-08T21:22:49.739818+0000","last_unstale":"2024-05-08T21:23:04.637794+0000","last_undegraded":"2024-05-08T21:23:04.637794+0000","last_fullsized":"2024-05-08T21:23:04.637794+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:39:42.856227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.495393+0000","last_change":"2024-05-08T21:23:05.490240+0000","last_active":"2024-05-08T21:23:05.495393+0000","last_peered":"2024-05-08T21:23:05.495393+0000","last_clean":"2024-05-08T21:23:05.495393+0000","last_became_active":"2024-05-08T21:23:05.489936+0000","last_became_peered":"2024-05-08T21:23:05.489936+0000","last_unstale":"2024-05-08T21:23:05.495393+0000","last_undegraded":"2024-05-08T21:23:05.495393+0000","last_fullsized":"2024-05-08T21:23:05.495393+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:03:21.459659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"34'11","reported_seq":27,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634791+0000","last_change":"2024-05-08T21:22:50.204717+0000","last_active":"2024-05-08T21:23:04.634791+0000","last_peered":"2024-05-08T21:23:04.634791+0000","last_clean":"2024-05-08T21:23:04.634791+0000","last_became_active":"2024-05-08T21:22:50.204404+0000","last_became_peered":"2024-05-08T21:22:50.204404+0000","last_unstale":"2024-05-08T21:23:04.634791+0000","last_undegraded":"2024-05-08T21:23:04.634791+0000","last_fullsized":"2024-05-08T21:23:04.634791+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:27:40.074356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"36'5","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638992+0000","last_change":"2024-05-08T21:22:49.736860+0000","last_active":"2024-05-08T21:23:04.638992+0000","last_peered":"2024-05-08T21:23:04.638992+0000","last_clean":"2024-05-08T21:23:04.638992+0000","last_became_active":"2024-05-08T21:22:49.736497+0000","last_became_peered":"2024-05-08T21:22:49.736497+0000","last_unstale":"2024-05-08T21:23:04.638992+0000","last_undegraded":"2024-05-08T21:23:04.638992+0000","last_fullsized":"2024-05-08T21:23:04.638992+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:15:16.938725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.488443+0000","last_change":"2024-05-08T21:23:05.487624+0000","last_active":"2024-05-08T21:23:05.488443+0000","last_peered":"2024-05-08T21:23:05.488443+0000","last_clean":"2024-05-08T21:23:05.488443+0000","last_became_active":"2024-05-08T21:23:05.487309+0000","last_became_peered":"2024-05-08T21:23:05.487309+0000","last_unstale":"2024-05-08T21:23:05.488443+0000","last_undegraded":"2024-05-08T21:23:05.488443+0000","last_fullsized":"2024-05-08T21:23:05.488443+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:11:55.531943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"34'6","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638796+0000","last_change":"2024-05-08T21:22:50.205233+0000","last_active":"2024-05-08T21:23:04.638796+0000","last_peered":"2024-05-08T21:23:04.638796+0000","last_clean":"2024-05-08T21:23:04.638796+0000","last_became_active":"2024-05-08T21:22:50.204889+0000","last_became_peered":"2024-05-08T21:22:50.204889+0000","last_unstale":"2024-05-08T21:23:04.638796+0000","last_undegraded":"2024-05-08T21:23:04.638796+0000","last_fullsized":"2024-05-08T21:23:04.638796+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:38:13.012930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"36'22","reported_seq":39,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634884+0000","last_change":"2024-05-08T21:22:49.743647+0000","last_active":"2024-05-08T21:23:04.634884+0000","last_peered":"2024-05-08T21:23:04.634884+0000","last_clean":"2024-05-08T21:23:04.634884+0000","last_became_active":"2024-05-08T21:22:49.743348+0000","last_became_peered":"2024-05-08T21:22:49.743348+0000","last_unstale":"2024-05-08T21:23:04.634884+0000","last_undegraded":"2024-05-08T21:23:04.634884+0000","last_fullsized":"2024-05-08T21:23:04.634884+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:06:09.036243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493148+0000","last_change":"2024-05-08T21:23:05.490136+0000","last_active":"2024-05-08T21:23:05.493148+0000","last_peered":"2024-05-08T21:23:05.493148+0000","last_clean":"2024-05-08T21:23:05.493148+0000","last_became_active":"2024-05-08T21:23:05.489860+0000","last_became_peered":"2024-05-08T21:23:05.489860+0000","last_unstale":"2024-05-08T21:23:05.493148+0000","last_undegraded":"2024-05-08T21:23:05.493148+0000","last_fullsized":"2024-05-08T21:23:05.493148+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:23.962061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"34'10","reported_seq":26,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.640738+0000","last_change":"2024-05-08T21:22:50.202647+0000","last_active":"2024-05-08T21:23:04.640738+0000","last_peered":"2024-05-08T21:23:04.640738+0000","last_clean":"2024-05-08T21:23:04.640738+0000","last_became_active":"2024-05-08T21:22:50.202144+0000","last_became_peered":"2024-05-08T21:22:50.202144+0000","last_unstale":"2024-05-08T21:23:04.640738+0000","last_undegraded":"2024-05-08T21:23:04.640738+0000","last_fullsized":"2024-05-08T21:23:04.640738+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:10:48.376692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"37'5","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632726+0000","last_change":"2024-05-08T21:22:49.735484+0000","last_active":"2024-05-08T21:23:04.632726+0000","last_peered":"2024-05-08T21:23:04.632726+0000","last_clean":"2024-05-08T21:23:04.632726+0000","last_became_active":"2024-05-08T21:22:49.735232+0000","last_became_peered":"2024-05-08T21:22:49.735232+0000","last_unstale":"2024-05-08T21:23:04.632726+0000","last_undegraded":"2024-05-08T21:23:04.632726+0000","last_fullsized":"2024-05-08T21:23:04.632726+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:13:03.055943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493638+0000","last_change":"2024-05-08T21:23:05.491656+0000","last_active":"2024-05-08T21:23:05.493638+0000","last_peered":"2024-05-08T21:23:05.493638+0000","last_clean":"2024-05-08T21:23:05.493638+0000","last_became_active":"2024-05-08T21:23:05.491419+0000","last_became_peered":"2024-05-08T21:23:05.491419+0000","last_unstale":"2024-05-08T21:23:05.493638+0000","last_undegraded":"2024-05-08T21:23:05.493638+0000","last_fullsized":"2024-05-08T21:23:05.493638+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:58:36.326421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"34'15","reported_seq":31,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637478+0000","last_change":"2024-05-08T21:22:50.204754+0000","last_active":"2024-05-08T21:23:04.637478+0000","last_peered":"2024-05-08T21:23:04.637478+0000","last_clean":"2024-05-08T21:23:04.637478+0000","last_became_active":"2024-05-08T21:22:50.204473+0000","last_became_peered":"2024-05-08T21:22:50.204473+0000","last_unstale":"2024-05-08T21:23:04.637478+0000","last_undegraded":"2024-05-08T21:23:04.637478+0000","last_fullsized":"2024-05-08T21:23:04.637478+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:14.396616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.640966+0000","last_change":"2024-05-08T21:22:49.741850+0000","last_active":"2024-05-08T21:23:04.640966+0000","last_peered":"2024-05-08T21:23:04.640966+0000","last_clean":"2024-05-08T21:23:04.640966+0000","last_became_active":"2024-05-08T21:22:49.741527+0000","last_became_peered":"2024-05-08T21:22:49.741527+0000","last_unstale":"2024-05-08T21:23:04.640966+0000","last_undegraded":"2024-05-08T21:23:04.640966+0000","last_fullsized":"2024-05-08T21:23:04.640966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:29:03.389411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493135+0000","last_change":"2024-05-08T21:23:05.490366+0000","last_active":"2024-05-08T21:23:05.493135+0000","last_peered":"2024-05-08T21:23:05.493135+0000","last_clean":"2024-05-08T21:23:05.493135+0000","last_became_active":"2024-05-08T21:23:05.489537+0000","last_became_peered":"2024-05-08T21:23:05.489537+0000","last_unstale":"2024-05-08T21:23:05.493135+0000","last_undegraded":"2024-05-08T21:23:05.493135+0000","last_fullsized":"2024-05-08T21:23:05.493135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:45:57.976239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"34'6","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635140+0000","last_change":"2024-05-08T21:22:50.211613+0000","last_active":"2024-05-08T21:23:04.635140+0000","last_peered":"2024-05-08T21:23:04.635140+0000","last_clean":"2024-05-08T21:23:04.635140+0000","last_became_active":"2024-05-08T21:22:50.211335+0000","last_became_peered":"2024-05-08T21:22:50.211335+0000","last_unstale":"2024-05-08T21:23:04.635140+0000","last_undegraded":"2024-05-08T21:23:04.635140+0000","last_fullsized":"2024-05-08T21:23:04.635140+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:43:00.051225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634106+0000","last_change":"2024-05-08T21:22:49.737257+0000","last_active":"2024-05-08T21:23:04.634106+0000","last_peered":"2024-05-08T21:23:04.634106+0000","last_clean":"2024-05-08T21:23:04.634106+0000","last_became_active":"2024-05-08T21:22:49.736591+0000","last_became_peered":"2024-05-08T21:22:49.736591+0000","last_unstale":"2024-05-08T21:23:04.634106+0000","last_undegraded":"2024-05-08T21:23:04.634106+0000","last_fullsized":"2024-05-08T21:23:04.634106+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:08.158087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.490343+0000","last_change":"2024-05-08T21:23:05.486693+0000","last_active":"2024-05-08T21:23:05.490343+0000","last_peered":"2024-05-08T21:23:05.490343+0000","last_clean":"2024-05-08T21:23:05.490343+0000","last_became_active":"2024-05-08T21:23:05.486375+0000","last_became_peered":"2024-05-08T21:23:05.486375+0000","last_unstale":"2024-05-08T21:23:05.490343+0000","last_undegraded":"2024-05-08T21:23:05.490343+0000","last_fullsized":"2024-05-08T21:23:05.490343+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:28:15.819527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"34'6","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637407+0000","last_change":"2024-05-08T21:22:50.209206+0000","last_active":"2024-05-08T21:23:04.637407+0000","last_peered":"2024-05-08T21:23:04.637407+0000","last_clean":"2024-05-08T21:23:04.637407+0000","last_became_active":"2024-05-08T21:22:50.208950+0000","last_became_peered":"2024-05-08T21:22:50.208950+0000","last_unstale":"2024-05-08T21:23:04.637407+0000","last_undegraded":"2024-05-08T21:23:04.637407+0000","last_fullsized":"2024-05-08T21:23:04.637407+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:15:17.946198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"36'15","reported_seq":32,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.633412+0000","last_change":"2024-05-08T21:22:49.743861+0000","last_active":"2024-05-08T21:23:04.633412+0000","last_peered":"2024-05-08T21:23:04.633412+0000","last_clean":"2024-05-08T21:23:04.633412+0000","last_became_active":"2024-05-08T21:22:49.743575+0000","last_became_peered":"2024-05-08T21:22:49.743575+0000","last_unstale":"2024-05-08T21:23:04.633412+0000","last_undegraded":"2024-05-08T21:23:04.633412+0000","last_fullsized":"2024-05-08T21:23:04.633412+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:51.977551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.495313+0000","last_change":"2024-05-08T21:23:05.491466+0000","last_active":"2024-05-08T21:23:05.495313+0000","last_peered":"2024-05-08T21:23:05.495313+0000","last_clean":"2024-05-08T21:23:05.495313+0000","last_became_active":"2024-05-08T21:23:05.491195+0000","last_became_peered":"2024-05-08T21:23:05.491195+0000","last_unstale":"2024-05-08T21:23:05.495313+0000","last_undegraded":"2024-05-08T21:23:05.495313+0000","last_fullsized":"2024-05-08T21:23:05.495313+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:54:44.860139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"34'5","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.633100+0000","last_change":"2024-05-08T21:22:50.203855+0000","last_active":"2024-05-08T21:23:04.633100+0000","last_peered":"2024-05-08T21:23:04.633100+0000","last_clean":"2024-05-08T21:23:04.633100+0000","last_became_active":"2024-05-08T21:22:50.203557+0000","last_became_peered":"2024-05-08T21:22:50.203557+0000","last_unstale":"2024-05-08T21:23:04.633100+0000","last_undegraded":"2024-05-08T21:23:04.633100+0000","last_fullsized":"2024-05-08T21:23:04.633100+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:09.165560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"36'2","reported_seq":19,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641397+0000","last_change":"2024-05-08T21:22:49.742208+0000","last_active":"2024-05-08T21:23:04.641397+0000","last_peered":"2024-05-08T21:23:04.641397+0000","last_clean":"2024-05-08T21:23:04.641397+0000","last_became_active":"2024-05-08T21:22:49.741894+0000","last_became_peered":"2024-05-08T21:22:49.741894+0000","last_unstale":"2024-05-08T21:23:04.641397+0000","last_undegraded":"2024-05-08T21:23:04.641397+0000","last_fullsized":"2024-05-08T21:23:04.641397+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:59:36.798032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.519266+0000","last_change":"2024-05-08T21:23:05.518243+0000","last_active":"2024-05-08T21:23:05.519266+0000","last_peered":"2024-05-08T21:23:05.519266+0000","last_clean":"2024-05-08T21:23:05.519266+0000","last_became_active":"2024-05-08T21:23:05.517600+0000","last_became_peered":"2024-05-08T21:23:05.517600+0000","last_unstale":"2024-05-08T21:23:05.519266+0000","last_undegraded":"2024-05-08T21:23:05.519266+0000","last_fullsized":"2024-05-08T21:23:05.519266+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:12:43.572456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"34'9","reported_seq":26,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.639389+0000","last_change":"2024-05-08T21:22:50.208102+0000","last_active":"2024-05-08T21:23:04.639389+0000","last_peered":"2024-05-08T21:23:04.639389+0000","last_clean":"2024-05-08T21:23:04.639389+0000","last_became_active":"2024-05-08T21:22:50.207765+0000","last_became_peered":"2024-05-08T21:22:50.207765+0000","last_unstale":"2024-05-08T21:23:04.639389+0000","last_undegraded":"2024-05-08T21:23:04.639389+0000","last_fullsized":"2024-05-08T21:23:04.639389+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:14.396616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"36'3","reported_seq":21,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.633073+0000","last_change":"2024-05-08T21:22:49.734619+0000","last_active":"2024-05-08T21:23:04.633073+0000","last_peered":"2024-05-08T21:23:04.633073+0000","last_clean":"2024-05-08T21:23:04.633073+0000","last_became_active":"2024-05-08T21:22:49.734354+0000","last_became_peered":"2024-05-08T21:22:49.734354+0000","last_unstale":"2024-05-08T21:23:04.633073+0000","last_undegraded":"2024-05-08T21:23:04.633073+0000","last_fullsized":"2024-05-08T21:23:04.633073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:23:35.220950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.492794+0000","last_change":"2024-05-08T21:23:05.491235+0000","last_active":"2024-05-08T21:23:05.492794+0000","last_peered":"2024-05-08T21:23:05.492794+0000","last_clean":"2024-05-08T21:23:05.492794+0000","last_became_active":"2024-05-08T21:23:05.490957+0000","last_became_peered":"2024-05-08T21:23:05.490957+0000","last_unstale":"2024-05-08T21:23:05.492794+0000","last_undegraded":"2024-05-08T21:23:05.492794+0000","last_fullsized":"2024-05-08T21:23:05.492794+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:38.632583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"34'6","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641077+0000","last_change":"2024-05-08T21:22:50.210640+0000","last_active":"2024-05-08T21:23:04.641077+0000","last_peered":"2024-05-08T21:23:04.641077+0000","last_clean":"2024-05-08T21:23:04.641077+0000","last_became_active":"2024-05-08T21:22:50.210283+0000","last_became_peered":"2024-05-08T21:22:50.210283+0000","last_unstale":"2024-05-08T21:23:04.641077+0000","last_undegraded":"2024-05-08T21:23:04.641077+0000","last_fullsized":"2024-05-08T21:23:04.641077+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:58:46.704338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"36'2","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632967+0000","last_change":"2024-05-08T21:22:49.737099+0000","last_active":"2024-05-08T21:23:04.632967+0000","last_peered":"2024-05-08T21:23:04.632967+0000","last_clean":"2024-05-08T21:23:04.632967+0000","last_became_active":"2024-05-08T21:22:49.736839+0000","last_became_peered":"2024-05-08T21:22:49.736839+0000","last_unstale":"2024-05-08T21:23:04.632967+0000","last_undegraded":"2024-05-08T21:23:04.632967+0000","last_fullsized":"2024-05-08T21:23:04.632967+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:05:26.886351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.498088+0000","last_change":"2024-05-08T21:23:05.495475+0000","last_active":"2024-05-08T21:23:05.498088+0000","last_peered":"2024-05-08T21:23:05.498088+0000","last_clean":"2024-05-08T21:23:05.498088+0000","last_became_active":"2024-05-08T21:23:05.495181+0000","last_became_peered":"2024-05-08T21:23:05.495181+0000","last_unstale":"2024-05-08T21:23:05.498088+0000","last_undegraded":"2024-05-08T21:23:05.498088+0000","last_fullsized":"2024-05-08T21:23:05.498088+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:34:11.128072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"34'7","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634805+0000","last_change":"2024-05-08T21:22:50.206786+0000","last_active":"2024-05-08T21:23:04.634805+0000","last_peered":"2024-05-08T21:23:04.634805+0000","last_clean":"2024-05-08T21:23:04.634805+0000","last_became_active":"2024-05-08T21:22:50.206484+0000","last_became_peered":"2024-05-08T21:22:50.206484+0000","last_unstale":"2024-05-08T21:23:04.634805+0000","last_undegraded":"2024-05-08T21:23:04.634805+0000","last_fullsized":"2024-05-08T21:23:04.634805+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:36:44.630301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632995+0000","last_change":"2024-05-08T21:22:49.740491+0000","last_active":"2024-05-08T21:23:04.632995+0000","last_peered":"2024-05-08T21:23:04.632995+0000","last_clean":"2024-05-08T21:23:04.632995+0000","last_became_active":"2024-05-08T21:22:49.740166+0000","last_became_peered":"2024-05-08T21:22:49.740166+0000","last_unstale":"2024-05-08T21:23:04.632995+0000","last_undegraded":"2024-05-08T21:23:04.632995+0000","last_fullsized":"2024-05-08T21:23:04.632995+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:27:16.794172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493593+0000","last_change":"2024-05-08T21:23:05.490329+0000","last_active":"2024-05-08T21:23:05.493593+0000","last_peered":"2024-05-08T21:23:05.493593+0000","last_clean":"2024-05-08T21:23:05.493593+0000","last_became_active":"2024-05-08T21:23:05.490030+0000","last_became_peered":"2024-05-08T21:23:05.490030+0000","last_unstale":"2024-05-08T21:23:05.493593+0000","last_undegraded":"2024-05-08T21:23:05.493593+0000","last_fullsized":"2024-05-08T21:23:05.493593+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:05:41.899542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"34'5","reported_seq":21,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634676+0000","last_change":"2024-05-08T21:22:50.207827+0000","last_active":"2024-05-08T21:23:04.634676+0000","last_peered":"2024-05-08T21:23:04.634676+0000","last_clean":"2024-05-08T21:23:04.634676+0000","last_became_active":"2024-05-08T21:22:50.207560+0000","last_became_peered":"2024-05-08T21:22:50.207560+0000","last_unstale":"2024-05-08T21:23:04.634676+0000","last_undegraded":"2024-05-08T21:23:04.634676+0000","last_fullsized":"2024-05-08T21:23:04.634676+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:29:04.396884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"37'4","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637281+0000","last_change":"2024-05-08T21:22:49.743647+0000","last_active":"2024-05-08T21:23:04.637281+0000","last_peered":"2024-05-08T21:23:04.637281+0000","last_clean":"2024-05-08T21:23:04.637281+0000","last_became_active":"2024-05-08T21:22:49.743349+0000","last_became_peered":"2024-05-08T21:22:49.743349+0000","last_unstale":"2024-05-08T21:23:04.637281+0000","last_undegraded":"2024-05-08T21:23:04.637281+0000","last_fullsized":"2024-05-08T21:23:04.637281+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:53:44.252529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.489385+0000","last_change":"2024-05-08T21:23:05.486117+0000","last_active":"2024-05-08T21:23:05.489385+0000","last_peered":"2024-05-08T21:23:05.489385+0000","last_clean":"2024-05-08T21:23:05.489385+0000","last_became_active":"2024-05-08T21:23:05.485848+0000","last_became_peered":"2024-05-08T21:23:05.485848+0000","last_unstale":"2024-05-08T21:23:05.489385+0000","last_undegraded":"2024-05-08T21:23:05.489385+0000","last_fullsized":"2024-05-08T21:23:05.489385+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:30:04.803867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"34'8","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634374+0000","last_change":"2024-05-08T21:22:50.205819+0000","last_active":"2024-05-08T21:23:04.634374+0000","last_peered":"2024-05-08T21:23:04.634374+0000","last_clean":"2024-05-08T21:23:04.634374+0000","last_became_active":"2024-05-08T21:22:50.205239+0000","last_became_peered":"2024-05-08T21:22:50.205239+0000","last_unstale":"2024-05-08T21:23:04.634374+0000","last_undegraded":"2024-05-08T21:23:04.634374+0000","last_fullsized":"2024-05-08T21:23:04.634374+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:52.985024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637665+0000","last_change":"2024-05-08T21:22:49.743818+0000","last_active":"2024-05-08T21:23:04.637665+0000","last_peered":"2024-05-08T21:23:04.637665+0000","last_clean":"2024-05-08T21:23:04.637665+0000","last_became_active":"2024-05-08T21:22:49.743469+0000","last_became_peered":"2024-05-08T21:22:49.743469+0000","last_unstale":"2024-05-08T21:23:04.637665+0000","last_undegraded":"2024-05-08T21:23:04.637665+0000","last_fullsized":"2024-05-08T21:23:04.637665+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:06:42.743465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493381+0000","last_change":"2024-05-08T21:23:05.491490+0000","last_active":"2024-05-08T21:23:05.493381+0000","last_peered":"2024-05-08T21:23:05.493381+0000","last_clean":"2024-05-08T21:23:05.493381+0000","last_became_active":"2024-05-08T21:23:05.491231+0000","last_became_peered":"2024-05-08T21:23:05.491231+0000","last_unstale":"2024-05-08T21:23:05.493381+0000","last_undegraded":"2024-05-08T21:23:05.493381+0000","last_fullsized":"2024-05-08T21:23:05.493381+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:28:06.867361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"34'9","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634265+0000","last_change":"2024-05-08T21:22:50.203236+0000","last_active":"2024-05-08T21:23:04.634265+0000","last_peered":"2024-05-08T21:23:04.634265+0000","last_clean":"2024-05-08T21:23:04.634265+0000","last_became_active":"2024-05-08T21:22:50.201914+0000","last_became_peered":"2024-05-08T21:22:50.201914+0000","last_unstale":"2024-05-08T21:23:04.634265+0000","last_undegraded":"2024-05-08T21:23:04.634265+0000","last_fullsized":"2024-05-08T21:23:04.634265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:06:10.043716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632487+0000","last_change":"2024-05-08T21:22:49.739798+0000","last_active":"2024-05-08T21:23:04.632487+0000","last_peered":"2024-05-08T21:23:04.632487+0000","last_clean":"2024-05-08T21:23:04.632487+0000","last_became_active":"2024-05-08T21:22:49.739520+0000","last_became_peered":"2024-05-08T21:22:49.739520+0000","last_unstale":"2024-05-08T21:23:04.632487+0000","last_undegraded":"2024-05-08T21:23:04.632487+0000","last_fullsized":"2024-05-08T21:23:04.632487+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:47:50.170209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.518668+0000","last_change":"2024-05-08T21:23:05.517869+0000","last_active":"2024-05-08T21:23:05.518668+0000","last_peered":"2024-05-08T21:23:05.518668+0000","last_clean":"2024-05-08T21:23:05.518668+0000","last_became_active":"2024-05-08T21:23:05.517598+0000","last_became_peered":"2024-05-08T21:23:05.517598+0000","last_unstale":"2024-05-08T21:23:05.518668+0000","last_undegraded":"2024-05-08T21:23:05.518668+0000","last_fullsized":"2024-05-08T21:23:05.518668+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:52:59.903852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"34'2","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635163+0000","last_change":"2024-05-08T21:22:50.204561+0000","last_active":"2024-05-08T21:23:04.635163+0000","last_peered":"2024-05-08T21:23:04.635163+0000","last_clean":"2024-05-08T21:23:04.635163+0000","last_became_active":"2024-05-08T21:22:50.204296+0000","last_became_peered":"2024-05-08T21:22:50.204296+0000","last_unstale":"2024-05-08T21:23:04.635163+0000","last_undegraded":"2024-05-08T21:23:04.635163+0000","last_fullsized":"2024-05-08T21:23:04.635163+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:25:52.293163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638387+0000","last_change":"2024-05-08T21:22:49.737100+0000","last_active":"2024-05-08T21:23:04.638387+0000","last_peered":"2024-05-08T21:23:04.638387+0000","last_clean":"2024-05-08T21:23:04.638387+0000","last_became_active":"2024-05-08T21:22:49.736715+0000","last_became_peered":"2024-05-08T21:22:49.736715+0000","last_unstale":"2024-05-08T21:23:04.638387+0000","last_undegraded":"2024-05-08T21:23:04.638387+0000","last_fullsized":"2024-05-08T21:23:04.638387+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:52:46.503073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.519426+0000","last_change":"2024-05-08T21:23:05.518020+0000","last_active":"2024-05-08T21:23:05.519426+0000","last_peered":"2024-05-08T21:23:05.519426+0000","last_clean":"2024-05-08T21:23:05.519426+0000","last_became_active":"2024-05-08T21:23:05.517699+0000","last_became_peered":"2024-05-08T21:23:05.517699+0000","last_unstale":"2024-05-08T21:23:05.519426+0000","last_undegraded":"2024-05-08T21:23:05.519426+0000","last_fullsized":"2024-05-08T21:23:05.519426+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:19:40.125296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"34'7","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634647+0000","last_change":"2024-05-08T21:22:50.206910+0000","last_active":"2024-05-08T21:23:04.634647+0000","last_peered":"2024-05-08T21:23:04.634647+0000","last_clean":"2024-05-08T21:23:04.634647+0000","last_became_active":"2024-05-08T21:22:50.206623+0000","last_became_peered":"2024-05-08T21:22:50.206623+0000","last_unstale":"2024-05-08T21:23:04.634647+0000","last_undegraded":"2024-05-08T21:23:04.634647+0000","last_fullsized":"2024-05-08T21:23:04.634647+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:24:25.163652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"36'3","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641631+0000","last_change":"2024-05-08T21:22:49.743344+0000","last_active":"2024-05-08T21:23:04.641631+0000","last_peered":"2024-05-08T21:23:04.641631+0000","last_clean":"2024-05-08T21:23:04.641631+0000","last_became_active":"2024-05-08T21:22:49.742413+0000","last_became_peered":"2024-05-08T21:22:49.742413+0000","last_unstale":"2024-05-08T21:23:04.641631+0000","last_undegraded":"2024-05-08T21:23:04.641631+0000","last_fullsized":"2024-05-08T21:23:04.641631+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:06:42.743465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493284+0000","last_change":"2024-05-08T21:23:05.491687+0000","last_active":"2024-05-08T21:23:05.493284+0000","last_peered":"2024-05-08T21:23:05.493284+0000","last_clean":"2024-05-08T21:23:05.493284+0000","last_became_active":"2024-05-08T21:23:05.491402+0000","last_became_peered":"2024-05-08T21:23:05.491402+0000","last_unstale":"2024-05-08T21:23:05.493284+0000","last_undegraded":"2024-05-08T21:23:05.493284+0000","last_fullsized":"2024-05-08T21:23:05.493284+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:55:39.465260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"34'8","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.639350+0000","last_change":"2024-05-08T21:22:50.205911+0000","last_active":"2024-05-08T21:23:04.639350+0000","last_peered":"2024-05-08T21:23:04.639350+0000","last_clean":"2024-05-08T21:23:04.639350+0000","last_became_active":"2024-05-08T21:22:50.205609+0000","last_became_peered":"2024-05-08T21:22:50.205609+0000","last_unstale":"2024-05-08T21:23:04.639350+0000","last_undegraded":"2024-05-08T21:23:04.639350+0000","last_fullsized":"2024-05-08T21:23:04.639350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:41:28.819967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"37'3","reported_seq":19,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637909+0000","last_change":"2024-05-08T21:22:49.745047+0000","last_active":"2024-05-08T21:23:04.637909+0000","last_peered":"2024-05-08T21:23:04.637909+0000","last_clean":"2024-05-08T21:23:04.637909+0000","last_became_active":"2024-05-08T21:22:49.744747+0000","last_became_peered":"2024-05-08T21:22:49.744747+0000","last_unstale":"2024-05-08T21:23:04.637909+0000","last_undegraded":"2024-05-08T21:23:04.637909+0000","last_fullsized":"2024-05-08T21:23:04.637909+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:05:26.886351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493199+0000","last_change":"2024-05-08T21:23:05.491039+0000","last_active":"2024-05-08T21:23:05.493199+0000","last_peered":"2024-05-08T21:23:05.493199+0000","last_clean":"2024-05-08T21:23:05.493199+0000","last_became_active":"2024-05-08T21:23:05.490697+0000","last_became_peered":"2024-05-08T21:23:05.490697+0000","last_unstale":"2024-05-08T21:23:05.493199+0000","last_undegraded":"2024-05-08T21:23:05.493199+0000","last_fullsized":"2024-05-08T21:23:05.493199+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:54:44.860139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"34'5","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.636798+0000","last_change":"2024-05-08T21:22:50.203176+0000","last_active":"2024-05-08T21:23:04.636798+0000","last_peered":"2024-05-08T21:23:04.636798+0000","last_clean":"2024-05-08T21:23:04.636798+0000","last_became_active":"2024-05-08T21:22:50.202886+0000","last_became_peered":"2024-05-08T21:22:50.202886+0000","last_unstale":"2024-05-08T21:23:04.636798+0000","last_undegraded":"2024-05-08T21:23:04.636798+0000","last_fullsized":"2024-05-08T21:23:04.636798+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:40.200175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641367+0000","last_change":"2024-05-08T21:22:49.742028+0000","last_active":"2024-05-08T21:23:04.641367+0000","last_peered":"2024-05-08T21:23:04.641367+0000","last_clean":"2024-05-08T21:23:04.641367+0000","last_became_active":"2024-05-08T21:22:49.741706+0000","last_became_peered":"2024-05-08T21:22:49.741706+0000","last_unstale":"2024-05-08T21:23:04.641367+0000","last_undegraded":"2024-05-08T21:23:04.641367+0000","last_fullsized":"2024-05-08T21:23:04.641367+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:06:09.036243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.518482+0000","last_change":"2024-05-08T21:23:05.517966+0000","last_active":"2024-05-08T21:23:05.518482+0000","last_peered":"2024-05-08T21:23:05.518482+0000","last_clean":"2024-05-08T21:23:05.518482+0000","last_became_active":"2024-05-08T21:23:05.517646+0000","last_became_peered":"2024-05-08T21:23:05.517646+0000","last_unstale":"2024-05-08T21:23:05.518482+0000","last_undegraded":"2024-05-08T21:23:05.518482+0000","last_fullsized":"2024-05-08T21:23:05.518482+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:11:31.133470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"34'8","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637421+0000","last_change":"2024-05-08T21:22:50.210183+0000","last_active":"2024-05-08T21:23:04.637421+0000","last_peered":"2024-05-08T21:23:04.637421+0000","last_clean":"2024-05-08T21:23:04.637421+0000","last_became_active":"2024-05-08T21:22:50.209860+0000","last_became_peered":"2024-05-08T21:22:50.209860+0000","last_unstale":"2024-05-08T21:23:04.637421+0000","last_undegraded":"2024-05-08T21:23:04.637421+0000","last_fullsized":"2024-05-08T21:23:04.637421+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:28.358029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637180+0000","last_change":"2024-05-08T21:22:49.740650+0000","last_active":"2024-05-08T21:23:04.637180+0000","last_peered":"2024-05-08T21:23:04.637180+0000","last_clean":"2024-05-08T21:23:04.637180+0000","last_became_active":"2024-05-08T21:22:49.740373+0000","last_became_peered":"2024-05-08T21:22:49.740373+0000","last_unstale":"2024-05-08T21:23:04.637180+0000","last_undegraded":"2024-05-08T21:23:04.637180+0000","last_fullsized":"2024-05-08T21:23:04.637180+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:36:43.622828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.494023+0000","last_change":"2024-05-08T21:23:05.493142+0000","last_active":"2024-05-08T21:23:05.494023+0000","last_peered":"2024-05-08T21:23:05.494023+0000","last_clean":"2024-05-08T21:23:05.494023+0000","last_became_active":"2024-05-08T21:23:05.492813+0000","last_became_peered":"2024-05-08T21:23:05.492813+0000","last_unstale":"2024-05-08T21:23:05.494023+0000","last_undegraded":"2024-05-08T21:23:05.494023+0000","last_fullsized":"2024-05-08T21:23:05.494023+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:30.977530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"34'7","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637385+0000","last_change":"2024-05-08T21:22:50.208073+0000","last_active":"2024-05-08T21:23:04.637385+0000","last_peered":"2024-05-08T21:23:04.637385+0000","last_clean":"2024-05-08T21:23:04.637385+0000","last_became_active":"2024-05-08T21:22:50.207506+0000","last_became_peered":"2024-05-08T21:22:50.207506+0000","last_unstale":"2024-05-08T21:23:04.637385+0000","last_undegraded":"2024-05-08T21:23:04.637385+0000","last_fullsized":"2024-05-08T21:23:04.637385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:15:17.946198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.640849+0000","last_change":"2024-05-08T21:22:49.742392+0000","last_active":"2024-05-08T21:23:04.640849+0000","last_peered":"2024-05-08T21:23:04.640849+0000","last_clean":"2024-05-08T21:23:04.640849+0000","last_became_active":"2024-05-08T21:22:49.742093+0000","last_became_peered":"2024-05-08T21:22:49.742093+0000","last_unstale":"2024-05-08T21:23:04.640849+0000","last_undegraded":"2024-05-08T21:23:04.640849+0000","last_fullsized":"2024-05-08T21:23:04.640849+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:51.977551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.519268+0000","last_change":"2024-05-08T21:23:05.517833+0000","last_active":"2024-05-08T21:23:05.519268+0000","last_peered":"2024-05-08T21:23:05.519268+0000","last_clean":"2024-05-08T21:23:05.519268+0000","last_became_active":"2024-05-08T21:23:05.517531+0000","last_became_peered":"2024-05-08T21:23:05.517531+0000","last_unstale":"2024-05-08T21:23:05.519268+0000","last_undegraded":"2024-05-08T21:23:05.519268+0000","last_fullsized":"2024-05-08T21:23:05.519268+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:26:26.242337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"34'8","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.633068+0000","last_change":"2024-05-08T21:22:50.214641+0000","last_active":"2024-05-08T21:23:04.633068+0000","last_peered":"2024-05-08T21:23:04.633068+0000","last_clean":"2024-05-08T21:23:04.633068+0000","last_became_active":"2024-05-08T21:22:50.213619+0000","last_became_peered":"2024-05-08T21:22:50.213619+0000","last_unstale":"2024-05-08T21:23:04.633068+0000","last_undegraded":"2024-05-08T21:23:04.633068+0000","last_fullsized":"2024-05-08T21:23:04.633068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:21:31.919852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632569+0000","last_change":"2024-05-08T21:22:49.738120+0000","last_active":"2024-05-08T21:23:04.632569+0000","last_peered":"2024-05-08T21:23:04.632569+0000","last_clean":"2024-05-08T21:23:04.632569+0000","last_became_active":"2024-05-08T21:22:49.737805+0000","last_became_peered":"2024-05-08T21:22:49.737805+0000","last_unstale":"2024-05-08T21:23:04.632569+0000","last_undegraded":"2024-05-08T21:23:04.632569+0000","last_fullsized":"2024-05-08T21:23:04.632569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:55:00.385106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.495445+0000","last_change":"2024-05-08T21:23:05.490991+0000","last_active":"2024-05-08T21:23:05.495445+0000","last_peered":"2024-05-08T21:23:05.495445+0000","last_clean":"2024-05-08T21:23:05.495445+0000","last_became_active":"2024-05-08T21:23:05.490651+0000","last_became_peered":"2024-05-08T21:23:05.490651+0000","last_unstale":"2024-05-08T21:23:05.495445+0000","last_undegraded":"2024-05-08T21:23:05.495445+0000","last_fullsized":"2024-05-08T21:23:05.495445+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:48:55.773935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"34'5","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632336+0000","last_change":"2024-05-08T21:22:50.205043+0000","last_active":"2024-05-08T21:23:04.632336+0000","last_peered":"2024-05-08T21:23:04.632336+0000","last_clean":"2024-05-08T21:23:04.632336+0000","last_became_active":"2024-05-08T21:22:50.204630+0000","last_became_peered":"2024-05-08T21:22:50.204630+0000","last_unstale":"2024-05-08T21:23:04.632336+0000","last_undegraded":"2024-05-08T21:23:04.632336+0000","last_fullsized":"2024-05-08T21:23:04.632336+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:26:57.146675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.633296+0000","last_change":"2024-05-08T21:22:49.736783+0000","last_active":"2024-05-08T21:23:04.633296+0000","last_peered":"2024-05-08T21:23:04.633296+0000","last_clean":"2024-05-08T21:23:04.633296+0000","last_became_active":"2024-05-08T21:22:49.736479+0000","last_became_peered":"2024-05-08T21:22:49.736479+0000","last_unstale":"2024-05-08T21:23:04.633296+0000","last_undegraded":"2024-05-08T21:23:04.633296+0000","last_fullsized":"2024-05-08T21:23:04.633296+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:24:24.156179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.495438+0000","last_change":"2024-05-08T21:23:05.493679+0000","last_active":"2024-05-08T21:23:05.495438+0000","last_peered":"2024-05-08T21:23:05.495438+0000","last_clean":"2024-05-08T21:23:05.495438+0000","last_became_active":"2024-05-08T21:23:05.493429+0000","last_became_peered":"2024-05-08T21:23:05.493429+0000","last_unstale":"2024-05-08T21:23:05.495438+0000","last_undegraded":"2024-05-08T21:23:05.495438+0000","last_fullsized":"2024-05-08T21:23:05.495438+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:28:15.819527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"34'9","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635260+0000","last_change":"2024-05-08T21:22:50.202594+0000","last_active":"2024-05-08T21:23:04.635260+0000","last_peered":"2024-05-08T21:23:04.635260+0000","last_clean":"2024-05-08T21:23:04.635260+0000","last_became_active":"2024-05-08T21:22:50.202278+0000","last_became_peered":"2024-05-08T21:22:50.202278+0000","last_unstale":"2024-05-08T21:23:04.635260+0000","last_undegraded":"2024-05-08T21:23:04.635260+0000","last_fullsized":"2024-05-08T21:23:04.635260+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:44:18.055232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637697+0000","last_change":"2024-05-08T21:22:49.739311+0000","last_active":"2024-05-08T21:23:04.637697+0000","last_peered":"2024-05-08T21:23:04.637697+0000","last_clean":"2024-05-08T21:23:04.637697+0000","last_became_active":"2024-05-08T21:22:49.739006+0000","last_became_peered":"2024-05-08T21:22:49.739006+0000","last_unstale":"2024-05-08T21:23:04.637697+0000","last_undegraded":"2024-05-08T21:23:04.637697+0000","last_fullsized":"2024-05-08T21:23:04.637697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:39:42.856227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":183,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635012+0000","last_change":"2024-05-08T21:21:42.333097+0000","last_active":"2024-05-08T21:23:04.635012+0000","last_peered":"2024-05-08T21:23:04.635012+0000","last_clean":"2024-05-08T21:23:04.635012+0000","last_became_active":"2024-05-08T21:21:42.332033+0000","last_became_peered":"2024-05-08T21:21:42.332033+0000","last_unstale":"2024-05-08T21:23:04.635012+0000","last_undegraded":"2024-05-08T21:23:04.635012+0000","last_fullsized":"2024-05-08T21:23:04.635012+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_clean_scrub_stamp":"2024-05-08T21:21:40.862622+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:39:24.772551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":6291456,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12582912,"data_stored":12582912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":504,"ondisk_log_size":504,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":92,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30112,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341728,"statfs":{"total":96636764160,"available":96605929472,"internally_reserved":0,"allocated":3162112,"data_stored":2883557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.72199999999999998}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.76900000000000002}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.82599999999999996}]},{"osd":3,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.80100000000000005}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.69899999999999995}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.57599999999999996}]},{"osd":6,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.63800000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29516,"kb_used_data":2496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342324,"statfs":{"total":96636764160,"available":96606539776,"internally_reserved":0,"allocated":2555904,"data_stored":2285873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.497},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.56100000000000005}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.443},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.64800000000000002}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.66100000000000003}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.73899999999999999}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.68200000000000005}]},{"osd":7,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.752}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31564,"kb_used_data":4544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340276,"statfs":{"total":96636764160,"available":96604442624,"internally_reserved":0,"allocated":4653056,"data_stored":4383025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.375},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.35599999999999998}]},{"osd":2,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.504}]},{"osd":3,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.55200000000000005}]},{"osd":4,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.56899999999999995}]},{"osd":5,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.44900000000000001}]},{"osd":6,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.52500000000000002}]},{"osd":7,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.624},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.59199999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343776,"statfs":{"total":96636764160,"available":96608026624,"internally_reserved":0,"allocated":1064960,"data_stored":786405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.53700000000000003}]},{"osd":2,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.436}]},{"osd":3,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.57299999999999995}]},{"osd":4,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.55800000000000005}]},{"osd":5,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.61499999999999999}]},{"osd":6,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.69099999999999995}]},{"osd":7,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.65100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":458752,"data_stored":188721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.41899999999999998}]},{"osd":1,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.33200000000000002}]},{"osd":3,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.432}]},{"osd":4,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.495}]},{"osd":5,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.51100000000000001}]},{"osd":6,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.76000000000000001}]},{"osd":7,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.623}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":458752,"data_stored":188721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":2.7919999999999998,"5min":2.7919999999999998,"15min":2.7919999999999998},"last":0.41099999999999998}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.438},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.54500000000000004}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":2.9369999999999998,"5min":2.9369999999999998,"15min":2.9369999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.57899999999999996}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.8380000000000001,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":0.66000000000000003}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.9060000000000001,"5min":2.9060000000000001,"15min":2.9060000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.68600000000000005}]},{"osd":6,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":2.8730000000000002,"5min":2.8730000000000002,"15min":2.8730000000000002},"last":0.76200000000000001}]},{"osd":7,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.64300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":458752,"data_stored":188721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.50700000000000001}]},{"osd":1,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.38200000000000001}]},{"osd":2,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.622},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.45900000000000002}]},{"osd":3,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.52800000000000002}]},{"osd":5,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.42199999999999999}]},{"osd":6,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65400000000000003}]},{"osd":7,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.59999999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31564,"kb_used_data":4544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340276,"statfs":{"total":96636764160,"available":96604442624,"internally_reserved":0,"allocated":4653056,"data_stored":4383025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.36299999999999999}]},{"osd":1,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.43099999999999999}]},{"osd":2,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.495}]},{"osd":3,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.376},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.45300000000000001}]},{"osd":4,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.40799999999999997}]},{"osd":6,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.443}]},{"osd":7,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2097152,"data_stored":2097152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2097152,"data_stored":2097152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T21:23:09.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T21:23:10.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.058+0000 7f49d052d700 1 Processor -- start 2024-05-08T21:23:10.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.058+0000 7f49d052d700 1 -- start start 2024-05-08T21:23:10.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.059+0000 7f49d052d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f49c80ccbd0 0x7f49c8168580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.059+0000 7f49d052d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c809a870 con 0x7f49c80b2480 2024-05-08T21:23:10.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.059+0000 7f49d052d700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c80d0840 con 0x7f49c8161030 2024-05-08T21:23:10.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.059+0000 7f49d052d700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c806ccc0 con 0x7f49c80ccbd0 2024-05-08T21:23:10.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.059+0000 7f49cdac8700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f49c80b2480 0x7f49c809a160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:42892/0 (socket says 172.21.15.3:42892) 2024-05-08T21:23:10.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.059+0000 7f49cdac8700 1 -- 172.21.15.3:0/2251256460 learned_addr learned my addr 172.21.15.3:0/2251256460 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:10.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.060+0000 7f49ceaca700 1 --2- 172.21.15.3:0/2251256460 >> v2:172.21.15.186:3300/0 conn(0x7f49c80ccbd0 0x7f49c8168580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.060+0000 7f49cd2c7700 1 -- 172.21.15.3:0/2251256460 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724096229 0 0) 0x7f49c809a870 con 0x7f49c80b2480 2024-05-08T21:23:10.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.060+0000 7f49cd2c7700 1 -- 172.21.15.3:0/2251256460 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b40035b0 con 0x7f49c80b2480 2024-05-08T21:23:10.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.060+0000 7f49cd2c7700 1 -- 172.21.15.3:0/2251256460 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 771184686 0 0) 0x7f49b40035b0 con 0x7f49c80b2480 2024-05-08T21:23:10.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.060+0000 7f49ceaca700 1 -- 172.21.15.3:0/2251256460 >> v1:172.21.15.186:6789/0 conn(0x7f49c8161030 legacy=0x7f49c8099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.060+0000 7f49ceaca700 1 -- 172.21.15.3:0/2251256460 >> v1:172.21.15.3:6789/0 conn(0x7f49c80b2480 legacy=0x7f49c809a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.060+0000 7f49ceaca700 1 -- 172.21.15.3:0/2251256460 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c806c600 con 0x7f49c80ccbd0 2024-05-08T21:23:10.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.060+0000 7f49ceaca700 1 --2- 172.21.15.3:0/2251256460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49c80ccbd0 0x7f49c8168580 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009d90 tx=0x7f49c000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fa83875bb85163c server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.060+0000 7f49cd2c7700 1 -- 172.21.15.3:0/2251256460 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c000e070 con 0x7f49c80ccbd0 2024-05-08T21:23:10.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.061+0000 7f49cd2c7700 1 -- 172.21.15.3:0/2251256460 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c0003a20 con 0x7f49c80ccbd0 2024-05-08T21:23:10.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.061+0000 7f49cd2c7700 1 -- 172.21.15.3:0/2251256460 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b8001670 con 0x7f49c80ccbd0 2024-05-08T21:23:10.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.061+0000 7f49d052d700 1 -- 172.21.15.3:0/2251256460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49c80ccbd0 msgr2=0x7f49c8168580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.061+0000 7f49d052d700 1 --2- 172.21.15.3:0/2251256460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49c80ccbd0 0x7f49c8168580 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009d90 tx=0x7f49c000b330 comp rx=0 tx=0).stop 2024-05-08T21:23:10.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.061+0000 7f49d052d700 1 -- 172.21.15.3:0/2251256460 shutdown_connections 2024-05-08T21:23:10.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.061+0000 7f49d052d700 1 --2- 172.21.15.3:0/2251256460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49c80ccbd0 0x7f49c8168580 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.061+0000 7f49d052d700 1 -- 172.21.15.3:0/2251256460 >> 172.21.15.3:0/2251256460 conn(0x7f49c80b2f20 msgr2=0x7f49c80b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:10.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.061+0000 7f49d052d700 1 -- 172.21.15.3:0/2251256460 shutdown_connections 2024-05-08T21:23:10.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.061+0000 7f49d052d700 1 -- 172.21.15.3:0/2251256460 wait complete. 2024-05-08T21:23:10.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.063+0000 7f49d052d700 1 Processor -- start 2024-05-08T21:23:10.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.063+0000 7f49d052d700 1 -- start start 2024-05-08T21:23:10.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.063+0000 7f49d052d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49c80b2480 0x7f49c809ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.063+0000 7f49d052d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49c80ccbd0 0x7f49c80c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.063+0000 7f49d052d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49c8161030 0x7f49c815fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.063+0000 7f49d052d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80d0660 con 0x7f49c8161030 2024-05-08T21:23:10.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.063+0000 7f49d052d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80b5e90 con 0x7f49c80ccbd0 2024-05-08T21:23:10.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.063+0000 7f49d052d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80b6170 con 0x7f49c80b2480 2024-05-08T21:23:10.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.064+0000 7f49ceaca700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49c8161030 0x7f49c815fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.064+0000 7f49ceaca700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49c8161030 0x7f49c815fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54274/0 (socket says 172.21.15.3:54274) 2024-05-08T21:23:10.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.064+0000 7f49ceaca700 1 -- 172.21.15.3:0/2627393471 learned_addr learned my addr 172.21.15.3:0/2627393471 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:10.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.064+0000 7f49cdac8700 1 --2- 172.21.15.3:0/2627393471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49c80ccbd0 0x7f49c80c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.064+0000 7f49ce2c9700 1 --2- 172.21.15.3:0/2627393471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49c80b2480 0x7f49c809ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.064+0000 7f49ceaca700 1 -- 172.21.15.3:0/2627393471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49c80b2480 msgr2=0x7f49c809ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.064+0000 7f49ceaca700 1 --2- 172.21.15.3:0/2627393471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49c80b2480 0x7f49c809ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.064+0000 7f49ceaca700 1 -- 172.21.15.3:0/2627393471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49c80ccbd0 msgr2=0x7f49c80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.064+0000 7f49ceaca700 1 --2- 172.21.15.3:0/2627393471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49c80ccbd0 0x7f49c80c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.064+0000 7f49ceaca700 1 -- 172.21.15.3:0/2627393471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c0009a70 con 0x7f49c8161030 2024-05-08T21:23:10.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.064+0000 7f49ceaca700 1 --2- 172.21.15.3:0/2627393471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49c8161030 0x7f49c815fb60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f49c000b370 tx=0x7f49c0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.064+0000 7f49beffd700 1 -- 172.21.15.3:0/2627393471 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0004640 con 0x7f49c8161030 2024-05-08T21:23:10.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.064+0000 7f49beffd700 1 -- 172.21.15.3:0/2627393471 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c00047a0 con 0x7f49c8161030 2024-05-08T21:23:10.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.064+0000 7f49d052d700 1 -- 172.21.15.3:0/2627393471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c80b6450 con 0x7f49c8161030 2024-05-08T21:23:10.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.064+0000 7f49beffd700 1 -- 172.21.15.3:0/2627393471 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0012670 con 0x7f49c8161030 2024-05-08T21:23:10.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.064+0000 7f49d052d700 1 -- 172.21.15.3:0/2627393471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c81559c0 con 0x7f49c8161030 2024-05-08T21:23:10.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.065+0000 7f49d052d700 1 -- 172.21.15.3:0/2627393471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c8155b50 con 0x7f49c8161030 2024-05-08T21:23:10.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.066+0000 7f49beffd700 1 -- 172.21.15.3:0/2627393471 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f49c00127d0 con 0x7f49c8161030 2024-05-08T21:23:10.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.066+0000 7f49beffd700 1 --2- 172.21.15.3:0/2627393471 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f49ac06ac90 0x7f49ac06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.066+0000 7f49beffd700 1 -- 172.21.15.3:0/2627393471 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f49c008aa90 con 0x7f49c8161030 2024-05-08T21:23:10.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.066+0000 7f49ce2c9700 1 --2- 172.21.15.3:0/2627393471 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f49ac06ac90 0x7f49ac06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.067+0000 7f49ce2c9700 1 --2- 172.21.15.3:0/2627393471 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f49ac06ac90 0x7f49ac06d150 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f49c4000f60 tx=0x7f49c4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.071+0000 7f49beffd700 1 -- 172.21.15.3:0/2627393471 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f49c0055ae0 con 0x7f49c8161030 2024-05-08T21:23:10.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.283+0000 7f49d052d700 1 -- 172.21.15.3:0/2627393471 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f49c809ac90 con 0x7f49ac06ac90 2024-05-08T21:23:10.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.294+0000 7f49beffd700 1 -- 172.21.15.3:0/2627393471 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+369888 (secure 0 0 0) 0x7f49c40050d0 con 0x7f49ac06ac90 2024-05-08T21:23:10.300 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:10.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.298+0000 7f49d052d700 1 -- 172.21.15.3:0/2627393471 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f49ac06ac90 msgr2=0x7f49ac06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.298+0000 7f49d052d700 1 --2- 172.21.15.3:0/2627393471 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f49ac06ac90 0x7f49ac06d150 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f49c4000f60 tx=0x7f49c4009040 comp rx=0 tx=0).stop 2024-05-08T21:23:10.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.298+0000 7f49d052d700 1 -- 172.21.15.3:0/2627393471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49c8161030 msgr2=0x7f49c815fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.298+0000 7f49d052d700 1 --2- 172.21.15.3:0/2627393471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49c8161030 0x7f49c815fb60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f49c000b370 tx=0x7f49c0004ad0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.298+0000 7f49d052d700 1 -- 172.21.15.3:0/2627393471 shutdown_connections 2024-05-08T21:23:10.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.298+0000 7f49d052d700 1 --2- 172.21.15.3:0/2627393471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49c80b2480 0x7f49c809ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.298+0000 7f49d052d700 1 --2- 172.21.15.3:0/2627393471 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f49ac06ac90 0x7f49ac06d150 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.298+0000 7f49d052d700 1 --2- 172.21.15.3:0/2627393471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49c8161030 0x7f49c815fb60 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.298+0000 7f49d052d700 1 --2- 172.21.15.3:0/2627393471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49c80ccbd0 0x7f49c80c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.298+0000 7f49d052d700 1 -- 172.21.15.3:0/2627393471 >> 172.21.15.3:0/2627393471 conn(0x7f49c80b2f20 msgr2=0x7f49c80cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:10.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.298+0000 7f49d052d700 1 -- 172.21.15.3:0/2627393471 shutdown_connections 2024-05-08T21:23:10.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.298+0000 7f49d052d700 1 -- 172.21.15.3:0/2627393471 wait complete. 2024-05-08T21:23:10.305 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-08T21:23:10.313 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-05-08T21:23:08.549080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6881824,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":117,"num_read_kb":100,"num_write":255,"num_write_kb":8602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":702,"ondisk_log_size":702,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233224,"kb_used_data":17056,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754741496,"statfs":{"total":773094113280,"available":772855291904,"internally_reserved":0,"allocated":17465344,"data_stored":15288048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17437,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":42,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001561"},"pg_stats":[{"pgid":"5.3f","version":"34'8","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638017+0000","last_change":"2024-05-08T21:22:50.209349+0000","last_active":"2024-05-08T21:23:04.638017+0000","last_peered":"2024-05-08T21:23:04.638017+0000","last_clean":"2024-05-08T21:23:04.638017+0000","last_became_active":"2024-05-08T21:22:50.209092+0000","last_became_peered":"2024-05-08T21:22:50.209092+0000","last_unstale":"2024-05-08T21:23:04.638017+0000","last_undegraded":"2024-05-08T21:23:04.638017+0000","last_fullsized":"2024-05-08T21:23:04.638017+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:52:47.510546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"34'4","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635118+0000","last_change":"2024-05-08T21:22:50.205276+0000","last_active":"2024-05-08T21:23:04.635118+0000","last_peered":"2024-05-08T21:23:04.635118+0000","last_clean":"2024-05-08T21:23:04.635118+0000","last_became_active":"2024-05-08T21:22:50.205028+0000","last_became_peered":"2024-05-08T21:22:50.205028+0000","last_unstale":"2024-05-08T21:23:04.635118+0000","last_undegraded":"2024-05-08T21:23:04.635118+0000","last_fullsized":"2024-05-08T21:23:04.635118+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:24:34.674332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"34'7","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638887+0000","last_change":"2024-05-08T21:22:50.205185+0000","last_active":"2024-05-08T21:23:04.638887+0000","last_peered":"2024-05-08T21:23:04.638887+0000","last_clean":"2024-05-08T21:23:04.638887+0000","last_became_active":"2024-05-08T21:22:50.204826+0000","last_became_peered":"2024-05-08T21:22:50.204826+0000","last_unstale":"2024-05-08T21:23:04.638887+0000","last_undegraded":"2024-05-08T21:23:04.638887+0000","last_fullsized":"2024-05-08T21:23:04.638887+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:05:25.952745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"34'8","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.640890+0000","last_change":"2024-05-08T21:22:50.210536+0000","last_active":"2024-05-08T21:23:04.640890+0000","last_peered":"2024-05-08T21:23:04.640890+0000","last_clean":"2024-05-08T21:23:04.640890+0000","last_became_active":"2024-05-08T21:22:50.210248+0000","last_became_peered":"2024-05-08T21:22:50.210248+0000","last_unstale":"2024-05-08T21:23:04.640890+0000","last_undegraded":"2024-05-08T21:23:04.640890+0000","last_fullsized":"2024-05-08T21:23:04.640890+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:27:40.074356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"34'4","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634253+0000","last_change":"2024-05-08T21:22:50.208946+0000","last_active":"2024-05-08T21:23:04.634253+0000","last_peered":"2024-05-08T21:23:04.634253+0000","last_clean":"2024-05-08T21:23:04.634253+0000","last_became_active":"2024-05-08T21:22:50.208626+0000","last_became_peered":"2024-05-08T21:22:50.208626+0000","last_unstale":"2024-05-08T21:23:04.634253+0000","last_undegraded":"2024-05-08T21:23:04.634253+0000","last_fullsized":"2024-05-08T21:23:04.634253+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:09.165560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"34'6","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634170+0000","last_change":"2024-05-08T21:22:50.207734+0000","last_active":"2024-05-08T21:23:04.634170+0000","last_peered":"2024-05-08T21:23:04.634170+0000","last_clean":"2024-05-08T21:23:04.634170+0000","last_became_active":"2024-05-08T21:22:50.207392+0000","last_became_peered":"2024-05-08T21:22:50.207392+0000","last_unstale":"2024-05-08T21:23:04.634170+0000","last_undegraded":"2024-05-08T21:23:04.634170+0000","last_fullsized":"2024-05-08T21:23:04.634170+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:23:48.163955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"34'5","reported_seq":21,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635074+0000","last_change":"2024-05-08T21:22:50.210352+0000","last_active":"2024-05-08T21:23:04.635074+0000","last_peered":"2024-05-08T21:23:04.635074+0000","last_clean":"2024-05-08T21:23:04.635074+0000","last_became_active":"2024-05-08T21:22:50.210060+0000","last_became_peered":"2024-05-08T21:22:50.210060+0000","last_unstale":"2024-05-08T21:23:04.635074+0000","last_undegraded":"2024-05-08T21:23:04.635074+0000","last_fullsized":"2024-05-08T21:23:04.635074+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:58:46.704338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"34'9","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634963+0000","last_change":"2024-05-08T21:22:50.203181+0000","last_active":"2024-05-08T21:23:04.634963+0000","last_peered":"2024-05-08T21:23:04.634963+0000","last_clean":"2024-05-08T21:23:04.634963+0000","last_became_active":"2024-05-08T21:22:50.202914+0000","last_became_peered":"2024-05-08T21:22:50.202914+0000","last_unstale":"2024-05-08T21:23:04.634963+0000","last_undegraded":"2024-05-08T21:23:04.634963+0000","last_fullsized":"2024-05-08T21:23:04.634963+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:24:39.295215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"34'11","reported_seq":28,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.633163+0000","last_change":"2024-05-08T21:22:50.207050+0000","last_active":"2024-05-08T21:23:04.633163+0000","last_peered":"2024-05-08T21:23:04.633163+0000","last_clean":"2024-05-08T21:23:04.633163+0000","last_became_active":"2024-05-08T21:22:50.206737+0000","last_became_peered":"2024-05-08T21:22:50.206737+0000","last_unstale":"2024-05-08T21:23:04.633163+0000","last_undegraded":"2024-05-08T21:23:04.633163+0000","last_fullsized":"2024-05-08T21:23:04.633163+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:52:47.510546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"34'15","reported_seq":31,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641182+0000","last_change":"2024-05-08T21:22:50.212260+0000","last_active":"2024-05-08T21:23:04.641182+0000","last_peered":"2024-05-08T21:23:04.641182+0000","last_clean":"2024-05-08T21:23:04.641182+0000","last_became_active":"2024-05-08T21:22:50.211928+0000","last_became_peered":"2024-05-08T21:22:50.211928+0000","last_unstale":"2024-05-08T21:23:04.641182+0000","last_undegraded":"2024-05-08T21:23:04.641182+0000","last_fullsized":"2024-05-08T21:23:04.641182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:43:00.051225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"34'6","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635427+0000","last_change":"2024-05-08T21:22:50.205779+0000","last_active":"2024-05-08T21:23:04.635427+0000","last_peered":"2024-05-08T21:23:04.635427+0000","last_clean":"2024-05-08T21:23:04.635427+0000","last_became_active":"2024-05-08T21:22:50.205533+0000","last_became_peered":"2024-05-08T21:22:50.205533+0000","last_unstale":"2024-05-08T21:23:04.635427+0000","last_undegraded":"2024-05-08T21:23:04.635427+0000","last_fullsized":"2024-05-08T21:23:04.635427+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:24:34.674332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"34'7","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638783+0000","last_change":"2024-05-08T21:22:50.207359+0000","last_active":"2024-05-08T21:23:04.638783+0000","last_peered":"2024-05-08T21:23:04.638783+0000","last_clean":"2024-05-08T21:23:04.638783+0000","last_became_active":"2024-05-08T21:22:50.207045+0000","last_became_peered":"2024-05-08T21:22:50.207045+0000","last_unstale":"2024-05-08T21:23:04.638783+0000","last_undegraded":"2024-05-08T21:23:04.638783+0000","last_fullsized":"2024-05-08T21:23:04.638783+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:24:34.674332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"34'10","reported_seq":26,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634509+0000","last_change":"2024-05-08T21:22:50.207671+0000","last_active":"2024-05-08T21:23:04.634509+0000","last_peered":"2024-05-08T21:23:04.634509+0000","last_clean":"2024-05-08T21:23:04.634509+0000","last_became_active":"2024-05-08T21:22:50.207119+0000","last_became_peered":"2024-05-08T21:22:50.207119+0000","last_unstale":"2024-05-08T21:23:04.634509+0000","last_undegraded":"2024-05-08T21:23:04.634509+0000","last_fullsized":"2024-05-08T21:23:04.634509+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:59:37.805505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"34'8","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634288+0000","last_change":"2024-05-08T21:22:50.216066+0000","last_active":"2024-05-08T21:23:04.634288+0000","last_peered":"2024-05-08T21:23:04.634288+0000","last_clean":"2024-05-08T21:23:04.634288+0000","last_became_active":"2024-05-08T21:22:50.215796+0000","last_became_peered":"2024-05-08T21:22:50.215796+0000","last_unstale":"2024-05-08T21:23:04.634288+0000","last_undegraded":"2024-05-08T21:23:04.634288+0000","last_fullsized":"2024-05-08T21:23:04.634288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:24:39.295215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"34'7","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638525+0000","last_change":"2024-05-08T21:22:50.211332+0000","last_active":"2024-05-08T21:23:04.638525+0000","last_peered":"2024-05-08T21:23:04.638525+0000","last_clean":"2024-05-08T21:23:04.638525+0000","last_became_active":"2024-05-08T21:22:50.210865+0000","last_became_peered":"2024-05-08T21:22:50.210865+0000","last_unstale":"2024-05-08T21:23:04.638525+0000","last_undegraded":"2024-05-08T21:23:04.638525+0000","last_fullsized":"2024-05-08T21:23:04.638525+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:24:39.295215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"34'9","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635045+0000","last_change":"2024-05-08T21:22:50.215021+0000","last_active":"2024-05-08T21:23:04.635045+0000","last_peered":"2024-05-08T21:23:04.635045+0000","last_clean":"2024-05-08T21:23:04.635045+0000","last_became_active":"2024-05-08T21:22:50.214821+0000","last_became_peered":"2024-05-08T21:22:50.214821+0000","last_unstale":"2024-05-08T21:23:04.635045+0000","last_undegraded":"2024-05-08T21:23:04.635045+0000","last_fullsized":"2024-05-08T21:23:04.635045+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:14.396616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"34'9","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635686+0000","last_change":"2024-05-08T21:22:50.205302+0000","last_active":"2024-05-08T21:23:04.635686+0000","last_peered":"2024-05-08T21:23:04.635686+0000","last_clean":"2024-05-08T21:23:04.635686+0000","last_became_active":"2024-05-08T21:22:50.204962+0000","last_became_peered":"2024-05-08T21:22:50.204962+0000","last_unstale":"2024-05-08T21:23:04.635686+0000","last_undegraded":"2024-05-08T21:23:04.635686+0000","last_fullsized":"2024-05-08T21:23:04.635686+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:09:24.655227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097152,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"34'10","reported_seq":26,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635013+0000","last_change":"2024-05-08T21:22:50.208448+0000","last_active":"2024-05-08T21:23:04.635013+0000","last_peered":"2024-05-08T21:23:04.635013+0000","last_clean":"2024-05-08T21:23:04.635013+0000","last_became_active":"2024-05-08T21:22:50.208210+0000","last_became_peered":"2024-05-08T21:22:50.208210+0000","last_unstale":"2024-05-08T21:23:04.635013+0000","last_undegraded":"2024-05-08T21:23:04.635013+0000","last_fullsized":"2024-05-08T21:23:04.635013+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:38:13.012930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"34'7","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634646+0000","last_change":"2024-05-08T21:22:50.206974+0000","last_active":"2024-05-08T21:23:04.634646+0000","last_peered":"2024-05-08T21:23:04.634646+0000","last_clean":"2024-05-08T21:23:04.634646+0000","last_became_active":"2024-05-08T21:22:50.206688+0000","last_became_peered":"2024-05-08T21:22:50.206688+0000","last_unstale":"2024-05-08T21:23:04.634646+0000","last_undegraded":"2024-05-08T21:23:04.634646+0000","last_fullsized":"2024-05-08T21:23:04.634646+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:41:28.819967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"34'8","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635292+0000","last_change":"2024-05-08T21:22:50.207398+0000","last_active":"2024-05-08T21:23:04.635292+0000","last_peered":"2024-05-08T21:23:04.635292+0000","last_clean":"2024-05-08T21:23:04.635292+0000","last_became_active":"2024-05-08T21:22:50.207040+0000","last_became_peered":"2024-05-08T21:22:50.207040+0000","last_unstale":"2024-05-08T21:23:04.635292+0000","last_undegraded":"2024-05-08T21:23:04.635292+0000","last_fullsized":"2024-05-08T21:23:04.635292+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:28.358029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"34'11","reported_seq":28,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632605+0000","last_change":"2024-05-08T21:22:50.199407+0000","last_active":"2024-05-08T21:23:04.632605+0000","last_peered":"2024-05-08T21:23:04.632605+0000","last_clean":"2024-05-08T21:23:04.632605+0000","last_became_active":"2024-05-08T21:22:50.199133+0000","last_became_peered":"2024-05-08T21:22:50.199133+0000","last_unstale":"2024-05-08T21:23:04.632605+0000","last_undegraded":"2024-05-08T21:23:04.632605+0000","last_fullsized":"2024-05-08T21:23:04.632605+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:40.200175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"34'9","reported_seq":26,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632780+0000","last_change":"2024-05-08T21:22:50.207042+0000","last_active":"2024-05-08T21:23:04.632780+0000","last_peered":"2024-05-08T21:23:04.632780+0000","last_clean":"2024-05-08T21:23:04.632780+0000","last_became_active":"2024-05-08T21:22:50.206554+0000","last_became_peered":"2024-05-08T21:22:50.206554+0000","last_unstale":"2024-05-08T21:23:04.632780+0000","last_undegraded":"2024-05-08T21:23:04.632780+0000","last_fullsized":"2024-05-08T21:23:04.632780+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:15:17.946198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"34'8","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.636567+0000","last_change":"2024-05-08T21:22:50.200284+0000","last_active":"2024-05-08T21:23:04.636567+0000","last_peered":"2024-05-08T21:23:04.636567+0000","last_clean":"2024-05-08T21:23:04.636567+0000","last_became_active":"2024-05-08T21:22:50.200004+0000","last_became_peered":"2024-05-08T21:22:50.200004+0000","last_unstale":"2024-05-08T21:23:04.636567+0000","last_undegraded":"2024-05-08T21:23:04.636567+0000","last_fullsized":"2024-05-08T21:23:04.636567+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:28.358029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"34'6","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638390+0000","last_change":"2024-05-08T21:22:50.208138+0000","last_active":"2024-05-08T21:23:04.638390+0000","last_peered":"2024-05-08T21:23:04.638390+0000","last_clean":"2024-05-08T21:23:04.638390+0000","last_became_active":"2024-05-08T21:22:50.207813+0000","last_became_peered":"2024-05-08T21:22:50.207813+0000","last_unstale":"2024-05-08T21:23:04.638390+0000","last_undegraded":"2024-05-08T21:23:04.638390+0000","last_fullsized":"2024-05-08T21:23:04.638390+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:07:17.470994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"34'2","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641204+0000","last_change":"2024-05-08T21:22:50.202937+0000","last_active":"2024-05-08T21:23:04.641204+0000","last_peered":"2024-05-08T21:23:04.641204+0000","last_clean":"2024-05-08T21:23:04.641204+0000","last_became_active":"2024-05-08T21:22:50.202633+0000","last_became_peered":"2024-05-08T21:22:50.202633+0000","last_unstale":"2024-05-08T21:23:04.641204+0000","last_undegraded":"2024-05-08T21:23:04.641204+0000","last_fullsized":"2024-05-08T21:23:04.641204+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:48:46.138832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"34'15","reported_seq":31,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635221+0000","last_change":"2024-05-08T21:22:50.204581+0000","last_active":"2024-05-08T21:23:04.635221+0000","last_peered":"2024-05-08T21:23:04.635221+0000","last_clean":"2024-05-08T21:23:04.635221+0000","last_became_active":"2024-05-08T21:22:50.204228+0000","last_became_peered":"2024-05-08T21:22:50.204228+0000","last_unstale":"2024-05-08T21:23:04.635221+0000","last_undegraded":"2024-05-08T21:23:04.635221+0000","last_fullsized":"2024-05-08T21:23:04.635221+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:43:00.051225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"34'6","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641363+0000","last_change":"2024-05-08T21:22:50.210823+0000","last_active":"2024-05-08T21:23:04.641363+0000","last_peered":"2024-05-08T21:23:04.641363+0000","last_clean":"2024-05-08T21:23:04.641363+0000","last_became_active":"2024-05-08T21:22:50.210517+0000","last_became_peered":"2024-05-08T21:22:50.210517+0000","last_unstale":"2024-05-08T21:23:04.641363+0000","last_undegraded":"2024-05-08T21:23:04.641363+0000","last_fullsized":"2024-05-08T21:23:04.641363+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:09:24.655227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"34'4","reported_seq":21,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638802+0000","last_change":"2024-05-08T21:22:50.201860+0000","last_active":"2024-05-08T21:23:04.638802+0000","last_peered":"2024-05-08T21:23:04.638802+0000","last_clean":"2024-05-08T21:23:04.638802+0000","last_became_active":"2024-05-08T21:22:50.201204+0000","last_became_peered":"2024-05-08T21:22:50.201204+0000","last_unstale":"2024-05-08T21:23:04.638802+0000","last_undegraded":"2024-05-08T21:23:04.638802+0000","last_fullsized":"2024-05-08T21:23:04.638802+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:23:48.163955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"34'7","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632564+0000","last_change":"2024-05-08T21:22:50.206666+0000","last_active":"2024-05-08T21:23:04.632564+0000","last_peered":"2024-05-08T21:23:04.632564+0000","last_clean":"2024-05-08T21:23:04.632564+0000","last_became_active":"2024-05-08T21:22:50.206283+0000","last_became_peered":"2024-05-08T21:22:50.206283+0000","last_unstale":"2024-05-08T21:23:04.632564+0000","last_undegraded":"2024-05-08T21:23:04.632564+0000","last_fullsized":"2024-05-08T21:23:04.632564+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:53:39.386458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"34'8","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635149+0000","last_change":"2024-05-08T21:22:50.205319+0000","last_active":"2024-05-08T21:23:04.635149+0000","last_peered":"2024-05-08T21:23:04.635149+0000","last_clean":"2024-05-08T21:23:04.635149+0000","last_became_active":"2024-05-08T21:22:50.205014+0000","last_became_peered":"2024-05-08T21:22:50.205014+0000","last_unstale":"2024-05-08T21:23:04.635149+0000","last_undegraded":"2024-05-08T21:23:04.635149+0000","last_fullsized":"2024-05-08T21:23:04.635149+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:10:48.376692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"34'12","reported_seq":28,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637536+0000","last_change":"2024-05-08T21:22:50.210271+0000","last_active":"2024-05-08T21:23:04.637536+0000","last_peered":"2024-05-08T21:23:04.637536+0000","last_clean":"2024-05-08T21:23:04.637536+0000","last_became_active":"2024-05-08T21:22:50.209998+0000","last_became_peered":"2024-05-08T21:22:50.209998+0000","last_unstale":"2024-05-08T21:23:04.637536+0000","last_undegraded":"2024-05-08T21:23:04.637536+0000","last_fullsized":"2024-05-08T21:23:04.637536+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:09.165560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"34'9","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635308+0000","last_change":"2024-05-08T21:22:50.211741+0000","last_active":"2024-05-08T21:23:04.635308+0000","last_peered":"2024-05-08T21:23:04.635308+0000","last_clean":"2024-05-08T21:23:04.635308+0000","last_became_active":"2024-05-08T21:22:50.211430+0000","last_became_peered":"2024-05-08T21:22:50.211430+0000","last_unstale":"2024-05-08T21:23:04.635308+0000","last_undegraded":"2024-05-08T21:23:04.635308+0000","last_fullsized":"2024-05-08T21:23:04.635308+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:23:48.163955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493199+0000","last_change":"2024-05-08T21:23:05.490190+0000","last_active":"2024-05-08T21:23:05.493199+0000","last_peered":"2024-05-08T21:23:05.493199+0000","last_clean":"2024-05-08T21:23:05.493199+0000","last_became_active":"2024-05-08T21:23:05.489780+0000","last_became_peered":"2024-05-08T21:23:05.489780+0000","last_unstale":"2024-05-08T21:23:05.493199+0000","last_undegraded":"2024-05-08T21:23:05.493199+0000","last_fullsized":"2024-05-08T21:23:05.493199+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:13:01.896572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"34'5","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637599+0000","last_change":"2024-05-08T21:22:50.201116+0000","last_active":"2024-05-08T21:23:04.637599+0000","last_peered":"2024-05-08T21:23:04.637599+0000","last_clean":"2024-05-08T21:23:04.637599+0000","last_became_active":"2024-05-08T21:22:50.200817+0000","last_became_peered":"2024-05-08T21:22:50.200817+0000","last_unstale":"2024-05-08T21:23:04.637599+0000","last_undegraded":"2024-05-08T21:23:04.637599+0000","last_fullsized":"2024-05-08T21:23:04.637599+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:09.165560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"37'8","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638089+0000","last_change":"2024-05-08T21:22:49.740529+0000","last_active":"2024-05-08T21:23:04.638089+0000","last_peered":"2024-05-08T21:23:04.638089+0000","last_clean":"2024-05-08T21:23:04.638089+0000","last_became_active":"2024-05-08T21:22:49.740247+0000","last_became_peered":"2024-05-08T21:22:49.740247+0000","last_unstale":"2024-05-08T21:23:04.638089+0000","last_undegraded":"2024-05-08T21:23:04.638089+0000","last_fullsized":"2024-05-08T21:23:04.638089+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:53:44.252529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.492276+0000","last_change":"2024-05-08T21:23:05.490147+0000","last_active":"2024-05-08T21:23:05.492276+0000","last_peered":"2024-05-08T21:23:05.492276+0000","last_clean":"2024-05-08T21:23:05.492276+0000","last_became_active":"2024-05-08T21:23:05.489698+0000","last_became_peered":"2024-05-08T21:23:05.489698+0000","last_unstale":"2024-05-08T21:23:05.492276+0000","last_undegraded":"2024-05-08T21:23:05.492276+0000","last_fullsized":"2024-05-08T21:23:05.492276+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:52:01.457943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"34'6","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638029+0000","last_change":"2024-05-08T21:22:50.208077+0000","last_active":"2024-05-08T21:23:04.638029+0000","last_peered":"2024-05-08T21:23:04.638029+0000","last_clean":"2024-05-08T21:23:04.638029+0000","last_became_active":"2024-05-08T21:22:50.207729+0000","last_became_peered":"2024-05-08T21:22:50.207729+0000","last_unstale":"2024-05-08T21:23:04.638029+0000","last_undegraded":"2024-05-08T21:23:04.638029+0000","last_fullsized":"2024-05-08T21:23:04.638029+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:40.200175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637700+0000","last_change":"2024-05-08T21:22:49.743083+0000","last_active":"2024-05-08T21:23:04.637700+0000","last_peered":"2024-05-08T21:23:04.637700+0000","last_clean":"2024-05-08T21:23:04.637700+0000","last_became_active":"2024-05-08T21:22:49.742790+0000","last_became_peered":"2024-05-08T21:22:49.742790+0000","last_unstale":"2024-05-08T21:23:04.637700+0000","last_undegraded":"2024-05-08T21:23:04.637700+0000","last_fullsized":"2024-05-08T21:23:04.637700+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:06:42.743465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.519256+0000","last_change":"2024-05-08T21:23:05.517988+0000","last_active":"2024-05-08T21:23:05.519256+0000","last_peered":"2024-05-08T21:23:05.519256+0000","last_clean":"2024-05-08T21:23:05.519256+0000","last_became_active":"2024-05-08T21:23:05.517596+0000","last_became_peered":"2024-05-08T21:23:05.517596+0000","last_unstale":"2024-05-08T21:23:05.519256+0000","last_undegraded":"2024-05-08T21:23:05.519256+0000","last_fullsized":"2024-05-08T21:23:05.519256+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:28:47.005653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"34'9","reported_seq":26,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632761+0000","last_change":"2024-05-08T21:22:50.207141+0000","last_active":"2024-05-08T21:23:04.632761+0000","last_peered":"2024-05-08T21:23:04.632761+0000","last_clean":"2024-05-08T21:23:04.632761+0000","last_became_active":"2024-05-08T21:22:50.206838+0000","last_became_peered":"2024-05-08T21:22:50.206838+0000","last_unstale":"2024-05-08T21:23:04.632761+0000","last_undegraded":"2024-05-08T21:23:04.632761+0000","last_fullsized":"2024-05-08T21:23:04.632761+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:16:58.048181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"36'3","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634757+0000","last_change":"2024-05-08T21:22:49.747636+0000","last_active":"2024-05-08T21:23:04.634757+0000","last_peered":"2024-05-08T21:23:04.634757+0000","last_clean":"2024-05-08T21:23:04.634757+0000","last_became_active":"2024-05-08T21:22:49.747345+0000","last_became_peered":"2024-05-08T21:22:49.747345+0000","last_unstale":"2024-05-08T21:23:04.634757+0000","last_undegraded":"2024-05-08T21:23:04.634757+0000","last_fullsized":"2024-05-08T21:23:04.634757+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:29:03.389411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.521132+0000","last_change":"2024-05-08T21:23:05.518010+0000","last_active":"2024-05-08T21:23:05.521132+0000","last_peered":"2024-05-08T21:23:05.521132+0000","last_clean":"2024-05-08T21:23:05.521132+0000","last_became_active":"2024-05-08T21:23:05.517594+0000","last_became_peered":"2024-05-08T21:23:05.517594+0000","last_unstale":"2024-05-08T21:23:05.521132+0000","last_undegraded":"2024-05-08T21:23:05.521132+0000","last_fullsized":"2024-05-08T21:23:05.521132+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:18:53.566436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"34'15","reported_seq":31,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641448+0000","last_change":"2024-05-08T21:22:50.209316+0000","last_active":"2024-05-08T21:23:04.641448+0000","last_peered":"2024-05-08T21:23:04.641448+0000","last_clean":"2024-05-08T21:23:04.641448+0000","last_became_active":"2024-05-08T21:22:50.209044+0000","last_became_peered":"2024-05-08T21:22:50.209044+0000","last_unstale":"2024-05-08T21:23:04.641448+0000","last_undegraded":"2024-05-08T21:23:04.641448+0000","last_fullsized":"2024-05-08T21:23:04.641448+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:25:52.293163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"36'3","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635395+0000","last_change":"2024-05-08T21:22:49.744152+0000","last_active":"2024-05-08T21:23:04.635395+0000","last_peered":"2024-05-08T21:23:04.635395+0000","last_clean":"2024-05-08T21:23:04.635395+0000","last_became_active":"2024-05-08T21:22:49.743867+0000","last_became_peered":"2024-05-08T21:22:49.743867+0000","last_unstale":"2024-05-08T21:23:04.635395+0000","last_undegraded":"2024-05-08T21:23:04.635395+0000","last_fullsized":"2024-05-08T21:23:04.635395+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:55:00.385106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.494808+0000","last_change":"2024-05-08T21:23:05.490761+0000","last_active":"2024-05-08T21:23:05.494808+0000","last_peered":"2024-05-08T21:23:05.494808+0000","last_clean":"2024-05-08T21:23:05.494808+0000","last_became_active":"2024-05-08T21:23:05.490093+0000","last_became_peered":"2024-05-08T21:23:05.490093+0000","last_unstale":"2024-05-08T21:23:05.494808+0000","last_undegraded":"2024-05-08T21:23:05.494808+0000","last_fullsized":"2024-05-08T21:23:05.494808+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:12:43.572456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"34'7","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635214+0000","last_change":"2024-05-08T21:22:50.204033+0000","last_active":"2024-05-08T21:23:04.635214+0000","last_peered":"2024-05-08T21:23:04.635214+0000","last_clean":"2024-05-08T21:23:04.635214+0000","last_became_active":"2024-05-08T21:22:50.203766+0000","last_became_peered":"2024-05-08T21:22:50.203766+0000","last_unstale":"2024-05-08T21:23:04.635214+0000","last_undegraded":"2024-05-08T21:23:04.635214+0000","last_fullsized":"2024-05-08T21:23:04.635214+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:48:46.138832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":19,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632687+0000","last_change":"2024-05-08T21:22:49.740399+0000","last_active":"2024-05-08T21:23:04.632687+0000","last_peered":"2024-05-08T21:23:04.632687+0000","last_clean":"2024-05-08T21:23:04.632687+0000","last_became_active":"2024-05-08T21:22:49.740125+0000","last_became_peered":"2024-05-08T21:22:49.740125+0000","last_unstale":"2024-05-08T21:23:04.632687+0000","last_undegraded":"2024-05-08T21:23:04.632687+0000","last_fullsized":"2024-05-08T21:23:04.632687+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:01:42.646117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493235+0000","last_change":"2024-05-08T21:23:05.489369+0000","last_active":"2024-05-08T21:23:05.493235+0000","last_peered":"2024-05-08T21:23:05.493235+0000","last_clean":"2024-05-08T21:23:05.493235+0000","last_became_active":"2024-05-08T21:23:05.489036+0000","last_became_peered":"2024-05-08T21:23:05.489036+0000","last_unstale":"2024-05-08T21:23:05.493235+0000","last_undegraded":"2024-05-08T21:23:05.493235+0000","last_fullsized":"2024-05-08T21:23:05.493235+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:28:20.539506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"34'8","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638573+0000","last_change":"2024-05-08T21:22:50.205784+0000","last_active":"2024-05-08T21:23:04.638573+0000","last_peered":"2024-05-08T21:23:04.638573+0000","last_clean":"2024-05-08T21:23:04.638573+0000","last_became_active":"2024-05-08T21:22:50.205525+0000","last_became_peered":"2024-05-08T21:22:50.205525+0000","last_unstale":"2024-05-08T21:23:04.638573+0000","last_undegraded":"2024-05-08T21:23:04.638573+0000","last_fullsized":"2024-05-08T21:23:04.638573+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:23.732223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":19,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632771+0000","last_change":"2024-05-08T21:22:49.739604+0000","last_active":"2024-05-08T21:23:04.632771+0000","last_peered":"2024-05-08T21:23:04.632771+0000","last_clean":"2024-05-08T21:23:04.632771+0000","last_became_active":"2024-05-08T21:22:49.739343+0000","last_became_peered":"2024-05-08T21:22:49.739343+0000","last_unstale":"2024-05-08T21:23:04.632771+0000","last_undegraded":"2024-05-08T21:23:04.632771+0000","last_fullsized":"2024-05-08T21:23:04.632771+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:55:43.121728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493154+0000","last_change":"2024-05-08T21:23:05.490982+0000","last_active":"2024-05-08T21:23:05.493154+0000","last_peered":"2024-05-08T21:23:05.493154+0000","last_clean":"2024-05-08T21:23:05.493154+0000","last_became_active":"2024-05-08T21:23:05.490662+0000","last_became_peered":"2024-05-08T21:23:05.490662+0000","last_unstale":"2024-05-08T21:23:05.493154+0000","last_undegraded":"2024-05-08T21:23:05.493154+0000","last_fullsized":"2024-05-08T21:23:05.493154+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:11:23.240353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"34'8","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632420+0000","last_change":"2024-05-08T21:22:50.206820+0000","last_active":"2024-05-08T21:23:04.632420+0000","last_peered":"2024-05-08T21:23:04.632420+0000","last_clean":"2024-05-08T21:23:04.632420+0000","last_became_active":"2024-05-08T21:22:50.206477+0000","last_became_peered":"2024-05-08T21:22:50.206477+0000","last_unstale":"2024-05-08T21:23:04.632420+0000","last_undegraded":"2024-05-08T21:23:04.632420+0000","last_fullsized":"2024-05-08T21:23:04.632420+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:28.358029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634004+0000","last_change":"2024-05-08T21:22:49.737447+0000","last_active":"2024-05-08T21:23:04.634004+0000","last_peered":"2024-05-08T21:23:04.634004+0000","last_clean":"2024-05-08T21:23:04.634004+0000","last_became_active":"2024-05-08T21:22:49.736975+0000","last_became_peered":"2024-05-08T21:22:49.736975+0000","last_unstale":"2024-05-08T21:23:04.634004+0000","last_undegraded":"2024-05-08T21:23:04.634004+0000","last_fullsized":"2024-05-08T21:23:04.634004+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:39.192702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.494913+0000","last_change":"2024-05-08T21:23:05.489998+0000","last_active":"2024-05-08T21:23:05.494913+0000","last_peered":"2024-05-08T21:23:05.494913+0000","last_clean":"2024-05-08T21:23:05.494913+0000","last_became_active":"2024-05-08T21:23:05.489630+0000","last_became_peered":"2024-05-08T21:23:05.489630+0000","last_unstale":"2024-05-08T21:23:05.494913+0000","last_undegraded":"2024-05-08T21:23:05.494913+0000","last_fullsized":"2024-05-08T21:23:05.494913+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:20:48.767679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"34'10","reported_seq":26,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641137+0000","last_change":"2024-05-08T21:22:50.205263+0000","last_active":"2024-05-08T21:23:04.641137+0000","last_peered":"2024-05-08T21:23:04.641137+0000","last_clean":"2024-05-08T21:23:04.641137+0000","last_became_active":"2024-05-08T21:22:50.204963+0000","last_became_peered":"2024-05-08T21:22:50.204963+0000","last_unstale":"2024-05-08T21:23:04.641137+0000","last_undegraded":"2024-05-08T21:23:04.641137+0000","last_fullsized":"2024-05-08T21:23:04.641137+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:44:18.055232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632382+0000","last_change":"2024-05-08T21:22:49.733981+0000","last_active":"2024-05-08T21:23:04.632382+0000","last_peered":"2024-05-08T21:23:04.632382+0000","last_clean":"2024-05-08T21:23:04.632382+0000","last_became_active":"2024-05-08T21:22:49.733714+0000","last_became_peered":"2024-05-08T21:22:49.733714+0000","last_unstale":"2024-05-08T21:23:04.632382+0000","last_undegraded":"2024-05-08T21:23:04.632382+0000","last_fullsized":"2024-05-08T21:23:04.632382+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:47.931452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.519462+0000","last_change":"2024-05-08T21:23:05.518321+0000","last_active":"2024-05-08T21:23:05.519462+0000","last_peered":"2024-05-08T21:23:05.519462+0000","last_clean":"2024-05-08T21:23:05.519462+0000","last_became_active":"2024-05-08T21:23:05.518059+0000","last_became_peered":"2024-05-08T21:23:05.518059+0000","last_unstale":"2024-05-08T21:23:05.519462+0000","last_undegraded":"2024-05-08T21:23:05.519462+0000","last_fullsized":"2024-05-08T21:23:05.519462+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:52:59.903852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"34'10","reported_seq":27,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.639115+0000","last_change":"2024-05-08T21:22:50.209646+0000","last_active":"2024-05-08T21:23:04.639115+0000","last_peered":"2024-05-08T21:23:04.639115+0000","last_clean":"2024-05-08T21:23:04.639115+0000","last_became_active":"2024-05-08T21:22:50.209335+0000","last_became_peered":"2024-05-08T21:22:50.209335+0000","last_unstale":"2024-05-08T21:23:04.639115+0000","last_undegraded":"2024-05-08T21:23:04.639115+0000","last_fullsized":"2024-05-08T21:23:04.639115+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:21:31.919852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.636375+0000","last_change":"2024-05-08T21:22:49.738039+0000","last_active":"2024-05-08T21:23:04.636375+0000","last_peered":"2024-05-08T21:23:04.636375+0000","last_clean":"2024-05-08T21:23:04.636375+0000","last_became_active":"2024-05-08T21:22:49.737785+0000","last_became_peered":"2024-05-08T21:22:49.737785+0000","last_unstale":"2024-05-08T21:23:04.636375+0000","last_undegraded":"2024-05-08T21:23:04.636375+0000","last_fullsized":"2024-05-08T21:23:04.636375+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:27.350557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.492900+0000","last_change":"2024-05-08T21:23:05.490857+0000","last_active":"2024-05-08T21:23:05.492900+0000","last_peered":"2024-05-08T21:23:05.492900+0000","last_clean":"2024-05-08T21:23:05.492900+0000","last_became_active":"2024-05-08T21:23:05.490603+0000","last_became_peered":"2024-05-08T21:23:05.490603+0000","last_unstale":"2024-05-08T21:23:05.492900+0000","last_undegraded":"2024-05-08T21:23:05.492900+0000","last_fullsized":"2024-05-08T21:23:05.492900+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:05:40.853105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"34'9","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.636726+0000","last_change":"2024-05-08T21:22:50.206695+0000","last_active":"2024-05-08T21:23:04.636726+0000","last_peered":"2024-05-08T21:23:04.636726+0000","last_clean":"2024-05-08T21:23:04.636726+0000","last_became_active":"2024-05-08T21:22:50.206398+0000","last_became_peered":"2024-05-08T21:22:50.206398+0000","last_unstale":"2024-05-08T21:23:04.636726+0000","last_undegraded":"2024-05-08T21:23:04.636726+0000","last_fullsized":"2024-05-08T21:23:04.636726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:07:17.470994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637794+0000","last_change":"2024-05-08T21:22:49.740160+0000","last_active":"2024-05-08T21:23:04.637794+0000","last_peered":"2024-05-08T21:23:04.637794+0000","last_clean":"2024-05-08T21:23:04.637794+0000","last_became_active":"2024-05-08T21:22:49.739818+0000","last_became_peered":"2024-05-08T21:22:49.739818+0000","last_unstale":"2024-05-08T21:23:04.637794+0000","last_undegraded":"2024-05-08T21:23:04.637794+0000","last_fullsized":"2024-05-08T21:23:04.637794+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:39:42.856227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.495393+0000","last_change":"2024-05-08T21:23:05.490240+0000","last_active":"2024-05-08T21:23:05.495393+0000","last_peered":"2024-05-08T21:23:05.495393+0000","last_clean":"2024-05-08T21:23:05.495393+0000","last_became_active":"2024-05-08T21:23:05.489936+0000","last_became_peered":"2024-05-08T21:23:05.489936+0000","last_unstale":"2024-05-08T21:23:05.495393+0000","last_undegraded":"2024-05-08T21:23:05.495393+0000","last_fullsized":"2024-05-08T21:23:05.495393+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:03:21.459659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"34'11","reported_seq":27,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634791+0000","last_change":"2024-05-08T21:22:50.204717+0000","last_active":"2024-05-08T21:23:04.634791+0000","last_peered":"2024-05-08T21:23:04.634791+0000","last_clean":"2024-05-08T21:23:04.634791+0000","last_became_active":"2024-05-08T21:22:50.204404+0000","last_became_peered":"2024-05-08T21:22:50.204404+0000","last_unstale":"2024-05-08T21:23:04.634791+0000","last_undegraded":"2024-05-08T21:23:04.634791+0000","last_fullsized":"2024-05-08T21:23:04.634791+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:27:40.074356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"36'5","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638992+0000","last_change":"2024-05-08T21:22:49.736860+0000","last_active":"2024-05-08T21:23:04.638992+0000","last_peered":"2024-05-08T21:23:04.638992+0000","last_clean":"2024-05-08T21:23:04.638992+0000","last_became_active":"2024-05-08T21:22:49.736497+0000","last_became_peered":"2024-05-08T21:22:49.736497+0000","last_unstale":"2024-05-08T21:23:04.638992+0000","last_undegraded":"2024-05-08T21:23:04.638992+0000","last_fullsized":"2024-05-08T21:23:04.638992+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:15:16.938725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.488443+0000","last_change":"2024-05-08T21:23:05.487624+0000","last_active":"2024-05-08T21:23:05.488443+0000","last_peered":"2024-05-08T21:23:05.488443+0000","last_clean":"2024-05-08T21:23:05.488443+0000","last_became_active":"2024-05-08T21:23:05.487309+0000","last_became_peered":"2024-05-08T21:23:05.487309+0000","last_unstale":"2024-05-08T21:23:05.488443+0000","last_undegraded":"2024-05-08T21:23:05.488443+0000","last_fullsized":"2024-05-08T21:23:05.488443+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:11:55.531943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"34'6","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638796+0000","last_change":"2024-05-08T21:22:50.205233+0000","last_active":"2024-05-08T21:23:04.638796+0000","last_peered":"2024-05-08T21:23:04.638796+0000","last_clean":"2024-05-08T21:23:04.638796+0000","last_became_active":"2024-05-08T21:22:50.204889+0000","last_became_peered":"2024-05-08T21:22:50.204889+0000","last_unstale":"2024-05-08T21:23:04.638796+0000","last_undegraded":"2024-05-08T21:23:04.638796+0000","last_fullsized":"2024-05-08T21:23:04.638796+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:38:13.012930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"36'22","reported_seq":39,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634884+0000","last_change":"2024-05-08T21:22:49.743647+0000","last_active":"2024-05-08T21:23:04.634884+0000","last_peered":"2024-05-08T21:23:04.634884+0000","last_clean":"2024-05-08T21:23:04.634884+0000","last_became_active":"2024-05-08T21:22:49.743348+0000","last_became_peered":"2024-05-08T21:22:49.743348+0000","last_unstale":"2024-05-08T21:23:04.634884+0000","last_undegraded":"2024-05-08T21:23:04.634884+0000","last_fullsized":"2024-05-08T21:23:04.634884+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:06:09.036243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493148+0000","last_change":"2024-05-08T21:23:05.490136+0000","last_active":"2024-05-08T21:23:05.493148+0000","last_peered":"2024-05-08T21:23:05.493148+0000","last_clean":"2024-05-08T21:23:05.493148+0000","last_became_active":"2024-05-08T21:23:05.489860+0000","last_became_peered":"2024-05-08T21:23:05.489860+0000","last_unstale":"2024-05-08T21:23:05.493148+0000","last_undegraded":"2024-05-08T21:23:05.493148+0000","last_fullsized":"2024-05-08T21:23:05.493148+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:23.962061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"34'10","reported_seq":26,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.640738+0000","last_change":"2024-05-08T21:22:50.202647+0000","last_active":"2024-05-08T21:23:04.640738+0000","last_peered":"2024-05-08T21:23:04.640738+0000","last_clean":"2024-05-08T21:23:04.640738+0000","last_became_active":"2024-05-08T21:22:50.202144+0000","last_became_peered":"2024-05-08T21:22:50.202144+0000","last_unstale":"2024-05-08T21:23:04.640738+0000","last_undegraded":"2024-05-08T21:23:04.640738+0000","last_fullsized":"2024-05-08T21:23:04.640738+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:10:48.376692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"37'5","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632726+0000","last_change":"2024-05-08T21:22:49.735484+0000","last_active":"2024-05-08T21:23:04.632726+0000","last_peered":"2024-05-08T21:23:04.632726+0000","last_clean":"2024-05-08T21:23:04.632726+0000","last_became_active":"2024-05-08T21:22:49.735232+0000","last_became_peered":"2024-05-08T21:22:49.735232+0000","last_unstale":"2024-05-08T21:23:04.632726+0000","last_undegraded":"2024-05-08T21:23:04.632726+0000","last_fullsized":"2024-05-08T21:23:04.632726+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:13:03.055943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493638+0000","last_change":"2024-05-08T21:23:05.491656+0000","last_active":"2024-05-08T21:23:05.493638+0000","last_peered":"2024-05-08T21:23:05.493638+0000","last_clean":"2024-05-08T21:23:05.493638+0000","last_became_active":"2024-05-08T21:23:05.491419+0000","last_became_peered":"2024-05-08T21:23:05.491419+0000","last_unstale":"2024-05-08T21:23:05.493638+0000","last_undegraded":"2024-05-08T21:23:05.493638+0000","last_fullsized":"2024-05-08T21:23:05.493638+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:58:36.326421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"34'15","reported_seq":31,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637478+0000","last_change":"2024-05-08T21:22:50.204754+0000","last_active":"2024-05-08T21:23:04.637478+0000","last_peered":"2024-05-08T21:23:04.637478+0000","last_clean":"2024-05-08T21:23:04.637478+0000","last_became_active":"2024-05-08T21:22:50.204473+0000","last_became_peered":"2024-05-08T21:22:50.204473+0000","last_unstale":"2024-05-08T21:23:04.637478+0000","last_undegraded":"2024-05-08T21:23:04.637478+0000","last_fullsized":"2024-05-08T21:23:04.637478+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:14.396616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.640966+0000","last_change":"2024-05-08T21:22:49.741850+0000","last_active":"2024-05-08T21:23:04.640966+0000","last_peered":"2024-05-08T21:23:04.640966+0000","last_clean":"2024-05-08T21:23:04.640966+0000","last_became_active":"2024-05-08T21:22:49.741527+0000","last_became_peered":"2024-05-08T21:22:49.741527+0000","last_unstale":"2024-05-08T21:23:04.640966+0000","last_undegraded":"2024-05-08T21:23:04.640966+0000","last_fullsized":"2024-05-08T21:23:04.640966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:29:03.389411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493135+0000","last_change":"2024-05-08T21:23:05.490366+0000","last_active":"2024-05-08T21:23:05.493135+0000","last_peered":"2024-05-08T21:23:05.493135+0000","last_clean":"2024-05-08T21:23:05.493135+0000","last_became_active":"2024-05-08T21:23:05.489537+0000","last_became_peered":"2024-05-08T21:23:05.489537+0000","last_unstale":"2024-05-08T21:23:05.493135+0000","last_undegraded":"2024-05-08T21:23:05.493135+0000","last_fullsized":"2024-05-08T21:23:05.493135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:45:57.976239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"34'6","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635140+0000","last_change":"2024-05-08T21:22:50.211613+0000","last_active":"2024-05-08T21:23:04.635140+0000","last_peered":"2024-05-08T21:23:04.635140+0000","last_clean":"2024-05-08T21:23:04.635140+0000","last_became_active":"2024-05-08T21:22:50.211335+0000","last_became_peered":"2024-05-08T21:22:50.211335+0000","last_unstale":"2024-05-08T21:23:04.635140+0000","last_undegraded":"2024-05-08T21:23:04.635140+0000","last_fullsized":"2024-05-08T21:23:04.635140+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:43:00.051225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634106+0000","last_change":"2024-05-08T21:22:49.737257+0000","last_active":"2024-05-08T21:23:04.634106+0000","last_peered":"2024-05-08T21:23:04.634106+0000","last_clean":"2024-05-08T21:23:04.634106+0000","last_became_active":"2024-05-08T21:22:49.736591+0000","last_became_peered":"2024-05-08T21:22:49.736591+0000","last_unstale":"2024-05-08T21:23:04.634106+0000","last_undegraded":"2024-05-08T21:23:04.634106+0000","last_fullsized":"2024-05-08T21:23:04.634106+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:08.158087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.490343+0000","last_change":"2024-05-08T21:23:05.486693+0000","last_active":"2024-05-08T21:23:05.490343+0000","last_peered":"2024-05-08T21:23:05.490343+0000","last_clean":"2024-05-08T21:23:05.490343+0000","last_became_active":"2024-05-08T21:23:05.486375+0000","last_became_peered":"2024-05-08T21:23:05.486375+0000","last_unstale":"2024-05-08T21:23:05.490343+0000","last_undegraded":"2024-05-08T21:23:05.490343+0000","last_fullsized":"2024-05-08T21:23:05.490343+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:28:15.819527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"34'6","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637407+0000","last_change":"2024-05-08T21:22:50.209206+0000","last_active":"2024-05-08T21:23:04.637407+0000","last_peered":"2024-05-08T21:23:04.637407+0000","last_clean":"2024-05-08T21:23:04.637407+0000","last_became_active":"2024-05-08T21:22:50.208950+0000","last_became_peered":"2024-05-08T21:22:50.208950+0000","last_unstale":"2024-05-08T21:23:04.637407+0000","last_undegraded":"2024-05-08T21:23:04.637407+0000","last_fullsized":"2024-05-08T21:23:04.637407+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:15:17.946198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"36'15","reported_seq":32,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.633412+0000","last_change":"2024-05-08T21:22:49.743861+0000","last_active":"2024-05-08T21:23:04.633412+0000","last_peered":"2024-05-08T21:23:04.633412+0000","last_clean":"2024-05-08T21:23:04.633412+0000","last_became_active":"2024-05-08T21:22:49.743575+0000","last_became_peered":"2024-05-08T21:22:49.743575+0000","last_unstale":"2024-05-08T21:23:04.633412+0000","last_undegraded":"2024-05-08T21:23:04.633412+0000","last_fullsized":"2024-05-08T21:23:04.633412+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:51.977551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.495313+0000","last_change":"2024-05-08T21:23:05.491466+0000","last_active":"2024-05-08T21:23:05.495313+0000","last_peered":"2024-05-08T21:23:05.495313+0000","last_clean":"2024-05-08T21:23:05.495313+0000","last_became_active":"2024-05-08T21:23:05.491195+0000","last_became_peered":"2024-05-08T21:23:05.491195+0000","last_unstale":"2024-05-08T21:23:05.495313+0000","last_undegraded":"2024-05-08T21:23:05.495313+0000","last_fullsized":"2024-05-08T21:23:05.495313+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:54:44.860139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"34'5","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.633100+0000","last_change":"2024-05-08T21:22:50.203855+0000","last_active":"2024-05-08T21:23:04.633100+0000","last_peered":"2024-05-08T21:23:04.633100+0000","last_clean":"2024-05-08T21:23:04.633100+0000","last_became_active":"2024-05-08T21:22:50.203557+0000","last_became_peered":"2024-05-08T21:22:50.203557+0000","last_unstale":"2024-05-08T21:23:04.633100+0000","last_undegraded":"2024-05-08T21:23:04.633100+0000","last_fullsized":"2024-05-08T21:23:04.633100+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:09.165560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"36'2","reported_seq":19,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641397+0000","last_change":"2024-05-08T21:22:49.742208+0000","last_active":"2024-05-08T21:23:04.641397+0000","last_peered":"2024-05-08T21:23:04.641397+0000","last_clean":"2024-05-08T21:23:04.641397+0000","last_became_active":"2024-05-08T21:22:49.741894+0000","last_became_peered":"2024-05-08T21:22:49.741894+0000","last_unstale":"2024-05-08T21:23:04.641397+0000","last_undegraded":"2024-05-08T21:23:04.641397+0000","last_fullsized":"2024-05-08T21:23:04.641397+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:59:36.798032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.519266+0000","last_change":"2024-05-08T21:23:05.518243+0000","last_active":"2024-05-08T21:23:05.519266+0000","last_peered":"2024-05-08T21:23:05.519266+0000","last_clean":"2024-05-08T21:23:05.519266+0000","last_became_active":"2024-05-08T21:23:05.517600+0000","last_became_peered":"2024-05-08T21:23:05.517600+0000","last_unstale":"2024-05-08T21:23:05.519266+0000","last_undegraded":"2024-05-08T21:23:05.519266+0000","last_fullsized":"2024-05-08T21:23:05.519266+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:12:43.572456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"34'9","reported_seq":26,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.639389+0000","last_change":"2024-05-08T21:22:50.208102+0000","last_active":"2024-05-08T21:23:04.639389+0000","last_peered":"2024-05-08T21:23:04.639389+0000","last_clean":"2024-05-08T21:23:04.639389+0000","last_became_active":"2024-05-08T21:22:50.207765+0000","last_became_peered":"2024-05-08T21:22:50.207765+0000","last_unstale":"2024-05-08T21:23:04.639389+0000","last_undegraded":"2024-05-08T21:23:04.639389+0000","last_fullsized":"2024-05-08T21:23:04.639389+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:14.396616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"36'3","reported_seq":21,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.633073+0000","last_change":"2024-05-08T21:22:49.734619+0000","last_active":"2024-05-08T21:23:04.633073+0000","last_peered":"2024-05-08T21:23:04.633073+0000","last_clean":"2024-05-08T21:23:04.633073+0000","last_became_active":"2024-05-08T21:22:49.734354+0000","last_became_peered":"2024-05-08T21:22:49.734354+0000","last_unstale":"2024-05-08T21:23:04.633073+0000","last_undegraded":"2024-05-08T21:23:04.633073+0000","last_fullsized":"2024-05-08T21:23:04.633073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:23:35.220950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.492794+0000","last_change":"2024-05-08T21:23:05.491235+0000","last_active":"2024-05-08T21:23:05.492794+0000","last_peered":"2024-05-08T21:23:05.492794+0000","last_clean":"2024-05-08T21:23:05.492794+0000","last_became_active":"2024-05-08T21:23:05.490957+0000","last_became_peered":"2024-05-08T21:23:05.490957+0000","last_unstale":"2024-05-08T21:23:05.492794+0000","last_undegraded":"2024-05-08T21:23:05.492794+0000","last_fullsized":"2024-05-08T21:23:05.492794+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:38.632583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"34'6","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641077+0000","last_change":"2024-05-08T21:22:50.210640+0000","last_active":"2024-05-08T21:23:04.641077+0000","last_peered":"2024-05-08T21:23:04.641077+0000","last_clean":"2024-05-08T21:23:04.641077+0000","last_became_active":"2024-05-08T21:22:50.210283+0000","last_became_peered":"2024-05-08T21:22:50.210283+0000","last_unstale":"2024-05-08T21:23:04.641077+0000","last_undegraded":"2024-05-08T21:23:04.641077+0000","last_fullsized":"2024-05-08T21:23:04.641077+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:58:46.704338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"36'2","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632967+0000","last_change":"2024-05-08T21:22:49.737099+0000","last_active":"2024-05-08T21:23:04.632967+0000","last_peered":"2024-05-08T21:23:04.632967+0000","last_clean":"2024-05-08T21:23:04.632967+0000","last_became_active":"2024-05-08T21:22:49.736839+0000","last_became_peered":"2024-05-08T21:22:49.736839+0000","last_unstale":"2024-05-08T21:23:04.632967+0000","last_undegraded":"2024-05-08T21:23:04.632967+0000","last_fullsized":"2024-05-08T21:23:04.632967+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:05:26.886351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.498088+0000","last_change":"2024-05-08T21:23:05.495475+0000","last_active":"2024-05-08T21:23:05.498088+0000","last_peered":"2024-05-08T21:23:05.498088+0000","last_clean":"2024-05-08T21:23:05.498088+0000","last_became_active":"2024-05-08T21:23:05.495181+0000","last_became_peered":"2024-05-08T21:23:05.495181+0000","last_unstale":"2024-05-08T21:23:05.498088+0000","last_undegraded":"2024-05-08T21:23:05.498088+0000","last_fullsized":"2024-05-08T21:23:05.498088+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:34:11.128072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"34'7","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634805+0000","last_change":"2024-05-08T21:22:50.206786+0000","last_active":"2024-05-08T21:23:04.634805+0000","last_peered":"2024-05-08T21:23:04.634805+0000","last_clean":"2024-05-08T21:23:04.634805+0000","last_became_active":"2024-05-08T21:22:50.206484+0000","last_became_peered":"2024-05-08T21:22:50.206484+0000","last_unstale":"2024-05-08T21:23:04.634805+0000","last_undegraded":"2024-05-08T21:23:04.634805+0000","last_fullsized":"2024-05-08T21:23:04.634805+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:36:44.630301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632995+0000","last_change":"2024-05-08T21:22:49.740491+0000","last_active":"2024-05-08T21:23:04.632995+0000","last_peered":"2024-05-08T21:23:04.632995+0000","last_clean":"2024-05-08T21:23:04.632995+0000","last_became_active":"2024-05-08T21:22:49.740166+0000","last_became_peered":"2024-05-08T21:22:49.740166+0000","last_unstale":"2024-05-08T21:23:04.632995+0000","last_undegraded":"2024-05-08T21:23:04.632995+0000","last_fullsized":"2024-05-08T21:23:04.632995+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:27:16.794172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493593+0000","last_change":"2024-05-08T21:23:05.490329+0000","last_active":"2024-05-08T21:23:05.493593+0000","last_peered":"2024-05-08T21:23:05.493593+0000","last_clean":"2024-05-08T21:23:05.493593+0000","last_became_active":"2024-05-08T21:23:05.490030+0000","last_became_peered":"2024-05-08T21:23:05.490030+0000","last_unstale":"2024-05-08T21:23:05.493593+0000","last_undegraded":"2024-05-08T21:23:05.493593+0000","last_fullsized":"2024-05-08T21:23:05.493593+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:05:41.899542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"34'5","reported_seq":21,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634676+0000","last_change":"2024-05-08T21:22:50.207827+0000","last_active":"2024-05-08T21:23:04.634676+0000","last_peered":"2024-05-08T21:23:04.634676+0000","last_clean":"2024-05-08T21:23:04.634676+0000","last_became_active":"2024-05-08T21:22:50.207560+0000","last_became_peered":"2024-05-08T21:22:50.207560+0000","last_unstale":"2024-05-08T21:23:04.634676+0000","last_undegraded":"2024-05-08T21:23:04.634676+0000","last_fullsized":"2024-05-08T21:23:04.634676+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:29:04.396884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"37'4","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637281+0000","last_change":"2024-05-08T21:22:49.743647+0000","last_active":"2024-05-08T21:23:04.637281+0000","last_peered":"2024-05-08T21:23:04.637281+0000","last_clean":"2024-05-08T21:23:04.637281+0000","last_became_active":"2024-05-08T21:22:49.743349+0000","last_became_peered":"2024-05-08T21:22:49.743349+0000","last_unstale":"2024-05-08T21:23:04.637281+0000","last_undegraded":"2024-05-08T21:23:04.637281+0000","last_fullsized":"2024-05-08T21:23:04.637281+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:53:44.252529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.489385+0000","last_change":"2024-05-08T21:23:05.486117+0000","last_active":"2024-05-08T21:23:05.489385+0000","last_peered":"2024-05-08T21:23:05.489385+0000","last_clean":"2024-05-08T21:23:05.489385+0000","last_became_active":"2024-05-08T21:23:05.485848+0000","last_became_peered":"2024-05-08T21:23:05.485848+0000","last_unstale":"2024-05-08T21:23:05.489385+0000","last_undegraded":"2024-05-08T21:23:05.489385+0000","last_fullsized":"2024-05-08T21:23:05.489385+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:30:04.803867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"34'8","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634374+0000","last_change":"2024-05-08T21:22:50.205819+0000","last_active":"2024-05-08T21:23:04.634374+0000","last_peered":"2024-05-08T21:23:04.634374+0000","last_clean":"2024-05-08T21:23:04.634374+0000","last_became_active":"2024-05-08T21:22:50.205239+0000","last_became_peered":"2024-05-08T21:22:50.205239+0000","last_unstale":"2024-05-08T21:23:04.634374+0000","last_undegraded":"2024-05-08T21:23:04.634374+0000","last_fullsized":"2024-05-08T21:23:04.634374+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:52.985024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637665+0000","last_change":"2024-05-08T21:22:49.743818+0000","last_active":"2024-05-08T21:23:04.637665+0000","last_peered":"2024-05-08T21:23:04.637665+0000","last_clean":"2024-05-08T21:23:04.637665+0000","last_became_active":"2024-05-08T21:22:49.743469+0000","last_became_peered":"2024-05-08T21:22:49.743469+0000","last_unstale":"2024-05-08T21:23:04.637665+0000","last_undegraded":"2024-05-08T21:23:04.637665+0000","last_fullsized":"2024-05-08T21:23:04.637665+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:06:42.743465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493381+0000","last_change":"2024-05-08T21:23:05.491490+0000","last_active":"2024-05-08T21:23:05.493381+0000","last_peered":"2024-05-08T21:23:05.493381+0000","last_clean":"2024-05-08T21:23:05.493381+0000","last_became_active":"2024-05-08T21:23:05.491231+0000","last_became_peered":"2024-05-08T21:23:05.491231+0000","last_unstale":"2024-05-08T21:23:05.493381+0000","last_undegraded":"2024-05-08T21:23:05.493381+0000","last_fullsized":"2024-05-08T21:23:05.493381+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:28:06.867361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"34'9","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634265+0000","last_change":"2024-05-08T21:22:50.203236+0000","last_active":"2024-05-08T21:23:04.634265+0000","last_peered":"2024-05-08T21:23:04.634265+0000","last_clean":"2024-05-08T21:23:04.634265+0000","last_became_active":"2024-05-08T21:22:50.201914+0000","last_became_peered":"2024-05-08T21:22:50.201914+0000","last_unstale":"2024-05-08T21:23:04.634265+0000","last_undegraded":"2024-05-08T21:23:04.634265+0000","last_fullsized":"2024-05-08T21:23:04.634265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:06:10.043716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632487+0000","last_change":"2024-05-08T21:22:49.739798+0000","last_active":"2024-05-08T21:23:04.632487+0000","last_peered":"2024-05-08T21:23:04.632487+0000","last_clean":"2024-05-08T21:23:04.632487+0000","last_became_active":"2024-05-08T21:22:49.739520+0000","last_became_peered":"2024-05-08T21:22:49.739520+0000","last_unstale":"2024-05-08T21:23:04.632487+0000","last_undegraded":"2024-05-08T21:23:04.632487+0000","last_fullsized":"2024-05-08T21:23:04.632487+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:47:50.170209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.518668+0000","last_change":"2024-05-08T21:23:05.517869+0000","last_active":"2024-05-08T21:23:05.518668+0000","last_peered":"2024-05-08T21:23:05.518668+0000","last_clean":"2024-05-08T21:23:05.518668+0000","last_became_active":"2024-05-08T21:23:05.517598+0000","last_became_peered":"2024-05-08T21:23:05.517598+0000","last_unstale":"2024-05-08T21:23:05.518668+0000","last_undegraded":"2024-05-08T21:23:05.518668+0000","last_fullsized":"2024-05-08T21:23:05.518668+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:52:59.903852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"34'2","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635163+0000","last_change":"2024-05-08T21:22:50.204561+0000","last_active":"2024-05-08T21:23:04.635163+0000","last_peered":"2024-05-08T21:23:04.635163+0000","last_clean":"2024-05-08T21:23:04.635163+0000","last_became_active":"2024-05-08T21:22:50.204296+0000","last_became_peered":"2024-05-08T21:22:50.204296+0000","last_unstale":"2024-05-08T21:23:04.635163+0000","last_undegraded":"2024-05-08T21:23:04.635163+0000","last_fullsized":"2024-05-08T21:23:04.635163+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:25:52.293163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.638387+0000","last_change":"2024-05-08T21:22:49.737100+0000","last_active":"2024-05-08T21:23:04.638387+0000","last_peered":"2024-05-08T21:23:04.638387+0000","last_clean":"2024-05-08T21:23:04.638387+0000","last_became_active":"2024-05-08T21:22:49.736715+0000","last_became_peered":"2024-05-08T21:22:49.736715+0000","last_unstale":"2024-05-08T21:23:04.638387+0000","last_undegraded":"2024-05-08T21:23:04.638387+0000","last_fullsized":"2024-05-08T21:23:04.638387+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:52:46.503073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.519426+0000","last_change":"2024-05-08T21:23:05.518020+0000","last_active":"2024-05-08T21:23:05.519426+0000","last_peered":"2024-05-08T21:23:05.519426+0000","last_clean":"2024-05-08T21:23:05.519426+0000","last_became_active":"2024-05-08T21:23:05.517699+0000","last_became_peered":"2024-05-08T21:23:05.517699+0000","last_unstale":"2024-05-08T21:23:05.519426+0000","last_undegraded":"2024-05-08T21:23:05.519426+0000","last_fullsized":"2024-05-08T21:23:05.519426+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:19:40.125296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"34'7","reported_seq":23,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.634647+0000","last_change":"2024-05-08T21:22:50.206910+0000","last_active":"2024-05-08T21:23:04.634647+0000","last_peered":"2024-05-08T21:23:04.634647+0000","last_clean":"2024-05-08T21:23:04.634647+0000","last_became_active":"2024-05-08T21:22:50.206623+0000","last_became_peered":"2024-05-08T21:22:50.206623+0000","last_unstale":"2024-05-08T21:23:04.634647+0000","last_undegraded":"2024-05-08T21:23:04.634647+0000","last_fullsized":"2024-05-08T21:23:04.634647+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:24:25.163652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"36'3","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641631+0000","last_change":"2024-05-08T21:22:49.743344+0000","last_active":"2024-05-08T21:23:04.641631+0000","last_peered":"2024-05-08T21:23:04.641631+0000","last_clean":"2024-05-08T21:23:04.641631+0000","last_became_active":"2024-05-08T21:22:49.742413+0000","last_became_peered":"2024-05-08T21:22:49.742413+0000","last_unstale":"2024-05-08T21:23:04.641631+0000","last_undegraded":"2024-05-08T21:23:04.641631+0000","last_fullsized":"2024-05-08T21:23:04.641631+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:06:42.743465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493284+0000","last_change":"2024-05-08T21:23:05.491687+0000","last_active":"2024-05-08T21:23:05.493284+0000","last_peered":"2024-05-08T21:23:05.493284+0000","last_clean":"2024-05-08T21:23:05.493284+0000","last_became_active":"2024-05-08T21:23:05.491402+0000","last_became_peered":"2024-05-08T21:23:05.491402+0000","last_unstale":"2024-05-08T21:23:05.493284+0000","last_undegraded":"2024-05-08T21:23:05.493284+0000","last_fullsized":"2024-05-08T21:23:05.493284+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:55:39.465260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"34'8","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.639350+0000","last_change":"2024-05-08T21:22:50.205911+0000","last_active":"2024-05-08T21:23:04.639350+0000","last_peered":"2024-05-08T21:23:04.639350+0000","last_clean":"2024-05-08T21:23:04.639350+0000","last_became_active":"2024-05-08T21:22:50.205609+0000","last_became_peered":"2024-05-08T21:22:50.205609+0000","last_unstale":"2024-05-08T21:23:04.639350+0000","last_undegraded":"2024-05-08T21:23:04.639350+0000","last_fullsized":"2024-05-08T21:23:04.639350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:41:28.819967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"37'3","reported_seq":19,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637909+0000","last_change":"2024-05-08T21:22:49.745047+0000","last_active":"2024-05-08T21:23:04.637909+0000","last_peered":"2024-05-08T21:23:04.637909+0000","last_clean":"2024-05-08T21:23:04.637909+0000","last_became_active":"2024-05-08T21:22:49.744747+0000","last_became_peered":"2024-05-08T21:22:49.744747+0000","last_unstale":"2024-05-08T21:23:04.637909+0000","last_undegraded":"2024-05-08T21:23:04.637909+0000","last_fullsized":"2024-05-08T21:23:04.637909+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:05:26.886351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.493199+0000","last_change":"2024-05-08T21:23:05.491039+0000","last_active":"2024-05-08T21:23:05.493199+0000","last_peered":"2024-05-08T21:23:05.493199+0000","last_clean":"2024-05-08T21:23:05.493199+0000","last_became_active":"2024-05-08T21:23:05.490697+0000","last_became_peered":"2024-05-08T21:23:05.490697+0000","last_unstale":"2024-05-08T21:23:05.493199+0000","last_undegraded":"2024-05-08T21:23:05.493199+0000","last_fullsized":"2024-05-08T21:23:05.493199+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:54:44.860139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"34'5","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.636798+0000","last_change":"2024-05-08T21:22:50.203176+0000","last_active":"2024-05-08T21:23:04.636798+0000","last_peered":"2024-05-08T21:23:04.636798+0000","last_clean":"2024-05-08T21:23:04.636798+0000","last_became_active":"2024-05-08T21:22:50.202886+0000","last_became_peered":"2024-05-08T21:22:50.202886+0000","last_unstale":"2024-05-08T21:23:04.636798+0000","last_undegraded":"2024-05-08T21:23:04.636798+0000","last_fullsized":"2024-05-08T21:23:04.636798+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:40.200175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.641367+0000","last_change":"2024-05-08T21:22:49.742028+0000","last_active":"2024-05-08T21:23:04.641367+0000","last_peered":"2024-05-08T21:23:04.641367+0000","last_clean":"2024-05-08T21:23:04.641367+0000","last_became_active":"2024-05-08T21:22:49.741706+0000","last_became_peered":"2024-05-08T21:22:49.741706+0000","last_unstale":"2024-05-08T21:23:04.641367+0000","last_undegraded":"2024-05-08T21:23:04.641367+0000","last_fullsized":"2024-05-08T21:23:04.641367+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:06:09.036243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.518482+0000","last_change":"2024-05-08T21:23:05.517966+0000","last_active":"2024-05-08T21:23:05.518482+0000","last_peered":"2024-05-08T21:23:05.518482+0000","last_clean":"2024-05-08T21:23:05.518482+0000","last_became_active":"2024-05-08T21:23:05.517646+0000","last_became_peered":"2024-05-08T21:23:05.517646+0000","last_unstale":"2024-05-08T21:23:05.518482+0000","last_undegraded":"2024-05-08T21:23:05.518482+0000","last_fullsized":"2024-05-08T21:23:05.518482+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:11:31.133470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"34'8","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637421+0000","last_change":"2024-05-08T21:22:50.210183+0000","last_active":"2024-05-08T21:23:04.637421+0000","last_peered":"2024-05-08T21:23:04.637421+0000","last_clean":"2024-05-08T21:23:04.637421+0000","last_became_active":"2024-05-08T21:22:50.209860+0000","last_became_peered":"2024-05-08T21:22:50.209860+0000","last_unstale":"2024-05-08T21:23:04.637421+0000","last_undegraded":"2024-05-08T21:23:04.637421+0000","last_fullsized":"2024-05-08T21:23:04.637421+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:28.358029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637180+0000","last_change":"2024-05-08T21:22:49.740650+0000","last_active":"2024-05-08T21:23:04.637180+0000","last_peered":"2024-05-08T21:23:04.637180+0000","last_clean":"2024-05-08T21:23:04.637180+0000","last_became_active":"2024-05-08T21:22:49.740373+0000","last_became_peered":"2024-05-08T21:22:49.740373+0000","last_unstale":"2024-05-08T21:23:04.637180+0000","last_undegraded":"2024-05-08T21:23:04.637180+0000","last_fullsized":"2024-05-08T21:23:04.637180+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:36:43.622828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.494023+0000","last_change":"2024-05-08T21:23:05.493142+0000","last_active":"2024-05-08T21:23:05.494023+0000","last_peered":"2024-05-08T21:23:05.494023+0000","last_clean":"2024-05-08T21:23:05.494023+0000","last_became_active":"2024-05-08T21:23:05.492813+0000","last_became_peered":"2024-05-08T21:23:05.492813+0000","last_unstale":"2024-05-08T21:23:05.494023+0000","last_undegraded":"2024-05-08T21:23:05.494023+0000","last_fullsized":"2024-05-08T21:23:05.494023+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:13:30.977530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"34'7","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637385+0000","last_change":"2024-05-08T21:22:50.208073+0000","last_active":"2024-05-08T21:23:04.637385+0000","last_peered":"2024-05-08T21:23:04.637385+0000","last_clean":"2024-05-08T21:23:04.637385+0000","last_became_active":"2024-05-08T21:22:50.207506+0000","last_became_peered":"2024-05-08T21:22:50.207506+0000","last_unstale":"2024-05-08T21:23:04.637385+0000","last_undegraded":"2024-05-08T21:23:04.637385+0000","last_fullsized":"2024-05-08T21:23:04.637385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:15:17.946198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.640849+0000","last_change":"2024-05-08T21:22:49.742392+0000","last_active":"2024-05-08T21:23:04.640849+0000","last_peered":"2024-05-08T21:23:04.640849+0000","last_clean":"2024-05-08T21:23:04.640849+0000","last_became_active":"2024-05-08T21:22:49.742093+0000","last_became_peered":"2024-05-08T21:22:49.742093+0000","last_unstale":"2024-05-08T21:23:04.640849+0000","last_undegraded":"2024-05-08T21:23:04.640849+0000","last_fullsized":"2024-05-08T21:23:04.640849+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:51.977551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.519268+0000","last_change":"2024-05-08T21:23:05.517833+0000","last_active":"2024-05-08T21:23:05.519268+0000","last_peered":"2024-05-08T21:23:05.519268+0000","last_clean":"2024-05-08T21:23:05.519268+0000","last_became_active":"2024-05-08T21:23:05.517531+0000","last_became_peered":"2024-05-08T21:23:05.517531+0000","last_unstale":"2024-05-08T21:23:05.519268+0000","last_undegraded":"2024-05-08T21:23:05.519268+0000","last_fullsized":"2024-05-08T21:23:05.519268+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:26:26.242337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"34'8","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.633068+0000","last_change":"2024-05-08T21:22:50.214641+0000","last_active":"2024-05-08T21:23:04.633068+0000","last_peered":"2024-05-08T21:23:04.633068+0000","last_clean":"2024-05-08T21:23:04.633068+0000","last_became_active":"2024-05-08T21:22:50.213619+0000","last_became_peered":"2024-05-08T21:22:50.213619+0000","last_unstale":"2024-05-08T21:23:04.633068+0000","last_undegraded":"2024-05-08T21:23:04.633068+0000","last_fullsized":"2024-05-08T21:23:04.633068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:21:31.919852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632569+0000","last_change":"2024-05-08T21:22:49.738120+0000","last_active":"2024-05-08T21:23:04.632569+0000","last_peered":"2024-05-08T21:23:04.632569+0000","last_clean":"2024-05-08T21:23:04.632569+0000","last_became_active":"2024-05-08T21:22:49.737805+0000","last_became_peered":"2024-05-08T21:22:49.737805+0000","last_unstale":"2024-05-08T21:23:04.632569+0000","last_undegraded":"2024-05-08T21:23:04.632569+0000","last_fullsized":"2024-05-08T21:23:04.632569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:55:00.385106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.495445+0000","last_change":"2024-05-08T21:23:05.490991+0000","last_active":"2024-05-08T21:23:05.495445+0000","last_peered":"2024-05-08T21:23:05.495445+0000","last_clean":"2024-05-08T21:23:05.495445+0000","last_became_active":"2024-05-08T21:23:05.490651+0000","last_became_peered":"2024-05-08T21:23:05.490651+0000","last_unstale":"2024-05-08T21:23:05.495445+0000","last_undegraded":"2024-05-08T21:23:05.495445+0000","last_fullsized":"2024-05-08T21:23:05.495445+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:48:55.773935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"34'5","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.632336+0000","last_change":"2024-05-08T21:22:50.205043+0000","last_active":"2024-05-08T21:23:04.632336+0000","last_peered":"2024-05-08T21:23:04.632336+0000","last_clean":"2024-05-08T21:23:04.632336+0000","last_became_active":"2024-05-08T21:22:50.204630+0000","last_became_peered":"2024-05-08T21:22:50.204630+0000","last_unstale":"2024-05-08T21:23:04.632336+0000","last_undegraded":"2024-05-08T21:23:04.632336+0000","last_fullsized":"2024-05-08T21:23:04.632336+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:26:57.146675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.633296+0000","last_change":"2024-05-08T21:22:49.736783+0000","last_active":"2024-05-08T21:23:04.633296+0000","last_peered":"2024-05-08T21:23:04.633296+0000","last_clean":"2024-05-08T21:23:04.633296+0000","last_became_active":"2024-05-08T21:22:49.736479+0000","last_became_peered":"2024-05-08T21:22:49.736479+0000","last_unstale":"2024-05-08T21:23:04.633296+0000","last_undegraded":"2024-05-08T21:23:04.633296+0000","last_fullsized":"2024-05-08T21:23:04.633296+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:24:24.156179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-08T21:23:05.495438+0000","last_change":"2024-05-08T21:23:05.493679+0000","last_active":"2024-05-08T21:23:05.495438+0000","last_peered":"2024-05-08T21:23:05.495438+0000","last_clean":"2024-05-08T21:23:05.495438+0000","last_became_active":"2024-05-08T21:23:05.493429+0000","last_became_peered":"2024-05-08T21:23:05.493429+0000","last_unstale":"2024-05-08T21:23:05.495438+0000","last_undegraded":"2024-05-08T21:23:05.495438+0000","last_fullsized":"2024-05-08T21:23:05.495438+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:23:04.618405+0000","last_clean_scrub_stamp":"2024-05-08T21:23:04.618405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:28:15.819527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"34'9","reported_seq":25,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635260+0000","last_change":"2024-05-08T21:22:50.202594+0000","last_active":"2024-05-08T21:23:04.635260+0000","last_peered":"2024-05-08T21:23:04.635260+0000","last_clean":"2024-05-08T21:23:04.635260+0000","last_became_active":"2024-05-08T21:22:50.202278+0000","last_became_peered":"2024-05-08T21:22:50.202278+0000","last_unstale":"2024-05-08T21:23:04.635260+0000","last_undegraded":"2024-05-08T21:23:04.635260+0000","last_fullsized":"2024-05-08T21:23:04.635260+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:49.718045+0000","last_clean_scrub_stamp":"2024-05-08T21:22:49.718045+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:44:18.055232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.637697+0000","last_change":"2024-05-08T21:22:49.739311+0000","last_active":"2024-05-08T21:23:04.637697+0000","last_peered":"2024-05-08T21:23:04.637697+0000","last_clean":"2024-05-08T21:23:04.637697+0000","last_became_active":"2024-05-08T21:22:49.739006+0000","last_became_peered":"2024-05-08T21:22:49.739006+0000","last_unstale":"2024-05-08T21:23:04.637697+0000","last_undegraded":"2024-05-08T21:23:04.637697+0000","last_fullsized":"2024-05-08T21:23:04.637697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:22:48.710572+0000","last_clean_scrub_stamp":"2024-05-08T21:22:48.710572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:39:42.856227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":183,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-08T21:23:04.635012+0000","last_change":"2024-05-08T21:21:42.333097+0000","last_active":"2024-05-08T21:23:04.635012+0000","last_peered":"2024-05-08T21:23:04.635012+0000","last_clean":"2024-05-08T21:23:04.635012+0000","last_became_active":"2024-05-08T21:21:42.332033+0000","last_became_peered":"2024-05-08T21:21:42.332033+0000","last_unstale":"2024-05-08T21:23:04.635012+0000","last_undegraded":"2024-05-08T21:23:04.635012+0000","last_fullsized":"2024-05-08T21:23:04.635012+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_clean_scrub_stamp":"2024-05-08T21:21:40.862622+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:39:24.772551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":6291456,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12582912,"data_stored":12582912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":504,"ondisk_log_size":504,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":92,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30112,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341728,"statfs":{"total":96636764160,"available":96605929472,"internally_reserved":0,"allocated":3162112,"data_stored":2883557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.72199999999999998}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.76900000000000002}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.82599999999999996}]},{"osd":3,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.80100000000000005}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.69899999999999995}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.57599999999999996}]},{"osd":6,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.63800000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29516,"kb_used_data":2496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342324,"statfs":{"total":96636764160,"available":96606539776,"internally_reserved":0,"allocated":2555904,"data_stored":2285873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.497},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.56100000000000005}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.443},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.64800000000000002}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.66100000000000003}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.73899999999999999}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.68200000000000005}]},{"osd":7,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.752}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31564,"kb_used_data":4544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340276,"statfs":{"total":96636764160,"available":96604442624,"internally_reserved":0,"allocated":4653056,"data_stored":4383025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.375},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.35599999999999998}]},{"osd":2,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.504}]},{"osd":3,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.55200000000000005}]},{"osd":4,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.56899999999999995}]},{"osd":5,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.44900000000000001}]},{"osd":6,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.52500000000000002}]},{"osd":7,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.624},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.59199999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343776,"statfs":{"total":96636764160,"available":96608026624,"internally_reserved":0,"allocated":1064960,"data_stored":786405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.53700000000000003}]},{"osd":2,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.436}]},{"osd":3,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.57299999999999995}]},{"osd":4,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.55800000000000005}]},{"osd":5,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.61499999999999999}]},{"osd":6,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.69099999999999995}]},{"osd":7,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.65100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":458752,"data_stored":188721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.41899999999999998}]},{"osd":1,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.33200000000000002}]},{"osd":3,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.432}]},{"osd":4,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.495}]},{"osd":5,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.51100000000000001}]},{"osd":6,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.76000000000000001}]},{"osd":7,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.623}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":458752,"data_stored":188721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":2.7919999999999998,"5min":2.7919999999999998,"15min":2.7919999999999998},"last":0.41099999999999998}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.438},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.54500000000000004}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":2.9369999999999998,"5min":2.9369999999999998,"15min":2.9369999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.57899999999999996}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.8380000000000001,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":0.66000000000000003}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.9060000000000001,"5min":2.9060000000000001,"15min":2.9060000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.68600000000000005}]},{"osd":6,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":2.8730000000000002,"5min":2.8730000000000002,"15min":2.8730000000000002},"last":0.76200000000000001}]},{"osd":7,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.64300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":458752,"data_stored":188721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.50700000000000001}]},{"osd":1,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.38200000000000001}]},{"osd":2,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.622},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.45900000000000002}]},{"osd":3,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.52800000000000002}]},{"osd":5,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.42199999999999999}]},{"osd":6,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65400000000000003}]},{"osd":7,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.59999999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31564,"kb_used_data":4544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340276,"statfs":{"total":96636764160,"available":96604442624,"internally_reserved":0,"allocated":4653056,"data_stored":4383025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.36299999999999999}]},{"osd":1,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.43099999999999999}]},{"osd":2,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.495}]},{"osd":3,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.376},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.45300000000000001}]},{"osd":4,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.40799999999999997}]},{"osd":6,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.443}]},{"osd":7,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2097152,"data_stored":2097152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2097152,"data_stored":2097152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T21:23:10.319 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-08T21:23:10.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -s 2024-05-08T21:23:10.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.488+0000 7f82b95da700 1 Processor -- start 2024-05-08T21:23:10.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.488+0000 7f82b95da700 1 -- start start 2024-05-08T21:23:10.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.489+0000 7f82b95da700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82b40af1a0 0x7f82b406b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.489+0000 7f82b95da700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f82b406beb0 0x7f82b41615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.489+0000 7f82b95da700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b40b4f20 con 0x7f82b4161b30 2024-05-08T21:23:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.489+0000 7f82b95da700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82b406e6b0 con 0x7f82b406beb0 2024-05-08T21:23:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.489+0000 7f82b95da700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b406c330 con 0x7f82b40af1a0 2024-05-08T21:23:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.489+0000 7f82b2ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82b40af1a0 0x7f82b406b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.489+0000 7f82b37fe700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f82b4161b30 0x7f82b40b2bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:40814/0 (socket says 172.21.15.3:40814) 2024-05-08T21:23:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.489+0000 7f82b37fe700 1 -- 172.21.15.3:0/3057650949 learned_addr learned my addr 172.21.15.3:0/3057650949 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.489+0000 7f82abfff700 1 --2- 172.21.15.3:0/3057650949 >> v2:172.21.15.186:3300/0 conn(0x7f82b406beb0 0x7f82b41615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.489+0000 7f82b2ffd700 1 -- 172.21.15.3:0/3057650949 >> v2:172.21.15.186:3300/0 conn(0x7f82b406beb0 msgr2=0x7f82b41615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.489+0000 7f82b2ffd700 1 --2- 172.21.15.3:0/3057650949 >> v2:172.21.15.186:3300/0 conn(0x7f82b406beb0 0x7f82b41615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.489+0000 7f82b2ffd700 1 -- 172.21.15.3:0/3057650949 >> v1:172.21.15.186:6789/0 conn(0x7f82b4161b30 legacy=0x7f82b40b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.489+0000 7f82b2ffd700 1 -- 172.21.15.3:0/3057650949 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b4162380 con 0x7f82b40af1a0 2024-05-08T21:23:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.490+0000 7f82b2ffd700 1 --2- 172.21.15.3:0/3057650949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82b40af1a0 0x7f82b406b970 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f82a4009840 tx=0x7f82a4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=383aec30e7e61624 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.490+0000 7f82b27fc700 1 -- 172.21.15.3:0/3057650949 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a4011830 con 0x7f82b40af1a0 2024-05-08T21:23:10.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.490+0000 7f82b27fc700 1 -- 172.21.15.3:0/3057650949 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a4011990 con 0x7f82b40af1a0 2024-05-08T21:23:10.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.490+0000 7f82b27fc700 1 -- 172.21.15.3:0/3057650949 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a4011d20 con 0x7f82b40af1a0 2024-05-08T21:23:10.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.490+0000 7f82b95da700 1 -- 172.21.15.3:0/3057650949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82b40af1a0 msgr2=0x7f82b406b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.490+0000 7f82b95da700 1 --2- 172.21.15.3:0/3057650949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82b40af1a0 0x7f82b406b970 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f82a4009840 tx=0x7f82a4009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:10.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.490+0000 7f82b95da700 1 -- 172.21.15.3:0/3057650949 shutdown_connections 2024-05-08T21:23:10.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.490+0000 7f82b95da700 1 --2- 172.21.15.3:0/3057650949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82b40af1a0 0x7f82b406b970 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.490+0000 7f82b95da700 1 --2- 172.21.15.3:0/3057650949 >> v2:172.21.15.186:3300/0 conn(0x7f82b406beb0 0x7f82b41615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.491+0000 7f82b95da700 1 -- 172.21.15.3:0/3057650949 >> 172.21.15.3:0/3057650949 conn(0x7f82b4071410 msgr2=0x7f82b4071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:10.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.491+0000 7f82b95da700 1 -- 172.21.15.3:0/3057650949 shutdown_connections 2024-05-08T21:23:10.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.491+0000 7f82b95da700 1 -- 172.21.15.3:0/3057650949 wait complete. 2024-05-08T21:23:10.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.492+0000 7f82b95da700 1 Processor -- start 2024-05-08T21:23:10.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.492+0000 7f82b95da700 1 -- start start 2024-05-08T21:23:10.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82b95da700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82b406beb0 0x7f82b40cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82b95da700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82b40af1a0 0x7f82b41403e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82b95da700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82b4161b30 0x7f82b40cf100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82b95da700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b415d870 con 0x7f82b4161b30 2024-05-08T21:23:10.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82b95da700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b40ce620 con 0x7f82b40af1a0 2024-05-08T21:23:10.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82b95da700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b40ce900 con 0x7f82b406beb0 2024-05-08T21:23:10.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82b37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82b4161b30 0x7f82b40cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82b37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82b4161b30 0x7f82b40cf100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54288/0 (socket says 172.21.15.3:54288) 2024-05-08T21:23:10.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82b37fe700 1 -- 172.21.15.3:0/2603347504 learned_addr learned my addr 172.21.15.3:0/2603347504 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:10.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82b2ffd700 1 --2- 172.21.15.3:0/2603347504 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82b406beb0 0x7f82b40cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82abfff700 1 --2- 172.21.15.3:0/2603347504 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82b40af1a0 0x7f82b41403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82b37fe700 1 -- 172.21.15.3:0/2603347504 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82b406beb0 msgr2=0x7f82b40cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82b37fe700 1 --2- 172.21.15.3:0/2603347504 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82b406beb0 0x7f82b40cc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82b37fe700 1 -- 172.21.15.3:0/2603347504 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82b40af1a0 msgr2=0x7f82b41403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82b37fe700 1 --2- 172.21.15.3:0/2603347504 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82b40af1a0 0x7f82b41403e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82b37fe700 1 -- 172.21.15.3:0/2603347504 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a40094a0 con 0x7f82b4161b30 2024-05-08T21:23:10.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82b37fe700 1 --2- 172.21.15.3:0/2603347504 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82b4161b30 0x7f82b40cf100 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f82ac00cbc0 tx=0x7f82ac00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.493+0000 7f82ab7fe700 1 -- 172.21.15.3:0/2603347504 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac00a9e0 con 0x7f82b4161b30 2024-05-08T21:23:10.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.494+0000 7f82ab7fe700 1 -- 172.21.15.3:0/2603347504 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ac00ab40 con 0x7f82b4161b30 2024-05-08T21:23:10.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.494+0000 7f82b95da700 1 -- 172.21.15.3:0/2603347504 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b40cebe0 con 0x7f82b4161b30 2024-05-08T21:23:10.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.494+0000 7f82ab7fe700 1 -- 172.21.15.3:0/2603347504 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac012400 con 0x7f82b4161b30 2024-05-08T21:23:10.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.494+0000 7f82b95da700 1 -- 172.21.15.3:0/2603347504 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b40ccb70 con 0x7f82b4161b30 2024-05-08T21:23:10.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.495+0000 7f82b95da700 1 -- 172.21.15.3:0/2603347504 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b40cedd0 con 0x7f82b4161b30 2024-05-08T21:23:10.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.495+0000 7f82ab7fe700 1 -- 172.21.15.3:0/2603347504 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f82ac01b3b0 con 0x7f82b4161b30 2024-05-08T21:23:10.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.495+0000 7f82ab7fe700 1 --2- 172.21.15.3:0/2603347504 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f829406acc0 0x7f829406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.496+0000 7f82ab7fe700 1 -- 172.21.15.3:0/2603347504 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f82ac089030 con 0x7f82b4161b30 2024-05-08T21:23:10.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.496+0000 7f82b2ffd700 1 --2- 172.21.15.3:0/2603347504 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f829406acc0 0x7f829406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.496+0000 7f82b2ffd700 1 --2- 172.21.15.3:0/2603347504 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f829406acc0 0x7f829406d180 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f82a4000c00 tx=0x7f82a4008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.500+0000 7f82ab7fe700 1 -- 172.21.15.3:0/2603347504 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f82ac0540a0 con 0x7f82b4161b30 2024-05-08T21:23:10.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.812+0000 7f82b95da700 1 -- 172.21.15.3:0/2603347504 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f82b40ccdc0 con 0x7f82b4161b30 2024-05-08T21:23:10.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.813+0000 7f82ab7fe700 1 -- 172.21.15.3:0/2603347504 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+616 (secure 0 0 0) 0x7f82ac017280 con 0x7f82b4161b30 2024-05-08T21:23:10.814 INFO:teuthology.orchestra.run.smithi003.stdout: cluster: 2024-05-08T21:23:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: id: d0f58275-758f-4470-b003-84d6abe25489 2024-05-08T21:23:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: health: HEALTH_ERR 2024-05-08T21:23:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: 2 filesystems are degraded 2024-05-08T21:23:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: 2 filesystems are offline 2024-05-08T21:23:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: services: 2024-05-08T21:23:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: mon: 3 daemons, quorum a,b,c (age 102s) 2024-05-08T21:23:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: mgr: x(active, since 96s), standbys: y 2024-05-08T21:23:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: mds: 0/2 daemons up (2 failed), 4 standby 2024-05-08T21:23:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: osd: 8 osds: 8 up (since 91s), 8 in (since 99s) 2024-05-08T21:23:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: data: 2024-05-08T21:23:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: volumes: 0/2 healthy, 2 failed 2024-05-08T21:23:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: pools: 4 pools, 129 pgs 2024-05-08T21:23:10.816 INFO:teuthology.orchestra.run.smithi003.stdout: objects: 4 objects, 6.6 MiB 2024-05-08T21:23:10.816 INFO:teuthology.orchestra.run.smithi003.stdout: usage: 228 MiB used, 720 GiB / 720 GiB avail 2024-05-08T21:23:10.816 INFO:teuthology.orchestra.run.smithi003.stdout: pgs: 129 active+clean 2024-05-08T21:23:10.816 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:10.816 INFO:teuthology.orchestra.run.smithi003.stdout: io: 2024-05-08T21:23:10.816 INFO:teuthology.orchestra.run.smithi003.stdout: client: 0 B/s rd, 1.5 KiB/s wr, 0 op/s rd, 6 op/s wr 2024-05-08T21:23:10.816 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:10.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.815+0000 7f82b95da700 1 -- 172.21.15.3:0/2603347504 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f829406acc0 msgr2=0x7f829406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.815+0000 7f82b95da700 1 --2- 172.21.15.3:0/2603347504 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f829406acc0 0x7f829406d180 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f82a4000c00 tx=0x7f82a4008030 comp rx=0 tx=0).stop 2024-05-08T21:23:10.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.815+0000 7f82b95da700 1 -- 172.21.15.3:0/2603347504 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82b4161b30 msgr2=0x7f82b40cf100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.815+0000 7f82b95da700 1 --2- 172.21.15.3:0/2603347504 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82b4161b30 0x7f82b40cf100 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f82ac00cbc0 tx=0x7f82ac00ced0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.815+0000 7f82b95da700 1 -- 172.21.15.3:0/2603347504 shutdown_connections 2024-05-08T21:23:10.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.815+0000 7f82b95da700 1 --2- 172.21.15.3:0/2603347504 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82b406beb0 0x7f82b40cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.815+0000 7f82b95da700 1 --2- 172.21.15.3:0/2603347504 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f829406acc0 0x7f829406d180 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.815+0000 7f82b95da700 1 --2- 172.21.15.3:0/2603347504 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82b4161b30 0x7f82b40cf100 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.815+0000 7f82b95da700 1 --2- 172.21.15.3:0/2603347504 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82b40af1a0 0x7f82b41403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.815+0000 7f82b95da700 1 -- 172.21.15.3:0/2603347504 >> 172.21.15.3:0/2603347504 conn(0x7f82b4071410 msgr2=0x7f82b406a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:10.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.815+0000 7f82b95da700 1 -- 172.21.15.3:0/2603347504 shutdown_connections 2024-05-08T21:23:10.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.815+0000 7f82b95da700 1 -- 172.21.15.3:0/2603347504 wait complete. 2024-05-08T21:23:10.830 DEBUG:teuthology.orchestra.run.smithi003:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 init --force-init --filesystem recovery_fs --alternate-pool recovery_fs_metadata 2024-05-08T21:23:10.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.878+0000 7f6e389b1940 1 -- start start 2024-05-08T21:23:10.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.878+0000 7f6e389b1940 1 Processor -- start 2024-05-08T21:23:10.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e389b1940 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b591d21000 0x55b591db4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e389b1940 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b591d20c00 0x55b591db4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e389b1940 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b591d20800 0x55b591db5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e389b1940 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b591da4a00 con 0x55b591d21000 2024-05-08T21:23:10.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e389b1940 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b591da4780 con 0x55b591d20800 2024-05-08T21:23:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e389b1940 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b591da4640 con 0x55b591d20c00 2024-05-08T21:23:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e3695f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b591d21000 0x55b591db4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e3695f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b591d21000 0x55b591db4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54310/0 (socket says 172.21.15.3:54310) 2024-05-08T21:23:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e3695f700 1 -- 172.21.15.3:0/2808221855 learned_addr learned my addr 172.21.15.3:0/2808221855 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e37160700 1 --2- 172.21.15.3:0/2808221855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b591d20800 0x55b591db5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e3615e700 1 --2- 172.21.15.3:0/2808221855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b591d20c00 0x55b591db4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e3695f700 1 -- 172.21.15.3:0/2808221855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b591d20c00 msgr2=0x55b591db4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e3695f700 1 --2- 172.21.15.3:0/2808221855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b591d20c00 0x55b591db4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e3695f700 1 -- 172.21.15.3:0/2808221855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b591d20800 msgr2=0x55b591db5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e3695f700 1 --2- 172.21.15.3:0/2808221855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b591d20800 0x55b591db5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e3695f700 1 -- 172.21.15.3:0/2808221855 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b591d449c0 con 0x55b591d21000 2024-05-08T21:23:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.879+0000 7f6e3615e700 1 --2- 172.21.15.3:0/2808221855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b591d20c00 0x55b591db4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.880+0000 7f6e3695f700 1 --2- 172.21.15.3:0/2808221855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b591d21000 0x55b591db4580 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x55b591df1290 tx=0x55b592b9ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea52d5861f3c62b server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.880+0000 7f6e3415a700 1 -- 172.21.15.3:0/2808221855 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b591d51600 con 0x55b591d21000 2024-05-08T21:23:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.880+0000 7f6e3415a700 1 -- 172.21.15.3:0/2808221855 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b591d73080 con 0x55b591d21000 2024-05-08T21:23:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.880+0000 7f6e3415a700 1 -- 172.21.15.3:0/2808221855 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b591d511e0 con 0x55b591d21000 2024-05-08T21:23:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.880+0000 7f6e389b1940 1 -- 172.21.15.3:0/2808221855 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b591d44b60 con 0x55b591d21000 2024-05-08T21:23:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.880+0000 7f6e389b1940 1 -- 172.21.15.3:0/2808221855 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55b591d44340 con 0x55b591d21000 2024-05-08T21:23:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.880+0000 7f6e389b1940 4 waiting for MDS map... 2024-05-08T21:23:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.880+0000 7f6e3415a700 1 -- 172.21.15.3:0/2808221855 <== mon.0 v2:172.21.15.3:3300/0 4 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x55b591d5ec40 con 0x55b591d21000 2024-05-08T21:23:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.880+0000 7f6e3415a700 1 -- 172.21.15.3:0/2808221855 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 21) v1 ==== 3691+0+0 (secure 0 0 0) 0x55b592b9e300 con 0x55b591d21000 2024-05-08T21:23:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.880+0000 7f6e389b1940 4 Got MDS map 21 2024-05-08T21:23:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.880+0000 7f6e389b1940 4 datascan.main: Using metadata pool output 2024-05-08T21:23:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.880+0000 7f6e389b1940 4 datascan.main: connecting to RADOS... 2024-05-08T21:23:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.881+0000 7f6e389b1940 1 Processor -- start 2024-05-08T21:23:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.881+0000 7f6e389b1940 1 -- start start 2024-05-08T21:23:10.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.881+0000 7f6e389b1940 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b591d21c00 0x55b591db5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.881+0000 7f6e389b1940 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b592038000 0x55b591db5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.881+0000 7f6e389b1940 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b592038400 0x55b591db6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.881+0000 7f6e389b1940 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b591d5de00 con 0x55b591d21c00 2024-05-08T21:23:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.881+0000 7f6e389b1940 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b591da4780 con 0x55b592038000 2024-05-08T21:23:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.881+0000 7f6e389b1940 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b591da4500 con 0x55b592038400 2024-05-08T21:23:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.882+0000 7f6e37160700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b591d21c00 0x55b591db5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.882+0000 7f6e37160700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b591d21c00 0x55b591db5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54324/0 (socket says 172.21.15.3:54324) 2024-05-08T21:23:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.882+0000 7f6e37160700 1 -- 172.21.15.3:0/66997716 learned_addr learned my addr 172.21.15.3:0/66997716 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.882+0000 7f6e3695f700 1 --2- 172.21.15.3:0/66997716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b592038000 0x55b591db5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.882+0000 7f6e3615e700 1 --2- 172.21.15.3:0/66997716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b592038400 0x55b591db6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.882+0000 7f6e37160700 1 -- 172.21.15.3:0/66997716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b592038400 msgr2=0x55b591db6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.882+0000 7f6e37160700 1 --2- 172.21.15.3:0/66997716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b592038400 0x55b591db6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.882+0000 7f6e37160700 1 -- 172.21.15.3:0/66997716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b592038000 msgr2=0x55b591db5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.882+0000 7f6e37160700 1 --2- 172.21.15.3:0/66997716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b592038000 0x55b591db5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.882+0000 7f6e37160700 1 -- 172.21.15.3:0/66997716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b591d44000 con 0x55b591d21c00 2024-05-08T21:23:10.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.882+0000 7f6e3615e700 1 --2- 172.21.15.3:0/66997716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b592038400 0x55b591db6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:10.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.882+0000 7f6e37160700 1 --2- 172.21.15.3:0/66997716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b591d21c00 0x55b591db5600 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x55b592bbe060 tx=0x55b592ab9260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445956ba7935ce1e server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.882+0000 7f6e32156700 1 -- 172.21.15.3:0/66997716 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b591d51080 con 0x55b591d21c00 2024-05-08T21:23:10.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.882+0000 7f6e32156700 1 -- 172.21.15.3:0/66997716 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b592b9c300 con 0x55b591d21c00 2024-05-08T21:23:10.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.882+0000 7f6e32156700 1 -- 172.21.15.3:0/66997716 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b591d51600 con 0x55b591d21c00 2024-05-08T21:23:10.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.883+0000 7f6e389b1940 1 -- 172.21.15.3:0/66997716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b591d21c00 msgr2=0x55b591db5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.883+0000 7f6e389b1940 1 --2- 172.21.15.3:0/66997716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b591d21c00 0x55b591db5600 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x55b592bbe060 tx=0x55b592ab9260 comp rx=0 tx=0).stop 2024-05-08T21:23:10.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.883+0000 7f6e389b1940 1 -- 172.21.15.3:0/66997716 shutdown_connections 2024-05-08T21:23:10.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.883+0000 7f6e389b1940 1 --2- 172.21.15.3:0/66997716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b592038400 0x55b591db6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.883+0000 7f6e389b1940 1 --2- 172.21.15.3:0/66997716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b591d21c00 0x55b591db5600 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.883+0000 7f6e389b1940 1 --2- 172.21.15.3:0/66997716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b592038000 0x55b591db5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.883+0000 7f6e389b1940 1 -- 172.21.15.3:0/66997716 >> 172.21.15.3:0/66997716 conn(0x55b591d20400 msgr2=0x55b592aa1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:10.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.883+0000 7f6e389b1940 1 -- 172.21.15.3:0/66997716 shutdown_connections 2024-05-08T21:23:10.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.883+0000 7f6e389b1940 1 -- 172.21.15.3:0/66997716 wait complete. 2024-05-08T21:23:10.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.883+0000 7f6e389b1940 1 Processor -- start 2024-05-08T21:23:10.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.884+0000 7f6e389b1940 1 -- start start 2024-05-08T21:23:10.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.884+0000 7f6e389b1940 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b591d20400 0x55b591db7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.884+0000 7f6e389b1940 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b592040000 0x55b591db6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.884+0000 7f6e389b1940 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b592040400 0x55b591db6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.884+0000 7f6e389b1940 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b591da43c0 con 0x55b592040000 2024-05-08T21:23:10.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.884+0000 7f6e389b1940 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b591da4280 con 0x55b592040400 2024-05-08T21:23:10.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.884+0000 7f6e389b1940 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b591da4140 con 0x55b591d20400 2024-05-08T21:23:10.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e3695f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b592040000 0x55b591db6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e3695f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b592040000 0x55b591db6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54328/0 (socket says 172.21.15.3:54328) 2024-05-08T21:23:10.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e3695f700 1 -- 172.21.15.3:0/2448713333 learned_addr learned my addr 172.21.15.3:0/2448713333 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:10.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e37160700 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b591d20400 0x55b591db7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e3615e700 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b592040400 0x55b591db6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e3695f700 1 -- 172.21.15.3:0/2448713333 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b591d20400 msgr2=0x55b591db7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e3695f700 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b591d20400 0x55b591db7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e3695f700 1 -- 172.21.15.3:0/2448713333 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b592040400 msgr2=0x55b591db6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e3695f700 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b592040400 0x55b591db6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e3695f700 1 -- 172.21.15.3:0/2448713333 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b591d441a0 con 0x55b592040000 2024-05-08T21:23:10.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e3615e700 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b592040400 0x55b591db6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:10.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e3695f700 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b592040000 0x55b591db6c00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x55b591df1740 tx=0x55b592b9ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e30953700 1 -- 172.21.15.3:0/2448713333 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b591d50f20 con 0x55b592040000 2024-05-08T21:23:10.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e30953700 1 -- 172.21.15.3:0/2448713333 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b592b9ca80 con 0x55b592040000 2024-05-08T21:23:10.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e389b1940 1 -- 172.21.15.3:0/2448713333 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b591d44000 con 0x55b592040000 2024-05-08T21:23:10.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e30953700 1 -- 172.21.15.3:0/2448713333 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b591d51080 con 0x55b592040000 2024-05-08T21:23:10.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e389b1940 1 -- 172.21.15.3:0/2448713333 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b591d449c0 con 0x55b592040000 2024-05-08T21:23:10.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.885+0000 7f6e389b1940 4 datascan.init: forcing metadata pool 'recovery_fs_metadata' 2024-05-08T21:23:10.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.887+0000 7f6e30953700 1 -- 172.21.15.3:0/2448713333 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x55b591d82840 con 0x55b592040000 2024-05-08T21:23:10.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.887+0000 7f6e30953700 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55b59205a800 0x55b591db7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.887+0000 7f6e30953700 1 -- 172.21.15.3:0/2448713333 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x55b591d5ea80 con 0x55b592040000 2024-05-08T21:23:10.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.887+0000 7f6e389b1940 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55b592040800 0x55b591db6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.887+0000 7f6e389b1940 1 -- 172.21.15.3:0/2448713333 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:1 6.14 6:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55b592040c00 con 0x55b592040800 2024-05-08T21:23:10.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.887+0000 7f6e37160700 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55b59205a800 0x55b591db7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.887+0000 7f6e3615e700 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55b592040800 0x55b591db6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.888+0000 7f6e3615e700 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55b592040800 0x55b591db6100 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.888+0000 7f6e37160700 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55b59205a800 0x55b591db7700 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x55b592bbe060 tx=0x55b592ab92c0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.888+0000 7f6e3615e700 1 -- 172.21.15.3:0/2448713333 <== osd.2 v2:172.21.15.3:6808/47117 1 ==== osd_op_reply(1 1.00000000.inode [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 160+0+0 (crc 0 0 0) 0x55b592037200 con 0x55b592040800 2024-05-08T21:23:10.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.888+0000 7f6e389b1940 1 -- 172.21.15.3:0/2448713333 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:2 6.14 6:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55b592041000 con 0x55b592040800 2024-05-08T21:23:10.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.890+0000 7f6e3615e700 1 -- 172.21.15.3:0/2448713333 <== osd.2 v2:172.21.15.3:6808/47117 2 ==== osd_op_reply(2 1.00000000.inode [writefull 0~582] v42'1 uv1 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55b592037200 con 0x55b592040800 2024-05-08T21:23:10.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.890+0000 7f6e389b1940 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55b592041400 0x55b591db5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.890+0000 7f6e389b1940 1 -- 172.21.15.3:0/2448713333 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:3 6.1f 6:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55b592041800 con 0x55b592041400 2024-05-08T21:23:10.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.890+0000 7f6e37160700 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55b592041400 0x55b591db5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.890+0000 7f6e37160700 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55b592041400 0x55b591db5b80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.891+0000 7f6e37160700 1 -- 172.21.15.3:0/2448713333 <== osd.3 v2:172.21.15.3:6800/47118 1 ==== osd_op_reply(3 100.00000000.inode [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 162+0+0 (crc 0 0 0) 0x55b5920378c0 con 0x55b592041400 2024-05-08T21:23:10.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.891+0000 7f6e389b1940 1 -- 172.21.15.3:0/2448713333 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:4 6.1f 6:fe07bba1:::100.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55b592041c00 con 0x55b592041400 2024-05-08T21:23:10.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.892+0000 7f6e37160700 1 -- 172.21.15.3:0/2448713333 <== osd.3 v2:172.21.15.3:6800/47118 2 ==== osd_op_reply(4 100.00000000.inode [writefull 0~582] v42'1 uv1 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x55b5920378c0 con 0x55b592041400 2024-05-08T21:23:10.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.892+0000 7f6e389b1940 1 -- 172.21.15.3:0/2448713333 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:5 6.13 6:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55b59205ac00 con 0x55b592041400 2024-05-08T21:23:10.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.893+0000 7f6e37160700 1 -- 172.21.15.3:0/2448713333 <== osd.3 v2:172.21.15.3:6800/47118 3 ==== osd_op_reply(5 100.00000000 [omap-get-header] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x55b5920378c0 con 0x55b592041400 2024-05-08T21:23:10.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.893+0000 7f6e389b1940 10 datascan.find_or_create_dirfrag: read_version = 0 2024-05-08T21:23:10.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.893+0000 7f6e389b1940 1 -- 172.21.15.3:0/2448713333 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:6 6.13 6:cd5a64a3:::100.00000000:head [create,omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55b59205b000 con 0x55b592041400 2024-05-08T21:23:10.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.896+0000 7f6e37160700 1 -- 172.21.15.3:0/2448713333 <== osd.3 v2:172.21.15.3:6800/47118 4 ==== osd_op_reply(6 100.00000000 [create,omap-set-header] v42'1 uv1 ondisk = 0) v8 ==== 198+0+0 (crc 0 0 0) 0x55b5920378c0 con 0x55b592041400 2024-05-08T21:23:10.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.896+0000 7f6e389b1940 10 datascan.find_or_create_dirfrag: Created dirfrag: 0x0x100 2024-05-08T21:23:10.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.896+0000 7f6e389b1940 1 -- 172.21.15.3:0/2448713333 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55b592040800 msgr2=0x55b591db6100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.896+0000 7f6e389b1940 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55b592040800 0x55b591db6100 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.896+0000 7f6e389b1940 1 -- 172.21.15.3:0/2448713333 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55b592041400 msgr2=0x55b591db5b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.896+0000 7f6e389b1940 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55b592041400 0x55b591db5b80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.896+0000 7f6e389b1940 1 -- 172.21.15.3:0/2448713333 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55b59205a800 msgr2=0x55b591db7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.896+0000 7f6e389b1940 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55b59205a800 0x55b591db7700 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x55b592bbe060 tx=0x55b592ab92c0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.896+0000 7f6e389b1940 1 -- 172.21.15.3:0/2448713333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b592040000 msgr2=0x55b591db6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.896+0000 7f6e389b1940 1 --2- 172.21.15.3:0/2448713333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b592040000 0x55b591db6c00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x55b591df1740 tx=0x55b592b9ac60 comp rx=0 tx=0).stop 2024-05-08T21:23:10.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.896+0000 7f6e3615e700 1 -- 172.21.15.3:0/2448713333 reap_dead start 2024-05-08T21:23:10.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.896+0000 7f6e389b1940 1 -- 172.21.15.3:0/2448713333 shutdown_connections 2024-05-08T21:23:10.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.896+0000 7f6e389b1940 1 -- 172.21.15.3:0/2448713333 >> 172.21.15.3:0/2448713333 conn(0x55b591d21c00 msgr2=0x55b592aa1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:10.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.896+0000 7f6e389b1940 1 -- 172.21.15.3:0/2448713333 shutdown_connections 2024-05-08T21:23:10.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.896+0000 7f6e389b1940 1 -- 172.21.15.3:0/2448713333 wait complete. 2024-05-08T21:23:10.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.897+0000 7f6e389b1940 1 -- 172.21.15.3:0/2808221855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b591d21000 msgr2=0x55b591db4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.897+0000 7f6e389b1940 1 --2- 172.21.15.3:0/2808221855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b591d21000 0x55b591db4580 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x55b591df1290 tx=0x55b592b9ad20 comp rx=0 tx=0).stop 2024-05-08T21:23:10.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.897+0000 7f6e389b1940 1 -- 172.21.15.3:0/2808221855 shutdown_connections 2024-05-08T21:23:10.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.897+0000 7f6e389b1940 1 --2- 172.21.15.3:0/2808221855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b591d20800 0x55b591db5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.897+0000 7f6e389b1940 1 --2- 172.21.15.3:0/2808221855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b591d21000 0x55b591db4580 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.897+0000 7f6e389b1940 1 --2- 172.21.15.3:0/2808221855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b591d20c00 0x55b591db4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.897+0000 7f6e389b1940 1 -- 172.21.15.3:0/2808221855 >> 172.21.15.3:0/2808221855 conn(0x55b591d21400 msgr2=0x55b592aa1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:10.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.897+0000 7f6e389b1940 1 -- 172.21.15.3:0/2808221855 shutdown_connections 2024-05-08T21:23:10.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.897+0000 7f6e389b1940 1 -- 172.21.15.3:0/2808221855 wait complete. 2024-05-08T21:23:10.903 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'init', '--force-init', '--filesystem', 'recovery_fs', '--alternate-pool', 'recovery_fs_metadata'] in time 0:00:00.072514, result: 2024-05-08T21:23:10.903 DEBUG:teuthology.orchestra.run.smithi003:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 recovery_fs:0 reset session 2024-05-08T21:23:10.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.947+0000 7ff879fcd440 1 -- start start 2024-05-08T21:23:10.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.948+0000 7ff879fcd440 1 Processor -- start 2024-05-08T21:23:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.948+0000 7ff879fcd440 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67dc89000 0x55f67dd1db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.948+0000 7ff879fcd440 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f67dc88c00 0x55f67dd1d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.948+0000 7ff879fcd440 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f67dc88800 0x55f67dd1d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.948+0000 7ff879fcd440 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f67dd0ca00 con 0x55f67dc89000 2024-05-08T21:23:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.948+0000 7ff879fcd440 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f67dd0c640 con 0x55f67dc88c00 2024-05-08T21:23:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.948+0000 7ff879fcd440 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f67dd0c500 con 0x55f67dc88800 2024-05-08T21:23:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.948+0000 7ff877f79700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67dc89000 0x55f67dd1db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.948+0000 7ff877f79700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67dc89000 0x55f67dd1db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54352/0 (socket says 172.21.15.3:54352) 2024-05-08T21:23:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.948+0000 7ff877f79700 1 -- 172.21.15.3:0/1307682848 learned_addr learned my addr 172.21.15.3:0/1307682848 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff877778700 1 --2- 172.21.15.3:0/1307682848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f67dc88c00 0x55f67dd1d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff87877a700 1 --2- 172.21.15.3:0/1307682848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f67dc88800 0x55f67dd1d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff877f79700 1 -- 172.21.15.3:0/1307682848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f67dc88800 msgr2=0x55f67dd1d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff877f79700 1 --2- 172.21.15.3:0/1307682848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f67dc88800 0x55f67dd1d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff877f79700 1 -- 172.21.15.3:0/1307682848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f67dc88c00 msgr2=0x55f67dd1d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff877f79700 1 --2- 172.21.15.3:0/1307682848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f67dc88c00 0x55f67dd1d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff877f79700 1 -- 172.21.15.3:0/1307682848 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f67dcac9c0 con 0x55f67dc89000 2024-05-08T21:23:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff877f79700 1 --2- 172.21.15.3:0/1307682848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67dc89000 0x55f67dd1db80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x55f67eaf66c0 tx=0x55f67ddf2ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c424f4379bccf6d server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff875774700 1 -- 172.21.15.3:0/1307682848 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f67dcb9600 con 0x55f67dc89000 2024-05-08T21:23:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff875774700 1 -- 172.21.15.3:0/1307682848 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f67dcdb080 con 0x55f67dc89000 2024-05-08T21:23:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff875774700 1 -- 172.21.15.3:0/1307682848 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f67dcb91e0 con 0x55f67dc89000 2024-05-08T21:23:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff879fcd440 1 -- 172.21.15.3:0/1307682848 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f67dcacb60 con 0x55f67dc89000 2024-05-08T21:23:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff879fcd440 1 -- 172.21.15.3:0/1307682848 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55f67dcac340 con 0x55f67dc89000 2024-05-08T21:23:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff879fcd440 4 waiting for MDS map... 2024-05-08T21:23:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff875774700 1 -- 172.21.15.3:0/1307682848 <== mon.0 v2:172.21.15.3:3300/0 4 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x55f67dcc6c40 con 0x55f67dc89000 2024-05-08T21:23:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff875774700 1 -- 172.21.15.3:0/1307682848 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 21) v1 ==== 3691+0+0 (secure 0 0 0) 0x55f67ea5e300 con 0x55f67dc89000 2024-05-08T21:23:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff879fcd440 4 Got MDS map 21 2024-05-08T21:23:10.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.949+0000 7ff879fcd440 10 main: main 2024-05-08T21:23:10.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.950+0000 7ff879fcd440 4 main: connecting to RADOS... 2024-05-08T21:23:10.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.950+0000 7ff879fcd440 1 Processor -- start 2024-05-08T21:23:10.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.950+0000 7ff879fcd440 1 -- start start 2024-05-08T21:23:10.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff879fcd440 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f67dc89c00 0x55f67dd1cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff879fcd440 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67df98000 0x55f67dd1c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff879fcd440 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f67df98400 0x55f67dd1e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff879fcd440 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f67dcc5e00 con 0x55f67df98000 2024-05-08T21:23:10.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff879fcd440 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f67dd0c640 con 0x55f67df98400 2024-05-08T21:23:10.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff879fcd440 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f67dd0c3c0 con 0x55f67dc89c00 2024-05-08T21:23:10.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff877f79700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67df98000 0x55f67dd1c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff877f79700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67df98000 0x55f67dd1c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54362/0 (socket says 172.21.15.3:54362) 2024-05-08T21:23:10.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff877f79700 1 -- 172.21.15.3:0/2535637069 learned_addr learned my addr 172.21.15.3:0/2535637069 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:10.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff87877a700 1 --2- 172.21.15.3:0/2535637069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f67dc89c00 0x55f67dd1cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff877778700 1 --2- 172.21.15.3:0/2535637069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f67df98400 0x55f67dd1e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff877f79700 1 -- 172.21.15.3:0/2535637069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f67dc89c00 msgr2=0x55f67dd1cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff877f79700 1 --2- 172.21.15.3:0/2535637069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f67dc89c00 0x55f67dd1cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff877f79700 1 -- 172.21.15.3:0/2535637069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f67df98400 msgr2=0x55f67dd1e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff877f79700 1 --2- 172.21.15.3:0/2535637069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f67df98400 0x55f67dd1e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff877f79700 1 -- 172.21.15.3:0/2535637069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f67dcac000 con 0x55f67df98000 2024-05-08T21:23:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff877778700 1 --2- 172.21.15.3:0/2535637069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f67df98400 0x55f67dd1e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.951+0000 7ff877f79700 1 --2- 172.21.15.3:0/2535637069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67df98000 0x55f67dd1c580 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x55f67eaf7bf0 tx=0x55f67ddf2b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24eae462deee5e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.952+0000 7ff873770700 1 -- 172.21.15.3:0/2535637069 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f67dcb8f20 con 0x55f67df98000 2024-05-08T21:23:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.952+0000 7ff873770700 1 -- 172.21.15.3:0/2535637069 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f67ea5c180 con 0x55f67df98000 2024-05-08T21:23:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.952+0000 7ff873770700 1 -- 172.21.15.3:0/2535637069 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f67dcb9600 con 0x55f67df98000 2024-05-08T21:23:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.952+0000 7ff879fcd440 1 -- 172.21.15.3:0/2535637069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67df98000 msgr2=0x55f67dd1c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.952+0000 7ff879fcd440 1 --2- 172.21.15.3:0/2535637069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67df98000 0x55f67dd1c580 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x55f67eaf7bf0 tx=0x55f67ddf2b40 comp rx=0 tx=0).stop 2024-05-08T21:23:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.952+0000 7ff879fcd440 1 -- 172.21.15.3:0/2535637069 shutdown_connections 2024-05-08T21:23:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.952+0000 7ff879fcd440 1 --2- 172.21.15.3:0/2535637069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f67dc89c00 0x55f67dd1cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.952+0000 7ff879fcd440 1 --2- 172.21.15.3:0/2535637069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f67df98400 0x55f67dd1e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.952+0000 7ff879fcd440 1 --2- 172.21.15.3:0/2535637069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67df98000 0x55f67dd1c580 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.952+0000 7ff879fcd440 1 -- 172.21.15.3:0/2535637069 >> 172.21.15.3:0/2535637069 conn(0x55f67dc88400 msgr2=0x55f67dd77800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.952+0000 7ff879fcd440 1 -- 172.21.15.3:0/2535637069 shutdown_connections 2024-05-08T21:23:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.952+0000 7ff879fcd440 1 -- 172.21.15.3:0/2535637069 wait complete. 2024-05-08T21:23:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.953+0000 7ff879fcd440 1 Processor -- start 2024-05-08T21:23:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.953+0000 7ff879fcd440 1 -- start start 2024-05-08T21:23:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.953+0000 7ff879fcd440 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f67dc88400 0x55f67dd1cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.953+0000 7ff879fcd440 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67dfb2000 0x55f67dd1c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.953+0000 7ff879fcd440 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f67dfb2400 0x55f67dd1e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.953+0000 7ff879fcd440 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f67dd0c280 con 0x55f67dfb2000 2024-05-08T21:23:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.953+0000 7ff879fcd440 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f67dd0c140 con 0x55f67dc88400 2024-05-08T21:23:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.953+0000 7ff879fcd440 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f67dd0c000 con 0x55f67dfb2400 2024-05-08T21:23:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff877f79700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67dfb2000 0x55f67dd1c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff877f79700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67dfb2000 0x55f67dd1c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54364/0 (socket says 172.21.15.3:54364) 2024-05-08T21:23:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff877f79700 1 -- 172.21.15.3:0/4156157138 learned_addr learned my addr 172.21.15.3:0/4156157138 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff87877a700 1 --2- 172.21.15.3:0/4156157138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f67dc88400 0x55f67dd1cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff877778700 1 --2- 172.21.15.3:0/4156157138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f67dfb2400 0x55f67dd1e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff877f79700 1 -- 172.21.15.3:0/4156157138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f67dfb2400 msgr2=0x55f67dd1e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff877f79700 1 --2- 172.21.15.3:0/4156157138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f67dfb2400 0x55f67dd1e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff877f79700 1 -- 172.21.15.3:0/4156157138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f67dc88400 msgr2=0x55f67dd1cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff877f79700 1 --2- 172.21.15.3:0/4156157138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f67dc88400 0x55f67dd1cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff877f79700 1 -- 172.21.15.3:0/4156157138 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f67dcac9c0 con 0x55f67dfb2000 2024-05-08T21:23:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff877f79700 1 --2- 172.21.15.3:0/4156157138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67dfb2000 0x55f67dd1c580 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x55f67eaf7e90 tx=0x55f67eb20c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff871f6d700 1 -- 172.21.15.3:0/4156157138 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f67dcb91e0 con 0x55f67dfb2000 2024-05-08T21:23:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff871f6d700 1 -- 172.21.15.3:0/4156157138 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f67ea5c000 con 0x55f67dfb2000 2024-05-08T21:23:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff879fcd440 1 -- 172.21.15.3:0/4156157138 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f67dcacb60 con 0x55f67dfb2000 2024-05-08T21:23:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff871f6d700 1 -- 172.21.15.3:0/4156157138 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f67dcb8dc0 con 0x55f67dfb2000 2024-05-08T21:23:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff879fcd440 1 -- 172.21.15.3:0/4156157138 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f67dcacea0 con 0x55f67dfb2000 2024-05-08T21:23:10.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.954+0000 7ff879fcd440 4 main: resolving pool 6 2024-05-08T21:23:10.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.956+0000 7ff871f6d700 1 -- 172.21.15.3:0/4156157138 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x55f67dcea840 con 0x55f67dfb2000 2024-05-08T21:23:10.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.956+0000 7ff871f6d700 1 --2- 172.21.15.3:0/4156157138 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55f67dfb8800 0x55f67e0ac000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.956+0000 7ff871f6d700 1 -- 172.21.15.3:0/4156157138 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x55f67dcc6a80 con 0x55f67dfb2000 2024-05-08T21:23:10.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.956+0000 7ff879fcd440 4 main: creating IoCtx.. 2024-05-08T21:23:10.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.956+0000 7ff879fcd440 1 --2- 172.21.15.3:0/4156157138 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55f67dfb2800 0x55f67dd1e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:10.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.956+0000 7ff879fcd440 1 -- 172.21.15.3:0/4156157138 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:1 6.b 6:d0630e4c:::mds0_sessionmap:head [omap-clear,omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f67dfb2c00 con 0x55f67dfb2800 2024-05-08T21:23:10.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.956+0000 7ff87877a700 1 --2- 172.21.15.3:0/4156157138 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55f67dfb8800 0x55f67e0ac000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.956+0000 7ff877778700 1 --2- 172.21.15.3:0/4156157138 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55f67dfb2800 0x55f67dd1e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:10.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.957+0000 7ff877778700 1 --2- 172.21.15.3:0/4156157138 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55f67dfb2800 0x55f67dd1e680 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.957+0000 7ff87877a700 1 --2- 172.21.15.3:0/4156157138 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55f67dfb8800 0x55f67e0ac000 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x55f67dd3d470 tx=0x55f67eb20cc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:10.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.958+0000 7ff877778700 1 -- 172.21.15.3:0/4156157138 <== osd.3 v2:172.21.15.3:6800/47118 1 ==== osd_op_reply(1 mds0_sessionmap [omap-clear,omap-set-header] v42'1 uv1 ondisk = 0) v8 ==== 201+0+0 (crc 0 0 0) 0x55f67eaf4b40 con 0x55f67dfb2800 2024-05-08T21:23:10.964 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:23:10.964 INFO:teuthology.orchestra.run.smithi003.stdout: "0": { 2024-05-08T21:23:10.964 INFO:teuthology.orchestra.run.smithi003.stdout: "data": {}, 2024-05-08T21:23:10.964 INFO:teuthology.orchestra.run.smithi003.stdout: "result": 0 2024-05-08T21:23:10.964 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:23:10.964 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:23:10.964 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:10.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.958+0000 7ff879fcd440 1 -- 172.21.15.3:0/4156157138 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55f67dfb2800 msgr2=0x55f67dd1e680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.958+0000 7ff879fcd440 1 --2- 172.21.15.3:0/4156157138 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55f67dfb2800 0x55f67dd1e680 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.958+0000 7ff879fcd440 1 -- 172.21.15.3:0/4156157138 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55f67dfb8800 msgr2=0x55f67e0ac000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.958+0000 7ff879fcd440 1 --2- 172.21.15.3:0/4156157138 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55f67dfb8800 0x55f67e0ac000 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x55f67dd3d470 tx=0x55f67eb20cc0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.958+0000 7ff879fcd440 1 -- 172.21.15.3:0/4156157138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67dfb2000 msgr2=0x55f67dd1c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.958+0000 7ff879fcd440 1 --2- 172.21.15.3:0/4156157138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67dfb2000 0x55f67dd1c580 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x55f67eaf7e90 tx=0x55f67eb20c00 comp rx=0 tx=0).stop 2024-05-08T21:23:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.958+0000 7ff877778700 1 -- 172.21.15.3:0/4156157138 reap_dead start 2024-05-08T21:23:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.958+0000 7ff879fcd440 1 -- 172.21.15.3:0/4156157138 shutdown_connections 2024-05-08T21:23:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.958+0000 7ff879fcd440 1 -- 172.21.15.3:0/4156157138 >> 172.21.15.3:0/4156157138 conn(0x55f67dc89c00 msgr2=0x55f67dd77800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.958+0000 7ff879fcd440 1 -- 172.21.15.3:0/4156157138 shutdown_connections 2024-05-08T21:23:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.958+0000 7ff879fcd440 1 -- 172.21.15.3:0/4156157138 wait complete. 2024-05-08T21:23:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.959+0000 7ff879fcd440 1 -- 172.21.15.3:0/1307682848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67dc89000 msgr2=0x55f67dd1db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.959+0000 7ff879fcd440 1 --2- 172.21.15.3:0/1307682848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67dc89000 0x55f67dd1db80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x55f67eaf66c0 tx=0x55f67ddf2ba0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.959+0000 7ff879fcd440 1 -- 172.21.15.3:0/1307682848 shutdown_connections 2024-05-08T21:23:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.959+0000 7ff879fcd440 1 --2- 172.21.15.3:0/1307682848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f67dc88800 0x55f67dd1d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.959+0000 7ff879fcd440 1 --2- 172.21.15.3:0/1307682848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f67dc89000 0x55f67dd1db80 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.959+0000 7ff879fcd440 1 --2- 172.21.15.3:0/1307682848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f67dc88c00 0x55f67dd1d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.959+0000 7ff879fcd440 1 -- 172.21.15.3:0/1307682848 >> 172.21.15.3:0/1307682848 conn(0x55f67dc89400 msgr2=0x55f67dd77000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.959+0000 7ff879fcd440 1 -- 172.21.15.3:0/1307682848 shutdown_connections 2024-05-08T21:23:10.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:10.959+0000 7ff879fcd440 1 -- 172.21.15.3:0/1307682848 wait complete. 2024-05-08T21:23:10.968 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'recovery_fs:0', 'reset', 'session'] in time 0:00:00.064641, result: { "0": { "data": {}, "result": 0 } } 2024-05-08T21:23:10.968 DEBUG:teuthology.orchestra.run.smithi003:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 recovery_fs:0 reset snap 2024-05-08T21:23:11.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.013+0000 7f6bed8e6440 1 -- start start 2024-05-08T21:23:11.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.013+0000 7f6bed8e6440 1 Processor -- start 2024-05-08T21:23:11.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.013+0000 7f6bed8e6440 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202bd1000 0x55e202c64580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.013+0000 7f6bed8e6440 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e202bd0c00 0x55e202c64b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.013+0000 7f6bed8e6440 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202bd0800 0x55e202c65080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.013+0000 7f6bed8e6440 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e202c54a00 con 0x55e202bd0800 2024-05-08T21:23:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.013+0000 7f6bed8e6440 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e202c54780 con 0x55e202bd1000 2024-05-08T21:23:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.013+0000 7f6bed8e6440 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e202c54640 con 0x55e202bd0c00 2024-05-08T21:23:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6bec093700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202bd0800 0x55e202c65080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6bec093700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202bd0800 0x55e202c65080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54376/0 (socket says 172.21.15.3:54376) 2024-05-08T21:23:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6bec093700 1 -- 172.21.15.3:0/2069409577 learned_addr learned my addr 172.21.15.3:0/2069409577 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6beb892700 1 --2- 172.21.15.3:0/2069409577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202bd1000 0x55e202c64580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6beb091700 1 --2- 172.21.15.3:0/2069409577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e202bd0c00 0x55e202c64b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6bec093700 1 -- 172.21.15.3:0/2069409577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e202bd0c00 msgr2=0x55e202c64b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6bec093700 1 --2- 172.21.15.3:0/2069409577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e202bd0c00 0x55e202c64b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6bec093700 1 -- 172.21.15.3:0/2069409577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202bd1000 msgr2=0x55e202c64580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6bec093700 1 --2- 172.21.15.3:0/2069409577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202bd1000 0x55e202c64580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6bec093700 1 -- 172.21.15.3:0/2069409577 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e202bf49c0 con 0x55e202bd0800 2024-05-08T21:23:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6beb892700 1 --2- 172.21.15.3:0/2069409577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202bd1000 0x55e202c64580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6beb091700 1 --2- 172.21.15.3:0/2069409577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e202bd0c00 0x55e202c64b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6bec093700 1 --2- 172.21.15.3:0/2069409577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202bd0800 0x55e202c65080 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x55e203a36720 tx=0x55e203a3ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1a7b7598d56e8a server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6be908d700 1 -- 172.21.15.3:0/2069409577 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e202c01600 con 0x55e202bd0800 2024-05-08T21:23:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6be908d700 1 -- 172.21.15.3:0/2069409577 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e2039a4000 con 0x55e202bd0800 2024-05-08T21:23:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6be908d700 1 -- 172.21.15.3:0/2069409577 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e202c011e0 con 0x55e202bd0800 2024-05-08T21:23:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6bed8e6440 1 -- 172.21.15.3:0/2069409577 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e202bf4b60 con 0x55e202bd0800 2024-05-08T21:23:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6bed8e6440 1 -- 172.21.15.3:0/2069409577 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55e202bf4340 con 0x55e202bd0800 2024-05-08T21:23:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.014+0000 7f6bed8e6440 4 waiting for MDS map... 2024-05-08T21:23:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.015+0000 7f6be908d700 1 -- 172.21.15.3:0/2069409577 <== mon.0 v2:172.21.15.3:3300/0 4 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x55e202c0ec40 con 0x55e202bd0800 2024-05-08T21:23:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.015+0000 7f6be908d700 1 -- 172.21.15.3:0/2069409577 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 21) v1 ==== 3691+0+0 (secure 0 0 0) 0x55e203a44300 con 0x55e202bd0800 2024-05-08T21:23:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.015+0000 7f6bed8e6440 4 Got MDS map 21 2024-05-08T21:23:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.015+0000 7f6bed8e6440 10 main: main 2024-05-08T21:23:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.015+0000 7f6bed8e6440 4 main: connecting to RADOS... 2024-05-08T21:23:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.015+0000 7f6bed8e6440 1 Processor -- start 2024-05-08T21:23:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.015+0000 7f6bed8e6440 1 -- start start 2024-05-08T21:23:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.016+0000 7f6bed8e6440 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e202bd1c00 0x55e202c65600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.016+0000 7f6bed8e6440 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202ede000 0x55e202c65b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.016+0000 7f6bed8e6440 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202ede400 0x55e202c66100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.016+0000 7f6bed8e6440 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e202c0de00 con 0x55e202ede000 2024-05-08T21:23:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.016+0000 7f6bed8e6440 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e202c54780 con 0x55e202ede400 2024-05-08T21:23:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.016+0000 7f6bed8e6440 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e202c54500 con 0x55e202bd1c00 2024-05-08T21:23:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.016+0000 7f6beb892700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202ede000 0x55e202c65b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.016+0000 7f6beb892700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202ede000 0x55e202c65b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54386/0 (socket says 172.21.15.3:54386) 2024-05-08T21:23:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.016+0000 7f6beb892700 1 -- 172.21.15.3:0/1966572015 learned_addr learned my addr 172.21.15.3:0/1966572015 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.016+0000 7f6beb091700 1 --2- 172.21.15.3:0/1966572015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202ede400 0x55e202c66100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.016+0000 7f6beb892700 1 -- 172.21.15.3:0/1966572015 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e202bd1c00 msgr2=0x55e202c65600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.016+0000 7f6beb892700 1 --2- 172.21.15.3:0/1966572015 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e202bd1c00 0x55e202c65600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.016+0000 7f6beb892700 1 -- 172.21.15.3:0/1966572015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202ede400 msgr2=0x55e202c66100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.016+0000 7f6beb892700 1 --2- 172.21.15.3:0/1966572015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202ede400 0x55e202c66100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.016+0000 7f6beb892700 1 -- 172.21.15.3:0/1966572015 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e202bf4000 con 0x55e202ede000 2024-05-08T21:23:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.017+0000 7f6beb892700 1 --2- 172.21.15.3:0/1966572015 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202ede000 0x55e202c65b80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x55e203a37cb0 tx=0x55e203a5ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f0eb1869f3e4f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.017+0000 7f6be7089700 1 -- 172.21.15.3:0/1966572015 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e202c01600 con 0x55e202ede000 2024-05-08T21:23:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.017+0000 7f6be7089700 1 -- 172.21.15.3:0/1966572015 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e2039a4180 con 0x55e202ede000 2024-05-08T21:23:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.017+0000 7f6be7089700 1 -- 172.21.15.3:0/1966572015 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e202c011e0 con 0x55e202ede000 2024-05-08T21:23:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.017+0000 7f6bed8e6440 1 -- 172.21.15.3:0/1966572015 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202ede000 msgr2=0x55e202c65b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.017+0000 7f6bed8e6440 1 --2- 172.21.15.3:0/1966572015 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202ede000 0x55e202c65b80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x55e203a37cb0 tx=0x55e203a5ec00 comp rx=0 tx=0).stop 2024-05-08T21:23:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.017+0000 7f6bed8e6440 1 -- 172.21.15.3:0/1966572015 shutdown_connections 2024-05-08T21:23:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.017+0000 7f6bed8e6440 1 --2- 172.21.15.3:0/1966572015 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e202bd1c00 0x55e202c65600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.017+0000 7f6bed8e6440 1 --2- 172.21.15.3:0/1966572015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202ede400 0x55e202c66100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.017+0000 7f6bed8e6440 1 --2- 172.21.15.3:0/1966572015 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202ede000 0x55e202c65b80 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.017+0000 7f6bed8e6440 1 -- 172.21.15.3:0/1966572015 >> 172.21.15.3:0/1966572015 conn(0x55e202bd0400 msgr2=0x55e202cad800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.017+0000 7f6bed8e6440 1 -- 172.21.15.3:0/1966572015 shutdown_connections 2024-05-08T21:23:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.017+0000 7f6bed8e6440 1 -- 172.21.15.3:0/1966572015 wait complete. 2024-05-08T21:23:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.018+0000 7f6bed8e6440 1 Processor -- start 2024-05-08T21:23:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.018+0000 7f6bed8e6440 1 -- start start 2024-05-08T21:23:11.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6bed8e6440 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e202bd0400 0x55e202c67180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6bed8e6440 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202f00000 0x55e202c66c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6bed8e6440 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202f00400 0x55e202c66680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6bed8e6440 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e202c543c0 con 0x55e202f00000 2024-05-08T21:23:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6bed8e6440 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e202c54280 con 0x55e202f00400 2024-05-08T21:23:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6bed8e6440 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e202c54140 con 0x55e202bd0400 2024-05-08T21:23:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6beb892700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202f00000 0x55e202c66c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6beb892700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202f00000 0x55e202c66c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54390/0 (socket says 172.21.15.3:54390) 2024-05-08T21:23:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6beb892700 1 -- 172.21.15.3:0/1079325442 learned_addr learned my addr 172.21.15.3:0/1079325442 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6beb091700 1 --2- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e202bd0400 0x55e202c67180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6beb091700 1 --2- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202f00400 0x55e202c66680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6beb892700 1 -- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e202bd0400 msgr2=0x55e202c67180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6beb892700 1 --2- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e202bd0400 0x55e202c67180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6beb892700 1 -- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202f00400 msgr2=0x55e202c66680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6beb892700 1 --2- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202f00400 0x55e202c66680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6beb892700 1 -- 172.21.15.3:0/1079325442 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e202bf49c0 con 0x55e202f00000 2024-05-08T21:23:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6beb091700 1 --2- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202f00400 0x55e202c66680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.019+0000 7f6beb892700 1 --2- 172.21.15.3:0/1079325442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202f00000 0x55e202c66c00 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x55e202efec30 tx=0x55e2039691a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.020+0000 7f6be5886700 1 -- 172.21.15.3:0/1079325442 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e202c00dc0 con 0x55e202f00000 2024-05-08T21:23:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.020+0000 7f6be5886700 1 -- 172.21.15.3:0/1079325442 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e2039a4300 con 0x55e202f00000 2024-05-08T21:23:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.020+0000 7f6bed8e6440 1 -- 172.21.15.3:0/1079325442 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e202bf4b60 con 0x55e202f00000 2024-05-08T21:23:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.020+0000 7f6be5886700 1 -- 172.21.15.3:0/1079325442 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e202c01600 con 0x55e202f00000 2024-05-08T21:23:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.020+0000 7f6bed8e6440 1 -- 172.21.15.3:0/1079325442 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e202bf4ea0 con 0x55e202f00000 2024-05-08T21:23:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.020+0000 7f6bed8e6440 4 main: resolving pool 6 2024-05-08T21:23:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.021+0000 7f6be5886700 1 -- 172.21.15.3:0/1079325442 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x55e202c32840 con 0x55e202f00000 2024-05-08T21:23:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.021+0000 7f6be5886700 1 --2- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55e202f0a400 0x55e202c67700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.021+0000 7f6be5886700 1 -- 172.21.15.3:0/1079325442 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x55e202c0ea80 con 0x55e202f00000 2024-05-08T21:23:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.022+0000 7f6bed8e6440 4 main: creating IoCtx.. 2024-05-08T21:23:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.022+0000 7f6bed8e6440 1 --2- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55e202f00800 0x55e202c66100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.022+0000 7f6bed8e6440 1 -- 172.21.15.3:0/1079325442 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:1 6.d 6:b50e409b:::mds_snaptable:head [writefull 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e202f00c00 con 0x55e202f00800 2024-05-08T21:23:11.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.022+0000 7f6beb091700 1 --2- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55e202f0a400 0x55e202c67700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.022+0000 7f6beb091700 1 --2- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55e202f00800 0x55e202c66100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.022+0000 7f6beb091700 1 --2- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55e202f0a400 0x55e202c67700 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x55e202c85ef0 tx=0x55e203969200 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.023+0000 7f6beb091700 1 --2- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55e202f00800 0x55e202c66100 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.024+0000 7f6beb091700 1 -- 172.21.15.3:0/1079325442 <== osd.5 v2:172.21.15.186:6800/47340 1 ==== osd_op_reply(1 mds_snaptable [writefull 0~70] v42'1 uv1 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x55e2039a2b40 con 0x55e202f00800 2024-05-08T21:23:11.029 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:23:11.029 INFO:teuthology.orchestra.run.smithi003.stdout: "result": 0 2024-05-08T21:23:11.029 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:23:11.029 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:11.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.024+0000 7f6bed8e6440 1 -- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55e202f00800 msgr2=0x55e202c66100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.024+0000 7f6bed8e6440 1 --2- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55e202f00800 0x55e202c66100 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.024+0000 7f6bed8e6440 1 -- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55e202f0a400 msgr2=0x55e202c67700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.024+0000 7f6bed8e6440 1 --2- 172.21.15.3:0/1079325442 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55e202f0a400 0x55e202c67700 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x55e202c85ef0 tx=0x55e203969200 comp rx=0 tx=0).stop 2024-05-08T21:23:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.024+0000 7f6bed8e6440 1 -- 172.21.15.3:0/1079325442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202f00000 msgr2=0x55e202c66c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.024+0000 7f6bed8e6440 1 --2- 172.21.15.3:0/1079325442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202f00000 0x55e202c66c00 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x55e202efec30 tx=0x55e2039691a0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.024+0000 7f6beb892700 1 -- 172.21.15.3:0/1079325442 reap_dead start 2024-05-08T21:23:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.024+0000 7f6bed8e6440 1 -- 172.21.15.3:0/1079325442 shutdown_connections 2024-05-08T21:23:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.024+0000 7f6bed8e6440 1 -- 172.21.15.3:0/1079325442 >> 172.21.15.3:0/1079325442 conn(0x55e202bd1c00 msgr2=0x55e202cad800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.024+0000 7f6bed8e6440 1 -- 172.21.15.3:0/1079325442 shutdown_connections 2024-05-08T21:23:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.024+0000 7f6bed8e6440 1 -- 172.21.15.3:0/1079325442 wait complete. 2024-05-08T21:23:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.025+0000 7f6bed8e6440 1 -- 172.21.15.3:0/2069409577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202bd0800 msgr2=0x55e202c65080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.025+0000 7f6bed8e6440 1 --2- 172.21.15.3:0/2069409577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202bd0800 0x55e202c65080 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x55e203a36720 tx=0x55e203a3ad20 comp rx=0 tx=0).stop 2024-05-08T21:23:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.025+0000 7f6bed8e6440 1 -- 172.21.15.3:0/2069409577 shutdown_connections 2024-05-08T21:23:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.025+0000 7f6bed8e6440 1 --2- 172.21.15.3:0/2069409577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e202bd0800 0x55e202c65080 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.025+0000 7f6bed8e6440 1 --2- 172.21.15.3:0/2069409577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202bd1000 0x55e202c64580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.025+0000 7f6bed8e6440 1 --2- 172.21.15.3:0/2069409577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e202bd0c00 0x55e202c64b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.025+0000 7f6bed8e6440 1 -- 172.21.15.3:0/2069409577 >> 172.21.15.3:0/2069409577 conn(0x55e202bd1400 msgr2=0x55e202cad000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.025+0000 7f6bed8e6440 1 -- 172.21.15.3:0/2069409577 shutdown_connections 2024-05-08T21:23:11.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.025+0000 7f6bed8e6440 1 -- 172.21.15.3:0/2069409577 wait complete. 2024-05-08T21:23:11.033 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'recovery_fs:0', 'reset', 'snap'] in time 0:00:00.064943, result: { "result": 0 } 2024-05-08T21:23:11.033 DEBUG:teuthology.orchestra.run.smithi003:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 recovery_fs:0 reset inode 2024-05-08T21:23:11.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.078+0000 7f2b3caa2440 1 -- start start 2024-05-08T21:23:11.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.078+0000 7f2b3caa2440 1 Processor -- start 2024-05-08T21:23:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3caa2440 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb4e6f000 0x55feb4f02580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3caa2440 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb4e6ec00 0x55feb4f02b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3caa2440 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55feb4e6e800 0x55feb4f03080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3caa2440 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feb4ef2a00 con 0x55feb4e6ec00 2024-05-08T21:23:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3caa2440 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feb4ef2780 con 0x55feb4e6f000 2024-05-08T21:23:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3caa2440 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55feb4ef2640 con 0x55feb4e6e800 2024-05-08T21:23:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3a24d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb4e6ec00 0x55feb4f02b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3a24d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb4e6ec00 0x55feb4f02b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54408/0 (socket says 172.21.15.3:54408) 2024-05-08T21:23:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3a24d700 1 -- 172.21.15.3:0/224011980 learned_addr learned my addr 172.21.15.3:0/224011980 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3b24f700 1 --2- 172.21.15.3:0/224011980 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55feb4e6e800 0x55feb4f03080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3aa4e700 1 --2- 172.21.15.3:0/224011980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb4e6f000 0x55feb4f02580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3a24d700 1 -- 172.21.15.3:0/224011980 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55feb4e6e800 msgr2=0x55feb4f03080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3a24d700 1 --2- 172.21.15.3:0/224011980 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55feb4e6e800 0x55feb4f03080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3a24d700 1 -- 172.21.15.3:0/224011980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb4e6f000 msgr2=0x55feb4f02580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3a24d700 1 --2- 172.21.15.3:0/224011980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb4e6f000 0x55feb4f02580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3a24d700 1 -- 172.21.15.3:0/224011980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55feb4e929c0 con 0x55feb4e6ec00 2024-05-08T21:23:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.079+0000 7f2b3a24d700 1 --2- 172.21.15.3:0/224011980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb4e6ec00 0x55feb4f02b00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x55feb5cd4720 tx=0x55feb5cd8d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91450708d0900428 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.080+0000 7f2b38249700 1 -- 172.21.15.3:0/224011980 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55feb4e9f600 con 0x55feb4e6ec00 2024-05-08T21:23:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.080+0000 7f2b38249700 1 -- 172.21.15.3:0/224011980 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55feb5c32000 con 0x55feb4e6ec00 2024-05-08T21:23:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.080+0000 7f2b38249700 1 -- 172.21.15.3:0/224011980 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55feb4e9f1e0 con 0x55feb4e6ec00 2024-05-08T21:23:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.080+0000 7f2b3caa2440 1 -- 172.21.15.3:0/224011980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55feb4e92b60 con 0x55feb4e6ec00 2024-05-08T21:23:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.080+0000 7f2b3caa2440 1 -- 172.21.15.3:0/224011980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55feb4e92340 con 0x55feb4e6ec00 2024-05-08T21:23:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.080+0000 7f2b3caa2440 4 waiting for MDS map... 2024-05-08T21:23:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.080+0000 7f2b38249700 1 -- 172.21.15.3:0/224011980 <== mon.0 v2:172.21.15.3:3300/0 4 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x55feb4eacc40 con 0x55feb4e6ec00 2024-05-08T21:23:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.080+0000 7f2b38249700 1 -- 172.21.15.3:0/224011980 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 21) v1 ==== 3691+0+0 (secure 0 0 0) 0x55feb5c3a300 con 0x55feb4e6ec00 2024-05-08T21:23:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.080+0000 7f2b3caa2440 4 Got MDS map 21 2024-05-08T21:23:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.080+0000 7f2b3caa2440 10 main: main 2024-05-08T21:23:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.080+0000 7f2b3caa2440 4 main: connecting to RADOS... 2024-05-08T21:23:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.080+0000 7f2b3caa2440 1 Processor -- start 2024-05-08T21:23:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.081+0000 7f2b3caa2440 1 -- start start 2024-05-08T21:23:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.081+0000 7f2b3caa2440 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb4e6fc00 0x55feb4f03600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.081+0000 7f2b3caa2440 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55feb5174000 0x55feb4f03b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.081+0000 7f2b3caa2440 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb5174400 0x55feb4f04100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.081+0000 7f2b3caa2440 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feb4eabe00 con 0x55feb4e6fc00 2024-05-08T21:23:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.081+0000 7f2b3caa2440 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feb4ef2780 con 0x55feb5174400 2024-05-08T21:23:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.081+0000 7f2b3caa2440 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55feb4ef2500 con 0x55feb5174000 2024-05-08T21:23:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.081+0000 7f2b3b24f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb4e6fc00 0x55feb4f03600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.081+0000 7f2b3b24f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb4e6fc00 0x55feb4f03600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54422/0 (socket says 172.21.15.3:54422) 2024-05-08T21:23:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.081+0000 7f2b3b24f700 1 -- 172.21.15.3:0/1696115675 learned_addr learned my addr 172.21.15.3:0/1696115675 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.081+0000 7f2b3aa4e700 1 --2- 172.21.15.3:0/1696115675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55feb5174000 0x55feb4f03b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.081+0000 7f2b3a24d700 1 --2- 172.21.15.3:0/1696115675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb5174400 0x55feb4f04100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.082+0000 7f2b3b24f700 1 -- 172.21.15.3:0/1696115675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55feb5174000 msgr2=0x55feb4f03b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.082+0000 7f2b3b24f700 1 --2- 172.21.15.3:0/1696115675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55feb5174000 0x55feb4f03b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.082+0000 7f2b3b24f700 1 -- 172.21.15.3:0/1696115675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb5174400 msgr2=0x55feb4f04100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.082+0000 7f2b3b24f700 1 --2- 172.21.15.3:0/1696115675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb5174400 0x55feb4f04100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.082+0000 7f2b3b24f700 1 -- 172.21.15.3:0/1696115675 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55feb4e92000 con 0x55feb4e6fc00 2024-05-08T21:23:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.082+0000 7f2b3b24f700 1 --2- 172.21.15.3:0/1696115675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb4e6fc00 0x55feb4f03600 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x55feb5cd5c50 tx=0x55feb5c07200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95899880a2d23b93 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.082+0000 7f2b36245700 1 -- 172.21.15.3:0/1696115675 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55feb4e9f080 con 0x55feb4e6fc00 2024-05-08T21:23:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.082+0000 7f2b36245700 1 -- 172.21.15.3:0/1696115675 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55feb5c32180 con 0x55feb4e6fc00 2024-05-08T21:23:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.082+0000 7f2b36245700 1 -- 172.21.15.3:0/1696115675 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55feb4e9f600 con 0x55feb4e6fc00 2024-05-08T21:23:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.083+0000 7f2b3caa2440 1 -- 172.21.15.3:0/1696115675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb4e6fc00 msgr2=0x55feb4f03600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.083+0000 7f2b3caa2440 1 --2- 172.21.15.3:0/1696115675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb4e6fc00 0x55feb4f03600 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x55feb5cd5c50 tx=0x55feb5c07200 comp rx=0 tx=0).stop 2024-05-08T21:23:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.083+0000 7f2b3caa2440 1 -- 172.21.15.3:0/1696115675 shutdown_connections 2024-05-08T21:23:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.083+0000 7f2b3caa2440 1 --2- 172.21.15.3:0/1696115675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb5174400 0x55feb4f04100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.083+0000 7f2b3caa2440 1 --2- 172.21.15.3:0/1696115675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb4e6fc00 0x55feb4f03600 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.083+0000 7f2b3caa2440 1 --2- 172.21.15.3:0/1696115675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55feb5174000 0x55feb4f03b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.083+0000 7f2b3caa2440 1 -- 172.21.15.3:0/1696115675 >> 172.21.15.3:0/1696115675 conn(0x55feb4e6e400 msgr2=0x55feb4f5d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.083+0000 7f2b3caa2440 1 -- 172.21.15.3:0/1696115675 shutdown_connections 2024-05-08T21:23:11.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.083+0000 7f2b3caa2440 1 -- 172.21.15.3:0/1696115675 wait complete. 2024-05-08T21:23:11.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.083+0000 7f2b3caa2440 1 Processor -- start 2024-05-08T21:23:11.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.083+0000 7f2b3caa2440 1 -- start start 2024-05-08T21:23:11.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.084+0000 7f2b3caa2440 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb4e6e400 0x55feb4f05180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.084+0000 7f2b3caa2440 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55feb5198000 0x55feb4f04c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.084+0000 7f2b3caa2440 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb5198400 0x55feb4f04680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.084+0000 7f2b3caa2440 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feb4ef23c0 con 0x55feb5198400 2024-05-08T21:23:11.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.084+0000 7f2b3caa2440 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feb4ef2280 con 0x55feb4e6e400 2024-05-08T21:23:11.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.084+0000 7f2b3caa2440 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55feb4ef2140 con 0x55feb5198000 2024-05-08T21:23:11.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.084+0000 7f2b3a24d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb5198400 0x55feb4f04680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.084+0000 7f2b3a24d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb5198400 0x55feb4f04680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54432/0 (socket says 172.21.15.3:54432) 2024-05-08T21:23:11.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.084+0000 7f2b3a24d700 1 -- 172.21.15.3:0/3338884673 learned_addr learned my addr 172.21.15.3:0/3338884673 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.084+0000 7f2b3aa4e700 1 --2- 172.21.15.3:0/3338884673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55feb5198000 0x55feb4f04c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.084+0000 7f2b3b24f700 1 --2- 172.21.15.3:0/3338884673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb4e6e400 0x55feb4f05180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.085+0000 7f2b3a24d700 1 -- 172.21.15.3:0/3338884673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55feb5198000 msgr2=0x55feb4f04c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.085+0000 7f2b3a24d700 1 --2- 172.21.15.3:0/3338884673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55feb5198000 0x55feb4f04c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.085+0000 7f2b3a24d700 1 -- 172.21.15.3:0/3338884673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb4e6e400 msgr2=0x55feb4f05180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.085+0000 7f2b3a24d700 1 --2- 172.21.15.3:0/3338884673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb4e6e400 0x55feb4f05180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.085+0000 7f2b3a24d700 1 -- 172.21.15.3:0/3338884673 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55feb4e921a0 con 0x55feb5198400 2024-05-08T21:23:11.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.085+0000 7f2b3a24d700 1 --2- 172.21.15.3:0/3338884673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb5198400 0x55feb4f04680 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x55feb5196c60 tx=0x55feb5cd8c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.085+0000 7f2b34a42700 1 -- 172.21.15.3:0/3338884673 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55feb4e9ef20 con 0x55feb5198400 2024-05-08T21:23:11.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.085+0000 7f2b34a42700 1 -- 172.21.15.3:0/3338884673 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55feb5c32c00 con 0x55feb5198400 2024-05-08T21:23:11.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.085+0000 7f2b3caa2440 1 -- 172.21.15.3:0/3338884673 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55feb4e92000 con 0x55feb5198400 2024-05-08T21:23:11.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.085+0000 7f2b34a42700 1 -- 172.21.15.3:0/3338884673 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55feb4e9f080 con 0x55feb5198400 2024-05-08T21:23:11.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.085+0000 7f2b3caa2440 1 -- 172.21.15.3:0/3338884673 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55feb4e929c0 con 0x55feb5198400 2024-05-08T21:23:11.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.085+0000 7f2b3caa2440 4 main: resolving pool 6 2024-05-08T21:23:11.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.086+0000 7f2b34a42700 1 -- 172.21.15.3:0/3338884673 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x55feb4ed0840 con 0x55feb5198400 2024-05-08T21:23:11.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.087+0000 7f2b34a42700 1 --2- 172.21.15.3:0/3338884673 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55feb519e800 0x55feb4f05700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.087+0000 7f2b34a42700 1 -- 172.21.15.3:0/3338884673 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x55feb4eaca80 con 0x55feb5198400 2024-05-08T21:23:11.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.087+0000 7f2b3caa2440 4 main: creating IoCtx.. 2024-05-08T21:23:11.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.087+0000 7f2b3caa2440 1 --2- 172.21.15.3:0/3338884673 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55feb5198800 0x55feb4f04100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.087+0000 7f2b3caa2440 1 -- 172.21.15.3:0/3338884673 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:1 6.13 6:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55feb5198c00 con 0x55feb5198800 2024-05-08T21:23:11.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.087+0000 7f2b3b24f700 1 --2- 172.21.15.3:0/3338884673 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55feb519e800 0x55feb4f05700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.087+0000 7f2b3aa4e700 1 --2- 172.21.15.3:0/3338884673 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55feb5198800 0x55feb4f04100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.087+0000 7f2b3b24f700 1 --2- 172.21.15.3:0/3338884673 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55feb519e800 0x55feb4f05700 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x55feb5cd5c50 tx=0x55feb5c071a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.088+0000 7f2b3aa4e700 1 --2- 172.21.15.3:0/3338884673 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55feb5198800 0x55feb4f04100 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.089+0000 7f2b3aa4e700 1 -- 172.21.15.3:0/3338884673 <== osd.3 v2:172.21.15.3:6800/47118 1 ==== osd_op_reply(1 mds0_inotable [writefull 0~34] v42'2 uv2 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x55feb5c40b40 con 0x55feb5198800 2024-05-08T21:23:11.096 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:23:11.096 INFO:teuthology.orchestra.run.smithi003.stdout: "0": { 2024-05-08T21:23:11.096 INFO:teuthology.orchestra.run.smithi003.stdout: "data": {}, 2024-05-08T21:23:11.096 INFO:teuthology.orchestra.run.smithi003.stdout: "result": 0 2024-05-08T21:23:11.096 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-05-08T21:23:11.097 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:23:11.097 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:11.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.089+0000 7f2b3caa2440 1 -- 172.21.15.3:0/3338884673 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55feb5198800 msgr2=0x55feb4f04100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.089+0000 7f2b3caa2440 1 --2- 172.21.15.3:0/3338884673 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55feb5198800 0x55feb4f04100 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.089+0000 7f2b3caa2440 1 -- 172.21.15.3:0/3338884673 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55feb519e800 msgr2=0x55feb4f05700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.089+0000 7f2b3caa2440 1 --2- 172.21.15.3:0/3338884673 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55feb519e800 0x55feb4f05700 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x55feb5cd5c50 tx=0x55feb5c071a0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.089+0000 7f2b3caa2440 1 -- 172.21.15.3:0/3338884673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb5198400 msgr2=0x55feb4f04680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.089+0000 7f2b3caa2440 1 --2- 172.21.15.3:0/3338884673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb5198400 0x55feb4f04680 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x55feb5196c60 tx=0x55feb5cd8c60 comp rx=0 tx=0).stop 2024-05-08T21:23:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.089+0000 7f2b3aa4e700 1 -- 172.21.15.3:0/3338884673 reap_dead start 2024-05-08T21:23:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.089+0000 7f2b3caa2440 1 -- 172.21.15.3:0/3338884673 shutdown_connections 2024-05-08T21:23:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.089+0000 7f2b3caa2440 1 -- 172.21.15.3:0/3338884673 >> 172.21.15.3:0/3338884673 conn(0x55feb4e6fc00 msgr2=0x55feb4f5d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.089+0000 7f2b3caa2440 1 -- 172.21.15.3:0/3338884673 shutdown_connections 2024-05-08T21:23:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.089+0000 7f2b3caa2440 1 -- 172.21.15.3:0/3338884673 wait complete. 2024-05-08T21:23:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.090+0000 7f2b3caa2440 1 -- 172.21.15.3:0/224011980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb4e6ec00 msgr2=0x55feb4f02b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.090+0000 7f2b3caa2440 1 --2- 172.21.15.3:0/224011980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb4e6ec00 0x55feb4f02b00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x55feb5cd4720 tx=0x55feb5cd8d20 comp rx=0 tx=0).stop 2024-05-08T21:23:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.090+0000 7f2b3caa2440 1 -- 172.21.15.3:0/224011980 shutdown_connections 2024-05-08T21:23:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.090+0000 7f2b3caa2440 1 --2- 172.21.15.3:0/224011980 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55feb4e6e800 0x55feb4f03080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.090+0000 7f2b3caa2440 1 --2- 172.21.15.3:0/224011980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb4e6f000 0x55feb4f02580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.090+0000 7f2b3caa2440 1 --2- 172.21.15.3:0/224011980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55feb4e6ec00 0x55feb4f02b00 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.090+0000 7f2b3caa2440 1 -- 172.21.15.3:0/224011980 >> 172.21.15.3:0/224011980 conn(0x55feb4e6f400 msgr2=0x55feb4f5d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.090+0000 7f2b3caa2440 1 -- 172.21.15.3:0/224011980 shutdown_connections 2024-05-08T21:23:11.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.090+0000 7f2b3caa2440 1 -- 172.21.15.3:0/224011980 wait complete. 2024-05-08T21:23:11.101 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'recovery_fs:0', 'reset', 'inode'] in time 0:00:00.067807, result: { "0": { "data": {}, "result": 0 } } 2024-05-08T21:23:11.101 DEBUG:teuthology.orchestra.run.smithi003:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_extents --alternate-pool recovery_fs_metadata --filesystem cephfs cephfs_data 2024-05-08T21:23:11.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.150+0000 7fcaf139f940 1 -- start start 2024-05-08T21:23:11.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.150+0000 7fcaf139f940 1 Processor -- start 2024-05-08T21:23:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaf139f940 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f0bbd0d000 0x55f0bbda0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaf139f940 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0bbd0cc00 0x55f0bbda0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaf139f940 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bbd0c800 0x55f0bbda1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaf139f940 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0bbd90a00 con 0x55f0bbd0c800 2024-05-08T21:23:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaf139f940 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0bbd90780 con 0x55f0bbd0cc00 2024-05-08T21:23:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaf139f940 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f0bbd90640 con 0x55f0bbd0d000 2024-05-08T21:23:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaefb4e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bbd0c800 0x55f0bbda1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaefb4e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bbd0c800 0x55f0bbda1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54450/0 (socket says 172.21.15.3:54450) 2024-05-08T21:23:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaefb4e700 1 -- 172.21.15.3:0/4099194915 learned_addr learned my addr 172.21.15.3:0/4099194915 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaef34d700 1 --2- 172.21.15.3:0/4099194915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f0bbd0d000 0x55f0bbda0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaeeb4c700 1 --2- 172.21.15.3:0/4099194915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0bbd0cc00 0x55f0bbda0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaefb4e700 1 -- 172.21.15.3:0/4099194915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f0bbd0d000 msgr2=0x55f0bbda0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaefb4e700 1 --2- 172.21.15.3:0/4099194915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f0bbd0d000 0x55f0bbda0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaefb4e700 1 -- 172.21.15.3:0/4099194915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0bbd0cc00 msgr2=0x55f0bbda0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaefb4e700 1 --2- 172.21.15.3:0/4099194915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0bbd0cc00 0x55f0bbda0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaefb4e700 1 -- 172.21.15.3:0/4099194915 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0bbd309c0 con 0x55f0bbd0c800 2024-05-08T21:23:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaefb4e700 1 --2- 172.21.15.3:0/4099194915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bbd0c800 0x55f0bbda1080 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x55f0bca77200 tx=0x55f0bcad8360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30033f658a9f426 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.151+0000 7fcaecb48700 1 -- 172.21.15.3:0/4099194915 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f0bbd3d600 con 0x55f0bbd0c800 2024-05-08T21:23:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.152+0000 7fcaecb48700 1 -- 172.21.15.3:0/4099194915 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f0bbd5f080 con 0x55f0bbd0c800 2024-05-08T21:23:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.152+0000 7fcaecb48700 1 -- 172.21.15.3:0/4099194915 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f0bbd3d1e0 con 0x55f0bbd0c800 2024-05-08T21:23:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.152+0000 7fcaf139f940 1 -- 172.21.15.3:0/4099194915 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f0bbd30b60 con 0x55f0bbd0c800 2024-05-08T21:23:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.152+0000 7fcaf139f940 1 -- 172.21.15.3:0/4099194915 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55f0bbd30340 con 0x55f0bbd0c800 2024-05-08T21:23:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.152+0000 7fcaf139f940 4 waiting for MDS map... 2024-05-08T21:23:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.152+0000 7fcaecb48700 1 -- 172.21.15.3:0/4099194915 <== mon.0 v2:172.21.15.3:3300/0 4 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x55f0bbd4ac40 con 0x55f0bbd0c800 2024-05-08T21:23:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.152+0000 7fcaecb48700 1 -- 172.21.15.3:0/4099194915 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 21) v1 ==== 3691+0+0 (secure 0 0 0) 0x55f0bcb98000 con 0x55f0bbd0c800 2024-05-08T21:23:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.152+0000 7fcaf139f940 4 Got MDS map 21 2024-05-08T21:23:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.152+0000 7fcaf139f940 4 datascan.main: Using metadata pool output 2024-05-08T21:23:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.152+0000 7fcaf139f940 4 datascan.main: connecting to RADOS... 2024-05-08T21:23:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.153+0000 7fcaf139f940 1 Processor -- start 2024-05-08T21:23:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.153+0000 7fcaf139f940 1 -- start start 2024-05-08T21:23:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.153+0000 7fcaf139f940 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0bbd0dc00 0x55f0bbda1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.153+0000 7fcaf139f940 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bc020000 0x55f0bbda1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.153+0000 7fcaf139f940 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f0bc020400 0x55f0bbda2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.153+0000 7fcaf139f940 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0bbd49e00 con 0x55f0bc020000 2024-05-08T21:23:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.153+0000 7fcaf139f940 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0bbd90780 con 0x55f0bbd0dc00 2024-05-08T21:23:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.153+0000 7fcaf139f940 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f0bbd90500 con 0x55f0bc020400 2024-05-08T21:23:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.153+0000 7fcaef34d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bc020000 0x55f0bbda1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.153+0000 7fcaef34d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bc020000 0x55f0bbda1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54466/0 (socket says 172.21.15.3:54466) 2024-05-08T21:23:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.153+0000 7fcaef34d700 1 -- 172.21.15.3:0/2527928211 learned_addr learned my addr 172.21.15.3:0/2527928211 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.153+0000 7fcaeeb4c700 1 --2- 172.21.15.3:0/2527928211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0bbd0dc00 0x55f0bbda1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.154+0000 7fcaeeb4c700 1 --2- 172.21.15.3:0/2527928211 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f0bc020400 0x55f0bbda2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.154+0000 7fcaef34d700 1 -- 172.21.15.3:0/2527928211 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f0bc020400 msgr2=0x55f0bbda2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.154+0000 7fcaef34d700 1 --2- 172.21.15.3:0/2527928211 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f0bc020400 0x55f0bbda2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.154+0000 7fcaef34d700 1 -- 172.21.15.3:0/2527928211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0bbd0dc00 msgr2=0x55f0bbda1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.154+0000 7fcaef34d700 1 --2- 172.21.15.3:0/2527928211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0bbd0dc00 0x55f0bbda1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.154+0000 7fcaef34d700 1 -- 172.21.15.3:0/2527928211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0bbd30000 con 0x55f0bc020000 2024-05-08T21:23:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.154+0000 7fcaeeb4c700 1 --2- 172.21.15.3:0/2527928211 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f0bc020400 0x55f0bbda2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.154+0000 7fcaef34d700 1 --2- 172.21.15.3:0/2527928211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bc020000 0x55f0bbda1b80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x55f0bc026210 tx=0x55f0bcbaac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=995b781a22e8b003 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.154+0000 7fcaeab44700 1 -- 172.21.15.3:0/2527928211 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f0bbd3d600 con 0x55f0bc020000 2024-05-08T21:23:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.154+0000 7fcaeab44700 1 -- 172.21.15.3:0/2527928211 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f0bcb8c300 con 0x55f0bc020000 2024-05-08T21:23:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.154+0000 7fcaeab44700 1 -- 172.21.15.3:0/2527928211 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f0bbd3d1e0 con 0x55f0bc020000 2024-05-08T21:23:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.154+0000 7fcaf139f940 1 -- 172.21.15.3:0/2527928211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bc020000 msgr2=0x55f0bbda1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.154+0000 7fcaf139f940 1 --2- 172.21.15.3:0/2527928211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bc020000 0x55f0bbda1b80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x55f0bc026210 tx=0x55f0bcbaac00 comp rx=0 tx=0).stop 2024-05-08T21:23:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.155+0000 7fcaf139f940 1 -- 172.21.15.3:0/2527928211 shutdown_connections 2024-05-08T21:23:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.155+0000 7fcaf139f940 1 --2- 172.21.15.3:0/2527928211 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f0bc020400 0x55f0bbda2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.155+0000 7fcaf139f940 1 --2- 172.21.15.3:0/2527928211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0bbd0dc00 0x55f0bbda1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.155+0000 7fcaf139f940 1 --2- 172.21.15.3:0/2527928211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bc020000 0x55f0bbda1b80 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.155+0000 7fcaf139f940 1 -- 172.21.15.3:0/2527928211 >> 172.21.15.3:0/2527928211 conn(0x55f0bbd0c400 msgr2=0x55f0bca8d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.155+0000 7fcaf139f940 1 -- 172.21.15.3:0/2527928211 shutdown_connections 2024-05-08T21:23:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.155+0000 7fcaf139f940 1 -- 172.21.15.3:0/2527928211 wait complete. 2024-05-08T21:23:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.155+0000 7fcaf139f940 1 Processor -- start 2024-05-08T21:23:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.155+0000 7fcaf139f940 1 -- start start 2024-05-08T21:23:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaf139f940 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bbd0c400 0x55f0bbda3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaf139f940 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0bc02a000 0x55f0bbda2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaf139f940 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f0bc02a400 0x55f0bbda2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaf139f940 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0bbd903c0 con 0x55f0bbd0c400 2024-05-08T21:23:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaf139f940 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0bbd90280 con 0x55f0bc02a000 2024-05-08T21:23:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaf139f940 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f0bbd90140 con 0x55f0bc02a400 2024-05-08T21:23:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaeeb4c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bbd0c400 0x55f0bbda3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaeeb4c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bbd0c400 0x55f0bbda3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54474/0 (socket says 172.21.15.3:54474) 2024-05-08T21:23:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 learned_addr learned my addr 172.21.15.3:0/1675240033 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaef34d700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0bc02a000 0x55f0bbda2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaeeb4c700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f0bc02a400 0x55f0bbda2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f0bc02a400 msgr2=0x55f0bbda2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaeeb4c700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f0bc02a400 0x55f0bbda2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0bc02a000 msgr2=0x55f0bbda2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaeeb4c700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0bc02a000 0x55f0bbda2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0bbd309c0 con 0x55f0bbd0c400 2024-05-08T21:23:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.156+0000 7fcaef34d700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0bc02a000 0x55f0bbda2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.157+0000 7fcaeeb4c700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bbd0c400 0x55f0bbda3180 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x55f0bc027470 tx=0x55f0bcad8240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.157+0000 7fcae9341700 1 -- 172.21.15.3:0/1675240033 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f0bbd3d080 con 0x55f0bbd0c400 2024-05-08T21:23:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.157+0000 7fcae9341700 1 -- 172.21.15.3:0/1675240033 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f0bbd5e180 con 0x55f0bbd0c400 2024-05-08T21:23:11.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.157+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f0bbd30b60 con 0x55f0bbd0c400 2024-05-08T21:23:11.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.157+0000 7fcae9341700 1 -- 172.21.15.3:0/1675240033 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f0bbd3d600 con 0x55f0bbd0c400 2024-05-08T21:23:11.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.157+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f0bbd30340 con 0x55f0bbd0c400 2024-05-08T21:23:11.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.157+0000 7fcaf139f940 4 datascan.init: forcing metadata pool 'recovery_fs_metadata' 2024-05-08T21:23:11.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.158+0000 7fcae9341700 1 -- 172.21.15.3:0/1675240033 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x55f0bbd6e840 con 0x55f0bbd0c400 2024-05-08T21:23:11.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.158+0000 7fcae9341700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55f0bc050400 0x55f0bbda3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.159+0000 7fcae9341700 1 -- 172.21.15.3:0/1675240033 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x55f0bbd4aa80 con 0x55f0bbd0c400 2024-05-08T21:23:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.159+0000 7fcaf139f940 4 datascan.main: data pool 'cephfs_data' has ID 5 2024-05-08T21:23:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.159+0000 7fcaf139f940 4 datascan.main: opening data pool 'cephfs_data' 2024-05-08T21:23:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.159+0000 7fcaf139f940 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55f0bc02a800 0x55f0bbda2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.159+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:1 5.0 5:00000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc02ac00 con 0x55f0bc02a800 2024-05-08T21:23:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.159+0000 7fcaef34d700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55f0bc050400 0x55f0bbda3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.159+0000 7fcaeeb4c700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55f0bc02a800 0x55f0bbda2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.159+0000 7fcaef34d700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55f0bc050400 0x55f0bbda3700 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x55f0bc026210 tx=0x55f0bcad8120 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.159+0000 7fcaeeb4c700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55f0bc02a800 0x55f0bbda2100 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.160+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.3 v2:172.21.15.3:6800/47118 1 ==== osd_op_reply(1 [pgnls start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc02a800 2024-05-08T21:23:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.160+0000 7fcaea343700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55f0bc050800 0x55f0bc14c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.160+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:2 5.20 5:04000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc050c00 con 0x55f0bc050800 2024-05-08T21:23:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.160+0000 7fcaefb4e700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55f0bc050800 0x55f0bc14c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.160+0000 7fcaefb4e700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55f0bc050800 0x55f0bc14c000 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.161+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.7 v2:172.21.15.186:6809/47341 1 ==== osd_op_reply(2 [pgnls start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc050800 2024-05-08T21:23:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.161+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:3 5.10 5:08000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc051400 con 0x55f0bc050800 2024-05-08T21:23:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.161+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.7 v2:172.21.15.186:6809/47341 2 ==== osd_op_reply(3 [pgnls start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc050800 2024-05-08T21:23:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.161+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:4 5.30 5:0c000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc051000 con 0x55f0bc050800 2024-05-08T21:23:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.161+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.7 v2:172.21.15.186:6809/47341 3 ==== osd_op_reply(4 [pgnls start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc050800 2024-05-08T21:23:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.161+0000 7fcaeab44700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55f0bc050c00 0x55f0bc14c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.161+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:5 5.8 5:10000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc051400 con 0x55f0bc050c00 2024-05-08T21:23:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.162+0000 7fcaef34d700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55f0bc050c00 0x55f0bc14c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.162+0000 7fcaef34d700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55f0bc050c00 0x55f0bc14c580 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.162+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 <== osd.2 v2:172.21.15.3:6808/47117 1 ==== osd_op_reply(5 [pgnls start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f680 con 0x55f0bc050c00 2024-05-08T21:23:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.162+0000 7fcaea343700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55f0bc051000 0x55f0bc14cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.162+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:6 5.28 5:14000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc051800 con 0x55f0bc051000 2024-05-08T21:23:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.163+0000 7fcaeeb4c700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55f0bc051000 0x55f0bc14cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.163+0000 7fcaeeb4c700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55f0bc051000 0x55f0bc14cb00 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.163+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.5 v2:172.21.15.186:6800/47340 1 ==== osd_op_reply(6 [pgnls start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051000 2024-05-08T21:23:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.163+0000 7fcaeab44700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x55f0bc16a000 0x55f0bc14d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.163+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:7 5.18 5:18000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16a400 con 0x55f0bc16a000 2024-05-08T21:23:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.164+0000 7fcaefb4e700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x55f0bc16a000 0x55f0bc14d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.164+0000 7fcaefb4e700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x55f0bc16a000 0x55f0bc14d600 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.164+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.4 v2:172.21.15.186:6810/47344 1 ==== osd_op_reply(7 [pgnls start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc16a000 2024-05-08T21:23:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.164+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:8 5.38 5:1c000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc051c00 con 0x55f0bc050800 2024-05-08T21:23:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.165+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.7 v2:172.21.15.186:6809/47341 4 ==== osd_op_reply(8 [pgnls start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc050800 2024-05-08T21:23:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.165+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:9 5.4 5:20000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16a800 con 0x55f0bc050800 2024-05-08T21:23:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.165+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.7 v2:172.21.15.186:6809/47341 5 ==== osd_op_reply(9 [pgnls start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc050800 2024-05-08T21:23:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.165+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:10 5.24 5:24000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16ac00 con 0x55f0bc051000 2024-05-08T21:23:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.165+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.5 v2:172.21.15.186:6800/47340 2 ==== osd_op_reply(10 [pgnls start_epoch 42 out=49b] v34'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051000 2024-05-08T21:23:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.165+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16ac00 con 0x55f0bc02a800 2024-05-08T21:23:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.165+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.3 v2:172.21.15.3:6800/47118 2 ==== osd_op_reply(11 [pgnls start_epoch 42 out=49b] v34'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc02a800 2024-05-08T21:23:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.166+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:12 5.34 5:2c000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16b000 con 0x55f0bc051000 2024-05-08T21:23:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.166+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.5 v2:172.21.15.186:6800/47340 3 ==== osd_op_reply(12 [pgnls start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051000 2024-05-08T21:23:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.166+0000 7fcaeab44700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x55f0bc16bc00 0x55f0bc14db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.166+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:13 5.c 5:30000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc051800 con 0x55f0bc16bc00 2024-05-08T21:23:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.166+0000 7fcaef34d700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x55f0bc16bc00 0x55f0bc14db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.166+0000 7fcaef34d700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x55f0bc16bc00 0x55f0bc14db80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.167+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 <== osd.1 v2:172.21.15.3:6816/47119 1 ==== osd_op_reply(13 [pgnls start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f680 con 0x55f0bc16bc00 2024-05-08T21:23:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.167+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:14 5.2c 5:34000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16b400 con 0x55f0bc050c00 2024-05-08T21:23:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.167+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 <== osd.2 v2:172.21.15.3:6808/47117 2 ==== osd_op_reply(14 [pgnls start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f680 con 0x55f0bc050c00 2024-05-08T21:23:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.167+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:15 5.1c 5:38000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc02ac00 con 0x55f0bc16a000 2024-05-08T21:23:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.167+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.4 v2:172.21.15.186:6810/47344 2 ==== osd_op_reply(15 [pgnls start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc16a000 2024-05-08T21:23:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.167+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:16 5.3c 5:3c000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16b800 con 0x55f0bc16bc00 2024-05-08T21:23:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.168+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 <== osd.1 v2:172.21.15.3:6816/47119 2 ==== osd_op_reply(16 [pgnls start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f680 con 0x55f0bc16bc00 2024-05-08T21:23:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.168+0000 7fcaeab44700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x55f0bc051c00 0x55f0bc14e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.168+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:17 5.2 5:40000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc051400 con 0x55f0bc051c00 2024-05-08T21:23:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.168+0000 7fcaeeb4c700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x55f0bc051c00 0x55f0bc14e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.168+0000 7fcaeeb4c700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x55f0bc051c00 0x55f0bc14e100 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.169+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.6 v2:172.21.15.186:6808/47351 1 ==== osd_op_reply(17 [pgnls start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051c00 2024-05-08T21:23:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.169+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:18 5.22 5:44000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc051800 con 0x55f0bc050800 2024-05-08T21:23:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.169+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.7 v2:172.21.15.186:6809/47341 6 ==== osd_op_reply(18 [pgnls start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc050800 2024-05-08T21:23:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.169+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:19 5.12 5:48000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc02ac00 con 0x55f0bc16bc00 2024-05-08T21:23:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.169+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 <== osd.1 v2:172.21.15.3:6816/47119 3 ==== osd_op_reply(19 [pgnls start_epoch 42 out=49b] v34'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f680 con 0x55f0bc16bc00 2024-05-08T21:23:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.169+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:20 5.32 5:4c000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc02ac00 con 0x55f0bc02a800 2024-05-08T21:23:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.169+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.3 v2:172.21.15.3:6800/47118 3 ==== osd_op_reply(20 [pgnls start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc02a800 2024-05-08T21:23:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.170+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:21 5.a 5:50000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16a400 con 0x55f0bc050c00 2024-05-08T21:23:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.170+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 <== osd.2 v2:172.21.15.3:6808/47117 3 ==== osd_op_reply(21 [pgnls start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f680 con 0x55f0bc050c00 2024-05-08T21:23:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.170+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:22 5.2a 5:54000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16b400 con 0x55f0bc16a000 2024-05-08T21:23:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.170+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.4 v2:172.21.15.186:6810/47344 3 ==== osd_op_reply(22 [pgnls start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc16a000 2024-05-08T21:23:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.170+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:23 5.1a 5:58000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16a800 con 0x55f0bc050800 2024-05-08T21:23:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.170+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.7 v2:172.21.15.186:6809/47341 7 ==== osd_op_reply(23 [pgnls start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc050800 2024-05-08T21:23:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.170+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:24 5.3a 5:5c000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16b800 con 0x55f0bc02a800 2024-05-08T21:23:11.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.171+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.3 v2:172.21.15.3:6800/47118 4 ==== osd_op_reply(24 [pgnls start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc02a800 2024-05-08T21:23:11.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.171+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:25 5.6 5:60000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc02ac00 con 0x55f0bc050c00 2024-05-08T21:23:11.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.171+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 <== osd.2 v2:172.21.15.3:6808/47117 4 ==== osd_op_reply(25 [pgnls start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f680 con 0x55f0bc050c00 2024-05-08T21:23:11.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.171+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:26 5.26 5:64000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc178000 con 0x55f0bc02a800 2024-05-08T21:23:11.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.171+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.3 v2:172.21.15.3:6800/47118 5 ==== osd_op_reply(26 [pgnls start_epoch 42 out=49b] v34'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc02a800 2024-05-08T21:23:11.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.171+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:27 5.16 5:68000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc051400 con 0x55f0bc051000 2024-05-08T21:23:11.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.172+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.5 v2:172.21.15.186:6800/47340 4 ==== osd_op_reply(27 [pgnls start_epoch 42 out=49b] v34'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051000 2024-05-08T21:23:11.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.172+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:28 5.36 5:6c000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc178400 con 0x55f0bc16bc00 2024-05-08T21:23:11.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.172+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 <== osd.1 v2:172.21.15.3:6816/47119 4 ==== osd_op_reply(28 [pgnls start_epoch 42 out=49b] v34'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f680 con 0x55f0bc16bc00 2024-05-08T21:23:11.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.172+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:29 5.e 5:70000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16ac00 con 0x55f0bc16a000 2024-05-08T21:23:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.172+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.4 v2:172.21.15.186:6810/47344 4 ==== osd_op_reply(29 [pgnls start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc16a000 2024-05-08T21:23:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.172+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:30 5.2e 5:74000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc178800 con 0x55f0bc02a800 2024-05-08T21:23:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.173+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.3 v2:172.21.15.3:6800/47118 6 ==== osd_op_reply(30 [pgnls start_epoch 42 out=49b] v34'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc02a800 2024-05-08T21:23:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.173+0000 7fcaeab44700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x55f0bc16b000 0x55f0bc14e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.173+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:31 5.1e 5:78000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc179c00 con 0x55f0bc16b000 2024-05-08T21:23:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.173+0000 7fcaefb4e700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x55f0bc16b000 0x55f0bc14e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.173+0000 7fcaefb4e700 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x55f0bc16b000 0x55f0bc14e680 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.173+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.0 v2:172.21.15.3:6824/47120 1 ==== osd_op_reply(31 [pgnls start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc16b000 2024-05-08T21:23:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.174+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:32 5.3e 5:7c000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc178c00 con 0x55f0bc02a800 2024-05-08T21:23:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.174+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.3 v2:172.21.15.3:6800/47118 7 ==== osd_op_reply(32 [pgnls start_epoch 42 out=49b] v34'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc02a800 2024-05-08T21:23:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.174+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:33 5.1 5:80000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc178000 con 0x55f0bc16a000 2024-05-08T21:23:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.174+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.4 v2:172.21.15.186:6810/47344 5 ==== osd_op_reply(33 [pgnls start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc16a000 2024-05-08T21:23:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.174+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:34 5.21 5:84000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc179000 con 0x55f0bc051c00 2024-05-08T21:23:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.175+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.6 v2:172.21.15.186:6808/47351 2 ==== osd_op_reply(34 [pgnls start_epoch 42 out=49b] v34'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051c00 2024-05-08T21:23:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.175+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:35 5.11 5:88000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc178800 con 0x55f0bc051c00 2024-05-08T21:23:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.175+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.6 v2:172.21.15.186:6808/47351 3 ==== osd_op_reply(35 [pgnls start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051c00 2024-05-08T21:23:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.175+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:36 5.31 5:8c000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc179400 con 0x55f0bc051000 2024-05-08T21:23:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.175+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.5 v2:172.21.15.186:6800/47340 5 ==== osd_op_reply(36 [pgnls start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051000 2024-05-08T21:23:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.175+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:37 5.9 5:90000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16b800 con 0x55f0bc050800 2024-05-08T21:23:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.176+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.7 v2:172.21.15.186:6809/47341 8 ==== osd_op_reply(37 [pgnls start_epoch 42 out=49b] v34'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc050800 2024-05-08T21:23:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.176+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:38 5.29 5:94000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc179800 con 0x55f0bc16b000 2024-05-08T21:23:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.176+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.0 v2:172.21.15.3:6824/47120 2 ==== osd_op_reply(38 [pgnls start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc16b000 2024-05-08T21:23:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.176+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:39 5.19 5:98000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16b400 con 0x55f0bc16bc00 2024-05-08T21:23:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.176+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 <== osd.1 v2:172.21.15.3:6816/47119 5 ==== osd_op_reply(39 [pgnls start_epoch 42 out=49b] v34'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f680 con 0x55f0bc16bc00 2024-05-08T21:23:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.176+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:40 5.39 5:9c000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc178400 con 0x55f0bc050800 2024-05-08T21:23:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.177+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.7 v2:172.21.15.186:6809/47341 9 ==== osd_op_reply(40 [pgnls start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc050800 2024-05-08T21:23:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.177+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:41 5.5 5:a0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16a800 con 0x55f0bc16b000 2024-05-08T21:23:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.177+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.0 v2:172.21.15.3:6824/47120 3 ==== osd_op_reply(41 [pgnls start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc16b000 2024-05-08T21:23:11.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.177+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:42 5.25 5:a4000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc178c00 con 0x55f0bc16bc00 2024-05-08T21:23:11.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.177+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 <== osd.1 v2:172.21.15.3:6816/47119 6 ==== osd_op_reply(42 [pgnls start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f680 con 0x55f0bc16bc00 2024-05-08T21:23:11.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.177+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:43 5.15 5:a8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16ac00 con 0x55f0bc051000 2024-05-08T21:23:11.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.178+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.5 v2:172.21.15.186:6800/47340 6 ==== osd_op_reply(43 [pgnls start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051000 2024-05-08T21:23:11.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.178+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:44 5.35 5:ac000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc179c00 con 0x55f0bc050800 2024-05-08T21:23:11.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.178+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.7 v2:172.21.15.186:6809/47341 10 ==== osd_op_reply(44 [pgnls start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc050800 2024-05-08T21:23:11.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.178+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:45 5.d 5:b0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc051400 con 0x55f0bc050c00 2024-05-08T21:23:11.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.179+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 <== osd.2 v2:172.21.15.3:6808/47117 5 ==== osd_op_reply(45 [pgnls start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f680 con 0x55f0bc050c00 2024-05-08T21:23:11.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.179+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:46 5.2d 5:b4000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16a400 con 0x55f0bc02a800 2024-05-08T21:23:11.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.179+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.3 v2:172.21.15.3:6800/47118 8 ==== osd_op_reply(46 [pgnls start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc02a800 2024-05-08T21:23:11.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.179+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:47 5.1d 5:b8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc051800 con 0x55f0bc16bc00 2024-05-08T21:23:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.179+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 <== osd.1 v2:172.21.15.3:6816/47119 7 ==== osd_op_reply(47 [pgnls start_epoch 42 out=49b] v34'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f680 con 0x55f0bc16bc00 2024-05-08T21:23:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.179+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:48 5.3d 5:bc000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16b400 con 0x55f0bc051000 2024-05-08T21:23:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.180+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.5 v2:172.21.15.186:6800/47340 7 ==== osd_op_reply(48 [pgnls start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051000 2024-05-08T21:23:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.180+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:49 5.3 5:c0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc179000 con 0x55f0bc16b000 2024-05-08T21:23:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.180+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.0 v2:172.21.15.3:6824/47120 4 ==== osd_op_reply(49 [pgnls start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc16b000 2024-05-08T21:23:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.180+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:50 5.23 5:c4000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc02ac00 con 0x55f0bc16a000 2024-05-08T21:23:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.180+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.4 v2:172.21.15.186:6810/47344 6 ==== osd_op_reply(50 [pgnls start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc16a000 2024-05-08T21:23:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.180+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:51 5.13 5:c8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc178800 con 0x55f0bc02a800 2024-05-08T21:23:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.181+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.3 v2:172.21.15.3:6800/47118 9 ==== osd_op_reply(51 [pgnls start_epoch 42 out=49b] v34'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc02a800 2024-05-08T21:23:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.181+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:52 5.33 5:cc000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc179800 con 0x55f0bc050c00 2024-05-08T21:23:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.181+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 <== osd.2 v2:172.21.15.3:6808/47117 6 ==== osd_op_reply(52 [pgnls start_epoch 42 out=49b] v34'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f680 con 0x55f0bc050c00 2024-05-08T21:23:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.181+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:53 5.b 5:d0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc179400 con 0x55f0bc050c00 2024-05-08T21:23:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.181+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 <== osd.2 v2:172.21.15.3:6808/47117 7 ==== osd_op_reply(53 [pgnls start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f680 con 0x55f0bc050c00 2024-05-08T21:23:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.181+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:54 5.2b 5:d4000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc178400 con 0x55f0bc16a000 2024-05-08T21:23:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.182+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.4 v2:172.21.15.186:6810/47344 7 ==== osd_op_reply(54 [pgnls start_epoch 42 out=49b] v34'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc16a000 2024-05-08T21:23:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.182+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:55 5.1b 5:d8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc178000 con 0x55f0bc051000 2024-05-08T21:23:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.182+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.5 v2:172.21.15.186:6800/47340 8 ==== osd_op_reply(55 [pgnls start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051000 2024-05-08T21:23:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.182+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:56 5.3b 5:dc000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc179c00 con 0x55f0bc050c00 2024-05-08T21:23:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.182+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 <== osd.2 v2:172.21.15.3:6808/47117 8 ==== osd_op_reply(56 [pgnls start_epoch 42 out=49b] v34'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f680 con 0x55f0bc050c00 2024-05-08T21:23:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.183+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:57 5.7 5:e0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16ac00 con 0x55f0bc051000 2024-05-08T21:23:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.183+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.5 v2:172.21.15.186:6800/47340 9 ==== osd_op_reply(57 [pgnls start_epoch 42 out=81b] v34'8 uv8 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051000 2024-05-08T21:23:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.183+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:58 5.7 5:e2e07930:::10000000001.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc02b000 con 0x55f0bc051000 2024-05-08T21:23:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.183+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.5 v2:172.21.15.186:6800/47340 10 ==== osd_op_reply(58 10000000001.00000000 [stat out=16b] v0'0 uv8 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051000 2024-05-08T21:23:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.183+0000 7fcaf139f940 10 datascan.operator(): handling object 1099511627777.0 2024-05-08T21:23:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.183+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:59 5.7 5:e2e07930:::10000000001.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc02b400 con 0x55f0bc051000 2024-05-08T21:23:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.185+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.5 v2:172.21.15.186:6800/47340 11 ==== osd_op_reply(59 10000000001.00000000 [call] v42'9 uv9 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051000 2024-05-08T21:23:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.185+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:60 5.27 5:e4000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc02b800 con 0x55f0bc16bc00 2024-05-08T21:23:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.186+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 <== osd.1 v2:172.21.15.3:6816/47119 8 ==== osd_op_reply(60 [pgnls start_epoch 42 out=49b] v34'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f680 con 0x55f0bc16bc00 2024-05-08T21:23:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.186+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:61 5.17 5:e8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16a400 con 0x55f0bc02a800 2024-05-08T21:23:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.186+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.3 v2:172.21.15.3:6800/47118 10 ==== osd_op_reply(61 [pgnls start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc02a800 2024-05-08T21:23:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.186+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:62 5.37 5:ec000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc178c00 con 0x55f0bc16a000 2024-05-08T21:23:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.186+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.4 v2:172.21.15.186:6810/47344 8 ==== osd_op_reply(62 [pgnls start_epoch 42 out=49b] v34'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc16a000 2024-05-08T21:23:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.187+0000 7fcaeab44700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:63 5.f 5:f0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16b400 con 0x55f0bc051000 2024-05-08T21:23:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.187+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.5 v2:172.21.15.186:6800/47340 12 ==== osd_op_reply(63 [pgnls start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051000 2024-05-08T21:23:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.187+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:64 5.2f 5:f4000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc179800 con 0x55f0bc050800 2024-05-08T21:23:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.187+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.7 v2:172.21.15.186:6809/47341 11 ==== osd_op_reply(64 [pgnls start_epoch 42 out=81b] v34'9 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc050800 2024-05-08T21:23:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.187+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:65 5.2f 5:f526278d:::10000000001.00000001:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc02bc00 con 0x55f0bc050800 2024-05-08T21:23:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.188+0000 7fcaefb4e700 1 -- 172.21.15.3:0/1675240033 <== osd.7 v2:172.21.15.186:6809/47341 12 ==== osd_op_reply(65 10000000001.00000001 [stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x55f0bc01f440 con 0x55f0bc050800 2024-05-08T21:23:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.188+0000 7fcaf139f940 10 datascan.operator(): handling object 1099511627777.1 2024-05-08T21:23:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.188+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:66 5.7 5:e2e07930:::10000000001.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16ac00 con 0x55f0bc051000 2024-05-08T21:23:11.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.189+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.5 v2:172.21.15.186:6800/47340 13 ==== osd_op_reply(66 10000000001.00000000 [call] v42'10 uv10 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051000 2024-05-08T21:23:11.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.189+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:67 5.1f 5:f8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc16a400 con 0x55f0bc051c00 2024-05-08T21:23:11.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.189+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.6 v2:172.21.15.186:6808/47351 4 ==== osd_op_reply(67 [pgnls start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051c00 2024-05-08T21:23:11.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.189+0000 7fcaea343700 1 -- 172.21.15.3:0/1675240033 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:68 5.3f 5:fc000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f0bc179400 con 0x55f0bc051c00 2024-05-08T21:23:11.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaeeb4c700 1 -- 172.21.15.3:0/1675240033 <== osd.6 v2:172.21.15.186:6808/47351 5 ==== osd_op_reply(68 [pgnls start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55f0bc01f200 con 0x55f0bc051c00 2024-05-08T21:23:11.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x55f0bc16b000 msgr2=0x55f0bc14e680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x55f0bc16b000 0x55f0bc14e680 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x55f0bc16bc00 msgr2=0x55f0bc14db80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x55f0bc16bc00 0x55f0bc14db80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55f0bc050c00 msgr2=0x55f0bc14c580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55f0bc050c00 0x55f0bc14c580 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55f0bc02a800 msgr2=0x55f0bbda2100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55f0bc02a800 0x55f0bbda2100 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x55f0bc16a000 msgr2=0x55f0bc14d600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x55f0bc16a000 0x55f0bc14d600 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55f0bc051000 msgr2=0x55f0bc14cb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55f0bc051000 0x55f0bc14cb00 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x55f0bc051c00 msgr2=0x55f0bc14e100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x55f0bc051c00 0x55f0bc14e100 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55f0bc050800 msgr2=0x55f0bc14c000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55f0bc050800 0x55f0bc14c000 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 reap_dead start 2024-05-08T21:23:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaef34d700 1 -- 172.21.15.3:0/1675240033 reap_dead start 2024-05-08T21:23:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55f0bc050400 msgr2=0x55f0bbda3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55f0bc050400 0x55f0bbda3700 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x55f0bc026210 tx=0x55f0bcad8120 comp rx=0 tx=0).stop 2024-05-08T21:23:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bbd0c400 msgr2=0x55f0bbda3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bbd0c400 0x55f0bbda3180 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x55f0bc027470 tx=0x55f0bcad8240 comp rx=0 tx=0).stop 2024-05-08T21:23:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 shutdown_connections 2024-05-08T21:23:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55f0bc050400 0x55f0bbda3700 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 --2- 172.21.15.3:0/1675240033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bbd0c400 0x55f0bbda3180 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 >> 172.21.15.3:0/1675240033 conn(0x55f0bbd0dc00 msgr2=0x55f0bca8d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 shutdown_connections 2024-05-08T21:23:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.190+0000 7fcaf139f940 1 -- 172.21.15.3:0/1675240033 wait complete. 2024-05-08T21:23:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.191+0000 7fcaf139f940 1 -- 172.21.15.3:0/4099194915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bbd0c800 msgr2=0x55f0bbda1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.191+0000 7fcaf139f940 1 --2- 172.21.15.3:0/4099194915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bbd0c800 0x55f0bbda1080 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x55f0bca77200 tx=0x55f0bcad8360 comp rx=0 tx=0).stop 2024-05-08T21:23:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.191+0000 7fcaf139f940 1 -- 172.21.15.3:0/4099194915 shutdown_connections 2024-05-08T21:23:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.191+0000 7fcaf139f940 1 --2- 172.21.15.3:0/4099194915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f0bbd0d000 0x55f0bbda0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.191+0000 7fcaf139f940 1 --2- 172.21.15.3:0/4099194915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f0bbd0c800 0x55f0bbda1080 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.191+0000 7fcaf139f940 1 --2- 172.21.15.3:0/4099194915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0bbd0cc00 0x55f0bbda0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.191+0000 7fcaf139f940 1 -- 172.21.15.3:0/4099194915 >> 172.21.15.3:0/4099194915 conn(0x55f0bbd0d400 msgr2=0x55f0bca8d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.191+0000 7fcaf139f940 1 -- 172.21.15.3:0/4099194915 shutdown_connections 2024-05-08T21:23:11.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.191+0000 7fcaf139f940 1 -- 172.21.15.3:0/4099194915 wait complete. 2024-05-08T21:23:11.201 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_extents', '--alternate-pool', 'recovery_fs_metadata', '--filesystem', 'cephfs', 'cephfs_data'] in time 0:00:00.100109, result: 2024-05-08T21:23:11.201 DEBUG:teuthology.orchestra.run.smithi003:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_inodes --alternate-pool recovery_fs_metadata --filesystem cephfs --force-corrupt --force-init cephfs_data 2024-05-08T21:23:11.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.254+0000 7fac33fa0940 1 -- start start 2024-05-08T21:23:11.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.254+0000 7fac33fa0940 1 Processor -- start 2024-05-08T21:23:11.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.255+0000 7fac33fa0940 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e47fc21000 0x55e47fcb4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.255+0000 7fac33fa0940 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e47fc20c00 0x55e47fcb4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.255+0000 7fac33fa0940 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47fc20800 0x55e47fcb5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.255+0000 7fac33fa0940 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e47fc5de00 con 0x55e47fc20800 2024-05-08T21:23:11.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.255+0000 7fac33fa0940 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e47fca4a00 con 0x55e47fc21000 2024-05-08T21:23:11.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.255+0000 7fac33fa0940 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e47fca4640 con 0x55e47fc20c00 2024-05-08T21:23:11.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.255+0000 7fac3274f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47fc20800 0x55e47fcb5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.255+0000 7fac3274f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47fc20800 0x55e47fcb5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54486/0 (socket says 172.21.15.3:54486) 2024-05-08T21:23:11.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.255+0000 7fac3274f700 1 -- 172.21.15.3:0/1755035574 learned_addr learned my addr 172.21.15.3:0/1755035574 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.255+0000 7fac31f4e700 1 --2- 172.21.15.3:0/1755035574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e47fc21000 0x55e47fcb4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.255+0000 7fac3174d700 1 --2- 172.21.15.3:0/1755035574 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e47fc20c00 0x55e47fcb4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac3274f700 1 -- 172.21.15.3:0/1755035574 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e47fc20c00 msgr2=0x55e47fcb4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac3274f700 1 --2- 172.21.15.3:0/1755035574 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e47fc20c00 0x55e47fcb4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac3274f700 1 -- 172.21.15.3:0/1755035574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e47fc21000 msgr2=0x55e47fcb4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac3274f700 1 --2- 172.21.15.3:0/1755035574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e47fc21000 0x55e47fcb4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac3274f700 1 -- 172.21.15.3:0/1755035574 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e47fc44b60 con 0x55e47fc20800 2024-05-08T21:23:11.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac31f4e700 1 --2- 172.21.15.3:0/1755035574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e47fc21000 0x55e47fcb4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:11.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac3274f700 1 --2- 172.21.15.3:0/1755035574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47fc20800 0x55e47fcb5080 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x55e48098b2f0 tx=0x55e4809ec000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c917bf8e47e667ce server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac2f749700 1 -- 172.21.15.3:0/1755035574 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e47fc51600 con 0x55e47fc20800 2024-05-08T21:23:11.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac2f749700 1 -- 172.21.15.3:0/1755035574 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e4809f4000 con 0x55e47fc20800 2024-05-08T21:23:11.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac2f749700 1 -- 172.21.15.3:0/1755035574 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e47fc511e0 con 0x55e47fc20800 2024-05-08T21:23:11.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac33fa0940 1 -- 172.21.15.3:0/1755035574 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e47fc441a0 con 0x55e47fc20800 2024-05-08T21:23:11.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac33fa0940 1 -- 172.21.15.3:0/1755035574 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55e47fc44b60 con 0x55e47fc20800 2024-05-08T21:23:11.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac33fa0940 4 waiting for MDS map... 2024-05-08T21:23:11.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac2f749700 1 -- 172.21.15.3:0/1755035574 <== mon.0 v2:172.21.15.3:3300/0 4 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x55e47fc5ec40 con 0x55e47fc20800 2024-05-08T21:23:11.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac2f749700 1 -- 172.21.15.3:0/1755035574 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 21) v1 ==== 3691+0+0 (secure 0 0 0) 0x55e47fc66f00 con 0x55e47fc20800 2024-05-08T21:23:11.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac33fa0940 4 Got MDS map 21 2024-05-08T21:23:11.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac33fa0940 4 datascan.main: Using metadata pool output 2024-05-08T21:23:11.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.256+0000 7fac33fa0940 4 datascan.main: connecting to RADOS... 2024-05-08T21:23:11.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.257+0000 7fac33fa0940 1 Processor -- start 2024-05-08T21:23:11.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.257+0000 7fac33fa0940 1 -- start start 2024-05-08T21:23:11.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac33fa0940 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e47ff2c000 0x55e47fcb5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac33fa0940 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47ff2c400 0x55e47fcb5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac33fa0940 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e47ff2c800 0x55e47fcb6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac33fa0940 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e47fca4500 con 0x55e47ff2c400 2024-05-08T21:23:11.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac33fa0940 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e47fca43c0 con 0x55e47ff2c000 2024-05-08T21:23:11.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac33fa0940 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e47fca4640 con 0x55e47ff2c800 2024-05-08T21:23:11.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac31f4e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47ff2c400 0x55e47fcb5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac31f4e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47ff2c400 0x55e47fcb5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54488/0 (socket says 172.21.15.3:54488) 2024-05-08T21:23:11.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac31f4e700 1 -- 172.21.15.3:0/2339037286 learned_addr learned my addr 172.21.15.3:0/2339037286 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac3174d700 1 --2- 172.21.15.3:0/2339037286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e47ff2c000 0x55e47fcb5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac3174d700 1 --2- 172.21.15.3:0/2339037286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e47ff2c800 0x55e47fcb6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac31f4e700 1 -- 172.21.15.3:0/2339037286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e47ff2c800 msgr2=0x55e47fcb6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac31f4e700 1 --2- 172.21.15.3:0/2339037286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e47ff2c800 0x55e47fcb6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac31f4e700 1 -- 172.21.15.3:0/2339037286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e47ff2c000 msgr2=0x55e47fcb5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac31f4e700 1 --2- 172.21.15.3:0/2339037286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e47ff2c000 0x55e47fcb5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac31f4e700 1 -- 172.21.15.3:0/2339037286 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e47fc44340 con 0x55e47ff2c400 2024-05-08T21:23:11.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.258+0000 7fac3174d700 1 --2- 172.21.15.3:0/2339037286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e47ff2c800 0x55e47fcb6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:11.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.259+0000 7fac31f4e700 1 --2- 172.21.15.3:0/2339037286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47ff2c400 0x55e47fcb5b80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x55e480ab2210 tx=0x55e480ab6d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab36f3c09cc45a server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.259+0000 7fac2d745700 1 -- 172.21.15.3:0/2339037286 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e47fc51600 con 0x55e47ff2c400 2024-05-08T21:23:11.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.259+0000 7fac2d745700 1 -- 172.21.15.3:0/2339037286 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e4809f4480 con 0x55e47ff2c400 2024-05-08T21:23:11.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.259+0000 7fac2d745700 1 -- 172.21.15.3:0/2339037286 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e47fc511e0 con 0x55e47ff2c400 2024-05-08T21:23:11.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.259+0000 7fac33fa0940 1 -- 172.21.15.3:0/2339037286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47ff2c400 msgr2=0x55e47fcb5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.259+0000 7fac33fa0940 1 --2- 172.21.15.3:0/2339037286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47ff2c400 0x55e47fcb5b80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x55e480ab2210 tx=0x55e480ab6d20 comp rx=0 tx=0).stop 2024-05-08T21:23:11.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.259+0000 7fac33fa0940 1 -- 172.21.15.3:0/2339037286 shutdown_connections 2024-05-08T21:23:11.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.259+0000 7fac33fa0940 1 --2- 172.21.15.3:0/2339037286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e47ff2c800 0x55e47fcb6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.259+0000 7fac33fa0940 1 --2- 172.21.15.3:0/2339037286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e47ff2c000 0x55e47fcb5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.259+0000 7fac33fa0940 1 --2- 172.21.15.3:0/2339037286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47ff2c400 0x55e47fcb5b80 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.259+0000 7fac33fa0940 1 -- 172.21.15.3:0/2339037286 >> 172.21.15.3:0/2339037286 conn(0x55e47fc20400 msgr2=0x55e4809a1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.259+0000 7fac33fa0940 1 -- 172.21.15.3:0/2339037286 shutdown_connections 2024-05-08T21:23:11.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.259+0000 7fac33fa0940 1 -- 172.21.15.3:0/2339037286 wait complete. 2024-05-08T21:23:11.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.260+0000 7fac33fa0940 1 Processor -- start 2024-05-08T21:23:11.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.260+0000 7fac33fa0940 1 -- start start 2024-05-08T21:23:11.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac33fa0940 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e47ff2c000 0x55e47fcb7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac33fa0940 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e47ff2c400 0x55e47fcb6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac33fa0940 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47ff2c800 0x55e47fcb6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac33fa0940 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e47fca4280 con 0x55e47ff2c800 2024-05-08T21:23:11.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac33fa0940 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e47fca4140 con 0x55e47ff2c000 2024-05-08T21:23:11.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac33fa0940 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e47fca4000 con 0x55e47ff2c400 2024-05-08T21:23:11.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac3174d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47ff2c800 0x55e47fcb6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac3174d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47ff2c800 0x55e47fcb6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54502/0 (socket says 172.21.15.3:54502) 2024-05-08T21:23:11.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 learned_addr learned my addr 172.21.15.3:0/1294965793 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac3174d700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e47ff2c000 0x55e47fcb7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac31f4e700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e47ff2c400 0x55e47fcb6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e47ff2c400 msgr2=0x55e47fcb6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac3174d700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e47ff2c400 0x55e47fcb6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e47ff2c000 msgr2=0x55e47fcb7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac3174d700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e47ff2c000 0x55e47fcb7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e47fc44000 con 0x55e47ff2c800 2024-05-08T21:23:11.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.261+0000 7fac3174d700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47ff2c800 0x55e47fcb6680 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x55e480ab3320 tx=0x55e4809ec120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.262+0000 7fac2bf42700 1 -- 172.21.15.3:0/1294965793 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e47fc51600 con 0x55e47ff2c800 2024-05-08T21:23:11.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.262+0000 7fac2bf42700 1 -- 172.21.15.3:0/1294965793 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e4809f4600 con 0x55e47ff2c800 2024-05-08T21:23:11.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.262+0000 7fac2bf42700 1 -- 172.21.15.3:0/1294965793 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e47fc511e0 con 0x55e47ff2c800 2024-05-08T21:23:11.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.262+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e47fc44ea0 con 0x55e47ff2c800 2024-05-08T21:23:11.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.262+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e47ff70000 con 0x55e47ff2c800 2024-05-08T21:23:11.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.262+0000 7fac33fa0940 4 datascan.init: forcing metadata pool 'recovery_fs_metadata' 2024-05-08T21:23:11.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.263+0000 7fac2bf42700 1 -- 172.21.15.3:0/1294965793 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x55e47fc82840 con 0x55e47ff2c800 2024-05-08T21:23:11.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.263+0000 7fac2bf42700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55e47ff2dc00 0x55e47fcb7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.263+0000 7fac2bf42700 1 -- 172.21.15.3:0/1294965793 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x55e47fc5ea80 con 0x55e47ff2c800 2024-05-08T21:23:11.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.263+0000 7fac33fa0940 4 datascan.main: data pool 'cephfs_data' has ID 5 2024-05-08T21:23:11.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.263+0000 7fac33fa0940 4 datascan.main: opening data pool 'cephfs_data' 2024-05-08T21:23:11.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.264+0000 7fac33fa0940 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55e47ff2cc00 0x55e47fcb6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.264+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:1 6.14 6:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e47ff2d000 con 0x55e47ff2cc00 2024-05-08T21:23:11.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.264+0000 7fac31f4e700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55e47ff2dc00 0x55e47fcb7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.264+0000 7fac3174d700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55e47ff2cc00 0x55e47fcb6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.264+0000 7fac31f4e700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55e47ff2dc00 0x55e47fcb7700 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x55e480ab2210 tx=0x55e47fc91320 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.264+0000 7fac3174d700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55e47ff2cc00 0x55e47fcb6100 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.264+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 160+0+16 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.264+0000 7fac33fa0940 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55e47ff2d400 0x55e47fcb5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.264+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:2 6.1f 6:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480066000 con 0x55e47ff2d400 2024-05-08T21:23:11.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.265+0000 7fac3274f700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55e47ff2d400 0x55e47fcb5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.265+0000 7fac3274f700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55e47ff2d400 0x55e47fcb5b80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.265+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 1 ==== osd_op_reply(2 100.00000000.inode [stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+16 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.265+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:3 5.0 5:00000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480066400 con 0x55e47ff2d400 2024-05-08T21:23:11.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.266+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 2 ==== osd_op_reply(3 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.266+0000 7fac2cf44700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55e480067c00 0x55e480074000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.266+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:4 5.20 5:04000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480078000 con 0x55e480067c00 2024-05-08T21:23:11.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.266+0000 7fac31f4e700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55e480067c00 0x55e480074000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.266+0000 7fac31f4e700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55e480067c00 0x55e480074000 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.267+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 1 ==== osd_op_reply(4 [pgnls-filter start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.267+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:5 5.10 5:08000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480078800 con 0x55e480067c00 2024-05-08T21:23:11.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.267+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 2 ==== osd_op_reply(5 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.267+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:6 5.30 5:0c000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480078400 con 0x55e480067c00 2024-05-08T21:23:11.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.267+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 3 ==== osd_op_reply(6 [pgnls-filter start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.267+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:7 5.8 5:10000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480078000 con 0x55e47ff2cc00 2024-05-08T21:23:11.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.268+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 2 ==== osd_op_reply(7 [pgnls-filter start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.268+0000 7fac2cf44700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55e480078800 0x55e480074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.268+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:8 5.28 5:14000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480078c00 con 0x55e480078800 2024-05-08T21:23:11.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.268+0000 7fac3174d700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55e480078800 0x55e480074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.269+0000 7fac3174d700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55e480078800 0x55e480074580 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.269+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 1 ==== osd_op_reply(8 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.269+0000 7fac2d745700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x55e480078400 0x55e480075080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.269+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480078000 con 0x55e480078400 2024-05-08T21:23:11.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.269+0000 7fac3274f700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x55e480078400 0x55e480075080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.270+0000 7fac3274f700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x55e480078400 0x55e480075080 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.270+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.4 v2:172.21.15.186:6810/47344 1 ==== osd_op_reply(9 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480078400 2024-05-08T21:23:11.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.270+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:10 5.38 5:1c000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480079000 con 0x55e480067c00 2024-05-08T21:23:11.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.270+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 4 ==== osd_op_reply(10 [pgnls-filter start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.270+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:11 5.4 5:20000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480078c00 con 0x55e480067c00 2024-05-08T21:23:11.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.270+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 5 ==== osd_op_reply(11 [pgnls-filter start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.271+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:12 5.24 5:24000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480078000 con 0x55e480078800 2024-05-08T21:23:11.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.271+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 2 ==== osd_op_reply(12 [pgnls-filter start_epoch 42 out=49b] v34'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.271+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:13 5.14 5:28000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e47ff2d000 con 0x55e47ff2d400 2024-05-08T21:23:11.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.271+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 3 ==== osd_op_reply(13 [pgnls-filter start_epoch 42 out=49b] v34'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.271+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:14 5.34 5:2c000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480079400 con 0x55e480078800 2024-05-08T21:23:11.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.271+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 3 ==== osd_op_reply(14 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.271+0000 7fac2d745700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x55e480079800 0x55e480075600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.271+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:15 5.c 5:30000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480079c00 con 0x55e480079800 2024-05-08T21:23:11.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.272+0000 7fac31f4e700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x55e480079800 0x55e480075600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.272+0000 7fac31f4e700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x55e480079800 0x55e480075600 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.272+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.1 v2:172.21.15.3:6816/47119 1 ==== osd_op_reply(15 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480079800 2024-05-08T21:23:11.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.272+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:16 5.2c 5:34000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480066000 con 0x55e47ff2cc00 2024-05-08T21:23:11.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.272+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 3 ==== osd_op_reply(16 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.273+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:17 5.1c 5:38000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082000 con 0x55e480078400 2024-05-08T21:23:11.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.273+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.4 v2:172.21.15.186:6810/47344 2 ==== osd_op_reply(17 [pgnls-filter start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480078400 2024-05-08T21:23:11.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.273+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:18 5.3c 5:3c000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480066400 con 0x55e480079800 2024-05-08T21:23:11.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.273+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.1 v2:172.21.15.3:6816/47119 2 ==== osd_op_reply(18 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480079800 2024-05-08T21:23:11.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.273+0000 7fac2d745700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x55e480082400 0x55e480075b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.273+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:19 5.2 5:40000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082800 con 0x55e480082400 2024-05-08T21:23:11.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.273+0000 7fac3174d700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x55e480082400 0x55e480075b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.274+0000 7fac3174d700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x55e480082400 0x55e480075b80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.274+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.6 v2:172.21.15.186:6808/47351 1 ==== osd_op_reply(19 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480082400 2024-05-08T21:23:11.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.274+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:20 5.22 5:44000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480078000 con 0x55e480067c00 2024-05-08T21:23:11.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.274+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 6 ==== osd_op_reply(20 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.274+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:21 5.12 5:48000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082c00 con 0x55e480079800 2024-05-08T21:23:11.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.275+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.1 v2:172.21.15.3:6816/47119 3 ==== osd_op_reply(21 [pgnls-filter start_epoch 42 out=49b] v34'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480079800 2024-05-08T21:23:11.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.275+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:22 5.32 5:4c000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480079400 con 0x55e47ff2d400 2024-05-08T21:23:11.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.275+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 4 ==== osd_op_reply(22 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.275+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:23 5.a 5:50000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083000 con 0x55e47ff2cc00 2024-05-08T21:23:11.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.275+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 4 ==== osd_op_reply(23 [pgnls-filter start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.275+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:24 5.2a 5:54000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480079000 con 0x55e480078400 2024-05-08T21:23:11.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.276+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.4 v2:172.21.15.186:6810/47344 3 ==== osd_op_reply(24 [pgnls-filter start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480078400 2024-05-08T21:23:11.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.276+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:25 5.1a 5:58000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082000 con 0x55e480067c00 2024-05-08T21:23:11.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.276+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 7 ==== osd_op_reply(25 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.276+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:26 5.3a 5:5c000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480078c00 con 0x55e47ff2d400 2024-05-08T21:23:11.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.276+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 5 ==== osd_op_reply(26 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.276+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:27 5.6 5:60000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480066000 con 0x55e47ff2cc00 2024-05-08T21:23:11.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.277+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 5 ==== osd_op_reply(27 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.277+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:28 5.26 5:64000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480079c00 con 0x55e47ff2d400 2024-05-08T21:23:11.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.277+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 6 ==== osd_op_reply(28 [pgnls-filter start_epoch 42 out=49b] v34'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.277+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:29 5.16 5:68000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480066400 con 0x55e480078800 2024-05-08T21:23:11.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.277+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 4 ==== osd_op_reply(29 [pgnls-filter start_epoch 42 out=49b] v34'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.277+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:30 5.36 5:6c000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082800 con 0x55e480079800 2024-05-08T21:23:11.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.277+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.1 v2:172.21.15.3:6816/47119 4 ==== osd_op_reply(30 [pgnls-filter start_epoch 42 out=49b] v34'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480079800 2024-05-08T21:23:11.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.277+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:31 5.e 5:70000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480079400 con 0x55e480078400 2024-05-08T21:23:11.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.278+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.4 v2:172.21.15.186:6810/47344 4 ==== osd_op_reply(31 [pgnls-filter start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480078400 2024-05-08T21:23:11.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.278+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:32 5.2e 5:74000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083400 con 0x55e47ff2d400 2024-05-08T21:23:11.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.278+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 7 ==== osd_op_reply(32 [pgnls-filter start_epoch 42 out=49b] v34'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.278+0000 7fac2d745700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x55e480079000 0x55e480076100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.278+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:33 5.1e 5:78000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e47ff2d000 con 0x55e480079000 2024-05-08T21:23:11.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.278+0000 7fac3274f700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x55e480079000 0x55e480076100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.279+0000 7fac3274f700 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x55e480079000 0x55e480076100 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.279+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.0 v2:172.21.15.3:6824/47120 1 ==== osd_op_reply(33 [pgnls-filter start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480079000 2024-05-08T21:23:11.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.279+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:34 5.3e 5:7c000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083800 con 0x55e47ff2d400 2024-05-08T21:23:11.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.279+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 8 ==== osd_op_reply(34 [pgnls-filter start_epoch 42 out=49b] v34'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.279+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:35 5.1 5:80000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e47ff2d000 con 0x55e480078400 2024-05-08T21:23:11.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.280+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.4 v2:172.21.15.186:6810/47344 5 ==== osd_op_reply(35 [pgnls-filter start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480078400 2024-05-08T21:23:11.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.280+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:36 5.21 5:84000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083c00 con 0x55e480082400 2024-05-08T21:23:11.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.280+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.6 v2:172.21.15.186:6808/47351 2 ==== osd_op_reply(36 [pgnls-filter start_epoch 42 out=49b] v34'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480082400 2024-05-08T21:23:11.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.280+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:37 5.11 5:88000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083400 con 0x55e480082400 2024-05-08T21:23:11.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.280+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.6 v2:172.21.15.186:6808/47351 3 ==== osd_op_reply(37 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480082400 2024-05-08T21:23:11.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.281+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:38 5.31 5:8c000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083000 con 0x55e480078800 2024-05-08T21:23:11.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.281+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 5 ==== osd_op_reply(38 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.281+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:39 5.9 5:90000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480078c00 con 0x55e480067c00 2024-05-08T21:23:11.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.281+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 8 ==== osd_op_reply(39 [pgnls-filter start_epoch 42 out=49b] v34'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.281+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:40 5.29 5:94000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083c00 con 0x55e480079000 2024-05-08T21:23:11.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.282+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.0 v2:172.21.15.3:6824/47120 2 ==== osd_op_reply(40 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480079000 2024-05-08T21:23:11.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.282+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:41 5.19 5:98000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480079c00 con 0x55e480079800 2024-05-08T21:23:11.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.282+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.1 v2:172.21.15.3:6816/47119 5 ==== osd_op_reply(41 [pgnls-filter start_epoch 42 out=49b] v34'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480079800 2024-05-08T21:23:11.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.282+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:42 5.39 5:9c000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083400 con 0x55e480067c00 2024-05-08T21:23:11.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.282+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 9 ==== osd_op_reply(42 [pgnls-filter start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.282+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:43 5.5 5:a0000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480079400 con 0x55e480079000 2024-05-08T21:23:11.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.283+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.0 v2:172.21.15.3:6824/47120 3 ==== osd_op_reply(43 [pgnls-filter start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480079000 2024-05-08T21:23:11.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.283+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:44 5.25 5:a4000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480066000 con 0x55e480079800 2024-05-08T21:23:11.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.283+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.1 v2:172.21.15.3:6816/47119 6 ==== osd_op_reply(44 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480079800 2024-05-08T21:23:11.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.283+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:45 5.15 5:a8000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480078000 con 0x55e480078800 2024-05-08T21:23:11.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.283+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 6 ==== osd_op_reply(45 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.283+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:46 5.35 5:ac000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480066400 con 0x55e480067c00 2024-05-08T21:23:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.284+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 10 ==== osd_op_reply(46 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.284+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:47 5.d 5:b0000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480078c00 con 0x55e47ff2cc00 2024-05-08T21:23:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.284+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 6 ==== osd_op_reply(47 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.284+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:48 5.2d 5:b4000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092000 con 0x55e47ff2d400 2024-05-08T21:23:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.284+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 9 ==== osd_op_reply(48 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.284+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:49 5.1d 5:b8000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082c00 con 0x55e480079800 2024-05-08T21:23:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.284+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.1 v2:172.21.15.3:6816/47119 7 ==== osd_op_reply(49 [pgnls-filter start_epoch 42 out=49b] v34'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480079800 2024-05-08T21:23:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.285+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:50 5.3d 5:bc000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092400 con 0x55e480078800 2024-05-08T21:23:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.285+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 7 ==== osd_op_reply(50 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.285+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:51 5.3 5:c0000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082000 con 0x55e480079000 2024-05-08T21:23:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.285+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.0 v2:172.21.15.3:6824/47120 4 ==== osd_op_reply(51 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480079000 2024-05-08T21:23:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.285+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:52 5.23 5:c4000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092800 con 0x55e480078400 2024-05-08T21:23:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.286+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.4 v2:172.21.15.186:6810/47344 6 ==== osd_op_reply(52 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480078400 2024-05-08T21:23:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.286+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:53 5.13 5:c8000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082800 con 0x55e47ff2d400 2024-05-08T21:23:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.286+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 10 ==== osd_op_reply(53 [pgnls-filter start_epoch 42 out=49b] v34'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.286+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:54 5.33 5:cc000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092c00 con 0x55e47ff2cc00 2024-05-08T21:23:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.286+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 7 ==== osd_op_reply(54 [pgnls-filter start_epoch 42 out=49b] v34'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.286+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:55 5.b 5:d0000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083800 con 0x55e47ff2cc00 2024-05-08T21:23:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.286+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 8 ==== osd_op_reply(55 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.286+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:56 5.2b 5:d4000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480093000 con 0x55e480078400 2024-05-08T21:23:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.287+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.4 v2:172.21.15.186:6810/47344 7 ==== osd_op_reply(56 [pgnls-filter start_epoch 42 out=49b] v34'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480078400 2024-05-08T21:23:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.287+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:57 5.1b 5:d8000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083c00 con 0x55e480078800 2024-05-08T21:23:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.287+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 8 ==== osd_op_reply(57 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.287+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:58 5.3b 5:dc000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480093400 con 0x55e47ff2cc00 2024-05-08T21:23:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.288+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 9 ==== osd_op_reply(58 [pgnls-filter start_epoch 42 out=49b] v34'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.288+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:59 5.7 5:e0000000::::head [pgnls-filter start_epoch 42 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082000 con 0x55e480078800 2024-05-08T21:23:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.288+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 9 ==== osd_op_reply(59 [pgnls-filter start_epoch 42 out=81b] v42'10 uv10 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.288+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:60 5.0 5:00000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480066800 con 0x55e47ff2d400 2024-05-08T21:23:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.288+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 11 ==== osd_op_reply(60 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.288+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:61 5.20 5:04000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092000 con 0x55e480067c00 2024-05-08T21:23:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.289+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 11 ==== osd_op_reply(61 [pgnls-filter start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.289+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:62 5.10 5:08000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480093800 con 0x55e480067c00 2024-05-08T21:23:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.289+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 12 ==== osd_op_reply(62 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.289+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:63 5.30 5:0c000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480079400 con 0x55e480067c00 2024-05-08T21:23:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.289+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 13 ==== osd_op_reply(63 [pgnls-filter start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.289+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:64 5.8 5:10000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480093c00 con 0x55e47ff2cc00 2024-05-08T21:23:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.290+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 10 ==== osd_op_reply(64 [pgnls-filter start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.290+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:65 5.28 5:14000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e47ff2d000 con 0x55e480078800 2024-05-08T21:23:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.290+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 10 ==== osd_op_reply(65 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.290+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:66 5.18 5:18000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083400 con 0x55e480078400 2024-05-08T21:23:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.290+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.4 v2:172.21.15.186:6810/47344 8 ==== osd_op_reply(66 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480078400 2024-05-08T21:23:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.290+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:67 5.38 5:1c000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082c00 con 0x55e480067c00 2024-05-08T21:23:11.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.291+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 14 ==== osd_op_reply(67 [pgnls-filter start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.291+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:68 5.4 5:20000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092000 con 0x55e480067c00 2024-05-08T21:23:11.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.291+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 15 ==== osd_op_reply(68 [pgnls-filter start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.291+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:69 5.24 5:24000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083000 con 0x55e480078800 2024-05-08T21:23:11.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.291+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 11 ==== osd_op_reply(69 [pgnls-filter start_epoch 42 out=49b] v34'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.291+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:70 5.14 5:28000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092400 con 0x55e47ff2d400 2024-05-08T21:23:11.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.292+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 12 ==== osd_op_reply(70 [pgnls-filter start_epoch 42 out=49b] v34'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.292+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:71 5.34 5:2c000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083800 con 0x55e480078800 2024-05-08T21:23:11.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.292+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 12 ==== osd_op_reply(71 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.292+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:72 5.c 5:30000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092c00 con 0x55e480079800 2024-05-08T21:23:11.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.292+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.1 v2:172.21.15.3:6816/47119 8 ==== osd_op_reply(72 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480079800 2024-05-08T21:23:11.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.292+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:73 5.2c 5:34000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083c00 con 0x55e47ff2cc00 2024-05-08T21:23:11.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.293+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 11 ==== osd_op_reply(73 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.293+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:74 5.1c 5:38000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480079c00 con 0x55e480078400 2024-05-08T21:23:11.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.293+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.4 v2:172.21.15.186:6810/47344 9 ==== osd_op_reply(74 [pgnls-filter start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480078400 2024-05-08T21:23:11.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.293+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:75 5.3c 5:3c000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082000 con 0x55e480079800 2024-05-08T21:23:11.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.293+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.1 v2:172.21.15.3:6816/47119 9 ==== osd_op_reply(75 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480079800 2024-05-08T21:23:11.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.293+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:76 5.2 5:40000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480078000 con 0x55e480082400 2024-05-08T21:23:11.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.294+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.6 v2:172.21.15.186:6808/47351 4 ==== osd_op_reply(76 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480082400 2024-05-08T21:23:11.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.294+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:77 5.22 5:44000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083000 con 0x55e480067c00 2024-05-08T21:23:11.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.294+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 16 ==== osd_op_reply(77 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.294+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:78 5.12 5:48000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480078c00 con 0x55e480079800 2024-05-08T21:23:11.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.294+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.1 v2:172.21.15.3:6816/47119 10 ==== osd_op_reply(78 [pgnls-filter start_epoch 42 out=49b] v34'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480079800 2024-05-08T21:23:11.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.294+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:79 5.32 5:4c000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083800 con 0x55e47ff2d400 2024-05-08T21:23:11.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.295+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 13 ==== osd_op_reply(79 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.295+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:80 5.a 5:50000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480066000 con 0x55e47ff2cc00 2024-05-08T21:23:11.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.295+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 12 ==== osd_op_reply(80 [pgnls-filter start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.295+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:81 5.2a 5:54000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e47ff2d000 con 0x55e480078400 2024-05-08T21:23:11.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.295+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.4 v2:172.21.15.186:6810/47344 10 ==== osd_op_reply(81 [pgnls-filter start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480078400 2024-05-08T21:23:11.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.295+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:82 5.1a 5:58000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480066400 con 0x55e480067c00 2024-05-08T21:23:11.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.296+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 17 ==== osd_op_reply(82 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.296+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:83 5.3a 5:5c000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480093400 con 0x55e47ff2d400 2024-05-08T21:23:11.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.296+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 14 ==== osd_op_reply(83 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.296+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:84 5.6 5:60000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480093800 con 0x55e47ff2cc00 2024-05-08T21:23:11.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.296+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 13 ==== osd_op_reply(84 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.296+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:85 5.26 5:64000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480093c00 con 0x55e47ff2d400 2024-05-08T21:23:11.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.297+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 15 ==== osd_op_reply(85 [pgnls-filter start_epoch 42 out=49b] v34'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.297+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:86 5.16 5:68000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092000 con 0x55e480078800 2024-05-08T21:23:11.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.297+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 13 ==== osd_op_reply(86 [pgnls-filter start_epoch 42 out=49b] v34'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.297+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:87 5.36 5:6c000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480079400 con 0x55e480079800 2024-05-08T21:23:11.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.298+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.1 v2:172.21.15.3:6816/47119 11 ==== osd_op_reply(87 [pgnls-filter start_epoch 42 out=49b] v34'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480079800 2024-05-08T21:23:11.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.298+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:88 5.e 5:70000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092c00 con 0x55e480078400 2024-05-08T21:23:11.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.298+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.4 v2:172.21.15.186:6810/47344 11 ==== osd_op_reply(88 [pgnls-filter start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480078400 2024-05-08T21:23:11.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.298+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:89 5.2e 5:74000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480079c00 con 0x55e47ff2d400 2024-05-08T21:23:11.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.298+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 16 ==== osd_op_reply(89 [pgnls-filter start_epoch 42 out=49b] v34'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.298+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:90 5.1e 5:78000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092800 con 0x55e480079000 2024-05-08T21:23:11.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.299+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.0 v2:172.21.15.3:6824/47120 5 ==== osd_op_reply(90 [pgnls-filter start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480079000 2024-05-08T21:23:11.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.299+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:91 5.3e 5:7c000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480093000 con 0x55e47ff2d400 2024-05-08T21:23:11.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.299+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 17 ==== osd_op_reply(91 [pgnls-filter start_epoch 42 out=49b] v34'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.299+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:92 5.1 5:80000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082c00 con 0x55e480078400 2024-05-08T21:23:11.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.299+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.4 v2:172.21.15.186:6810/47344 12 ==== osd_op_reply(92 [pgnls-filter start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480078400 2024-05-08T21:23:11.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.299+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:93 5.21 5:84000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092400 con 0x55e480082400 2024-05-08T21:23:11.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.300+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.6 v2:172.21.15.186:6808/47351 5 ==== osd_op_reply(93 [pgnls-filter start_epoch 42 out=49b] v34'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480082400 2024-05-08T21:23:11.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.300+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:94 5.11 5:88000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082000 con 0x55e480082400 2024-05-08T21:23:11.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.300+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.6 v2:172.21.15.186:6808/47351 6 ==== osd_op_reply(94 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480082400 2024-05-08T21:23:11.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.300+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:95 5.31 5:8c000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480093400 con 0x55e480078800 2024-05-08T21:23:11.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.300+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 14 ==== osd_op_reply(95 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.300+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:96 5.9 5:90000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083000 con 0x55e480067c00 2024-05-08T21:23:11.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.301+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 18 ==== osd_op_reply(96 [pgnls-filter start_epoch 42 out=49b] v34'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.301+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:97 5.29 5:94000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480093c00 con 0x55e480079000 2024-05-08T21:23:11.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.301+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.0 v2:172.21.15.3:6824/47120 6 ==== osd_op_reply(97 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480079000 2024-05-08T21:23:11.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.301+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:98 5.19 5:98000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082800 con 0x55e480079800 2024-05-08T21:23:11.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.301+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.1 v2:172.21.15.3:6816/47119 12 ==== osd_op_reply(98 [pgnls-filter start_epoch 42 out=49b] v34'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480079800 2024-05-08T21:23:11.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.301+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:99 5.39 5:9c000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092c00 con 0x55e480067c00 2024-05-08T21:23:11.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.302+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 19 ==== osd_op_reply(99 [pgnls-filter start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.302+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:100 5.5 5:a0000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083400 con 0x55e480079000 2024-05-08T21:23:11.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.302+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.0 v2:172.21.15.3:6824/47120 7 ==== osd_op_reply(100 [pgnls-filter start_epoch 42 out=49b] v34'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480079000 2024-05-08T21:23:11.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.302+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:101 5.25 5:a4000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092800 con 0x55e480079800 2024-05-08T21:23:11.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.302+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.1 v2:172.21.15.3:6816/47119 13 ==== osd_op_reply(101 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480079800 2024-05-08T21:23:11.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.303+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:102 5.15 5:a8000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480066800 con 0x55e480078800 2024-05-08T21:23:11.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.303+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 15 ==== osd_op_reply(102 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.303+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:103 5.35 5:ac000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480093000 con 0x55e480067c00 2024-05-08T21:23:11.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.303+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 20 ==== osd_op_reply(103 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.303+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:104 5.d 5:b0000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092c00 con 0x55e47ff2cc00 2024-05-08T21:23:11.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.304+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 14 ==== osd_op_reply(104 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.304+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:105 5.2d 5:b4000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480079c00 con 0x55e47ff2d400 2024-05-08T21:23:11.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.304+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 18 ==== osd_op_reply(105 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.304+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:106 5.1d 5:b8000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092800 con 0x55e480079800 2024-05-08T21:23:11.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.304+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.1 v2:172.21.15.3:6816/47119 14 ==== osd_op_reply(106 [pgnls-filter start_epoch 42 out=49b] v34'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480079800 2024-05-08T21:23:11.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.304+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:107 5.3d 5:bc000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e47ff2d000 con 0x55e480078800 2024-05-08T21:23:11.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.305+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 16 ==== osd_op_reply(107 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.305+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:108 5.3 5:c0000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083000 con 0x55e480079000 2024-05-08T21:23:11.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.305+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.0 v2:172.21.15.3:6824/47120 8 ==== osd_op_reply(108 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480079000 2024-05-08T21:23:11.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.305+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:109 5.23 5:c4000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083800 con 0x55e480078400 2024-05-08T21:23:11.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.305+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.4 v2:172.21.15.186:6810/47344 13 ==== osd_op_reply(109 [pgnls-filter start_epoch 42 out=49b] v34'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480078400 2024-05-08T21:23:11.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.305+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:110 5.13 5:c8000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082800 con 0x55e47ff2d400 2024-05-08T21:23:11.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.306+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 19 ==== osd_op_reply(110 [pgnls-filter start_epoch 42 out=49b] v34'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.306+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:111 5.33 5:cc000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480093000 con 0x55e47ff2cc00 2024-05-08T21:23:11.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.306+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 15 ==== osd_op_reply(111 [pgnls-filter start_epoch 42 out=49b] v34'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.306+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:112 5.b 5:d0000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083c00 con 0x55e47ff2cc00 2024-05-08T21:23:11.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.306+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 16 ==== osd_op_reply(112 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.306+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:113 5.2b 5:d4000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480093800 con 0x55e480078400 2024-05-08T21:23:11.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.307+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.4 v2:172.21.15.186:6810/47344 14 ==== osd_op_reply(113 [pgnls-filter start_epoch 42 out=49b] v34'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480078400 2024-05-08T21:23:11.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.307+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:114 5.1b 5:d8000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082000 con 0x55e480078800 2024-05-08T21:23:11.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.307+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 17 ==== osd_op_reply(114 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.307+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:115 5.3b 5:dc000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092000 con 0x55e47ff2cc00 2024-05-08T21:23:11.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.307+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.2 v2:172.21.15.3:6808/47117 17 ==== osd_op_reply(115 [pgnls-filter start_epoch 42 out=49b] v34'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e47ff2cc00 2024-05-08T21:23:11.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.307+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:116 5.7 5:e0000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480066400 con 0x55e480078800 2024-05-08T21:23:11.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.308+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 18 ==== osd_op_reply(116 [pgnls-filter start_epoch 42 out=81b] v42'10 uv10 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.308+0000 7fac33fa0940 20 datascan.forall_objects: OSD matched oid 10000000001.00000000 2024-05-08T21:23:11.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.308+0000 7fac33fa0940 10 datascan.operator(): handling object 10000000001.0 2024-05-08T21:23:11.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.308+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:117 5.7 5:e2e07930:::10000000001.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480066c00 con 0x55e480078800 2024-05-08T21:23:11.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.308+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 19 ==== osd_op_reply(117 10000000001.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (95) out=95b,getxattr (30) out=30b,getxattr] v0'0 uv10 ondisk = 0) v8 ==== 416+0+163 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.308+0000 7fac33fa0940 20 datascan.operator(): loaded layout from xattr: pi: 5 os: 4194304 sc: 1 su: 4194304 2024-05-08T21:23:11.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.308+0000 7fac33fa0940 10 datascan.inject_with_backtrace: inode: 0x0x10000000001 2024-05-08T21:23:11.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.308+0000 7fac33fa0940 10 datascan.inject_with_backtrace: backptr: 0x0x10000000000/sixmegs 2024-05-08T21:23:11.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.308+0000 7fac33fa0940 20 datascan.get_frag_of: dirino=0x10000000000 target_dname=sixmegs 2024-05-08T21:23:11.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.308+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:118 6.10 6:0cf6ad0f:::10000000000.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480067000 con 0x55e480079000 2024-05-08T21:23:11.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.309+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.0 v2:172.21.15.3:6824/47120 9 ==== osd_op_reply(118 10000000000.00000000 [getxattr] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480079000 2024-05-08T21:23:11.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.309+0000 7fac33fa0940 4 datascan.get_frag_of: Unexpected error on '10000000000.00000000': (2) No such file or directory 2024-05-08T21:23:11.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.309+0000 7fac33fa0940 20 datascan.inject_with_backtrace: don't know fragment for 0x0x10000000000/sixmegs, will insert to root 2024-05-08T21:23:11.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.309+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:119 6.10 6:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480067400 con 0x55e480079000 2024-05-08T21:23:11.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.309+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.0 v2:172.21.15.3:6824/47120 10 ==== osd_op_reply(119 10000000000.00000000 [omap-get-header] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480079000 2024-05-08T21:23:11.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.309+0000 7fac33fa0940 10 datascan.find_or_create_dirfrag: read_version = 0 2024-05-08T21:23:11.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.309+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:120 6.10 6:0cf6ad0f:::10000000000.00000000:head [create,omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480067800 con 0x55e480079000 2024-05-08T21:23:11.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.310+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.0 v2:172.21.15.3:6824/47120 11 ==== osd_op_reply(120 10000000000.00000000 [create,omap-set-header] v42'1 uv1 ondisk = 0) v8 ==== 206+0+0 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480079000 2024-05-08T21:23:11.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.310+0000 7fac33fa0940 10 datascan.find_or_create_dirfrag: Created dirfrag: 0x0x10000000000 2024-05-08T21:23:11.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.310+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:121 6.10 6:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082c00 con 0x55e480079000 2024-05-08T21:23:11.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.311+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.0 v2:172.21.15.3:6824/47120 12 ==== osd_op_reply(121 10000000000.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480079000 2024-05-08T21:23:11.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.311+0000 7fac33fa0940 20 datascan.read_dentry: oid=10000000000.00000000 dname=sixmegs frag=*, r=0 2024-05-08T21:23:11.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.311+0000 7fac33fa0940 20 datascan.read_dentry: sixmegs_head not found in result 2024-05-08T21:23:11.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.311+0000 7fac33fa0940 10 datascan.inject_with_backtrace: Linking inode 0x0x10000000001 at 0x0x10000000000/sixmegs with size=6291456 bytes 2024-05-08T21:23:11.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.311+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:122 6.10 6:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=508b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083400 con 0x55e480079000 2024-05-08T21:23:11.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.312+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.0 v2:172.21.15.3:6824/47120 13 ==== osd_op_reply(122 10000000000.00000000 [omap-set-vals] v42'2 uv2 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480079000 2024-05-08T21:23:11.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.312+0000 7fac33fa0940 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/sixmegs pointing to 0x0x10000000001 2024-05-08T21:23:11.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.312+0000 7fac33fa0940 10 datascan.inject_with_backtrace: backptr: 0x0x1/subdir 2024-05-08T21:23:11.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.312+0000 7fac33fa0940 20 datascan.get_frag_of: dirino=0x1 target_dname=subdir 2024-05-08T21:23:11.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.312+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:123 6.1f 6:ff5b34d6:::1.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083000 con 0x55e47ff2d400 2024-05-08T21:23:11.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.312+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 20 ==== osd_op_reply(123 1.00000000 [getxattr] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.312+0000 7fac33fa0940 4 datascan.get_frag_of: Unexpected error on '1.00000000': (2) No such file or directory 2024-05-08T21:23:11.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.312+0000 7fac33fa0940 20 datascan.inject_with_backtrace: don't know fragment for 0x0x1/subdir, will insert to root 2024-05-08T21:23:11.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.312+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:124 6.1f 6:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083800 con 0x55e47ff2d400 2024-05-08T21:23:11.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.312+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 21 ==== osd_op_reply(124 1.00000000 [omap-get-header] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.312+0000 7fac33fa0940 10 datascan.find_or_create_dirfrag: read_version = 0 2024-05-08T21:23:11.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.312+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:125 6.1f 6:ff5b34d6:::1.00000000:head [create,omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480082800 con 0x55e47ff2d400 2024-05-08T21:23:11.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.314+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 22 ==== osd_op_reply(125 1.00000000 [create,omap-set-header] v42'2 uv2 ondisk = 0) v8 ==== 196+0+0 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.314+0000 7fac33fa0940 10 datascan.find_or_create_dirfrag: Created dirfrag: 0x0x1 2024-05-08T21:23:11.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.314+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:126 6.1f 6:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480083c00 con 0x55e47ff2d400 2024-05-08T21:23:11.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.314+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 23 ==== osd_op_reply(126 1.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv2 ondisk = 0) v8 ==== 154+0+4 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.314+0000 7fac33fa0940 20 datascan.read_dentry: oid=1.00000000 dname=subdir frag=*, r=0 2024-05-08T21:23:11.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.314+0000 7fac33fa0940 20 datascan.read_dentry: subdir_head not found in result 2024-05-08T21:23:11.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.314+0000 7fac33fa0940 10 datascan.inject_with_backtrace: Linking ancestor directory of inode 0x0x10000000000 at 0x0x1:subdir 2024-05-08T21:23:11.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.314+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:127 6.1f 6:ff5b34d6:::1.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e47ff2d000 con 0x55e47ff2d400 2024-05-08T21:23:11.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.316+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 24 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v42'3 uv3 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.316+0000 7fac33fa0940 20 datascan.inject_linkage: Injected dentry 0x0x1/subdir pointing to 0x0x10000000000 2024-05-08T21:23:11.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.316+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:128 5.27 5:e4000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480079c00 con 0x55e480079800 2024-05-08T21:23:11.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.317+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.1 v2:172.21.15.3:6816/47119 15 ==== osd_op_reply(128 [pgnls-filter start_epoch 42 out=49b] v34'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480079800 2024-05-08T21:23:11.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.317+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:129 5.17 5:e8000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480066000 con 0x55e47ff2d400 2024-05-08T21:23:11.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.317+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.3 v2:172.21.15.3:6800/47118 25 ==== osd_op_reply(129 [pgnls-filter start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e47ff2d400 2024-05-08T21:23:11.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.317+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:130 5.37 5:ec000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480092400 con 0x55e480078400 2024-05-08T21:23:11.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.317+0000 7fac3274f700 1 -- 172.21.15.3:0/1294965793 <== osd.4 v2:172.21.15.186:6810/47344 15 ==== osd_op_reply(130 [pgnls-filter start_epoch 42 out=49b] v34'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b440 con 0x55e480078400 2024-05-08T21:23:11.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.317+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:131 5.f 5:f0000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480078c00 con 0x55e480078800 2024-05-08T21:23:11.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.318+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.5 v2:172.21.15.186:6800/47340 20 ==== osd_op_reply(131 [pgnls-filter start_epoch 42 out=49b] v34'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480078800 2024-05-08T21:23:11.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.318+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:132 5.2f 5:f4000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480093400 con 0x55e480067c00 2024-05-08T21:23:11.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.318+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 <== osd.7 v2:172.21.15.186:6809/47341 21 ==== osd_op_reply(132 [pgnls-filter start_epoch 42 out=49b] v34'9 uv1 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b680 con 0x55e480067c00 2024-05-08T21:23:11.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.318+0000 7fac2cf44700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:133 5.1f 5:f8000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480079400 con 0x55e480082400 2024-05-08T21:23:11.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.318+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.6 v2:172.21.15.186:6808/47351 7 ==== osd_op_reply(133 [pgnls-filter start_epoch 42 out=49b] v34'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480082400 2024-05-08T21:23:11.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.318+0000 7fac2d745700 1 -- 172.21.15.3:0/1294965793 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- osd_op(unknown.0.0:134 5.3f 5:fc000000::::head [pgnls-filter start_epoch 42 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55e480067400 con 0x55e480082400 2024-05-08T21:23:11.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac3174d700 1 -- 172.21.15.3:0/1294965793 <== osd.6 v2:172.21.15.186:6808/47351 8 ==== osd_op_reply(134 [pgnls-filter start_epoch 42 out=49b] v34'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e47ff2b200 con 0x55e480082400 2024-05-08T21:23:11.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x55e480079000 msgr2=0x55e480076100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x55e480079000 0x55e480076100 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x55e480079800 msgr2=0x55e480075600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x55e480079800 0x55e480075600 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55e47ff2cc00 msgr2=0x55e47fcb6100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55e47ff2cc00 0x55e47fcb6100 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55e47ff2d400 msgr2=0x55e47fcb5b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55e47ff2d400 0x55e47fcb5b80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x55e480078400 msgr2=0x55e480075080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x55e480078400 0x55e480075080 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55e480078800 msgr2=0x55e480074580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 reap_dead start 2024-05-08T21:23:11.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55e480078800 0x55e480074580 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x55e480082400 msgr2=0x55e480075b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x55e480082400 0x55e480075b80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55e480067c00 msgr2=0x55e480074000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55e480067c00 0x55e480074000 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55e47ff2dc00 msgr2=0x55e47fcb7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55e47ff2dc00 0x55e47fcb7700 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x55e480ab2210 tx=0x55e47fc91320 comp rx=0 tx=0).stop 2024-05-08T21:23:11.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47ff2c800 msgr2=0x55e47fcb6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 --2- 172.21.15.3:0/1294965793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47ff2c800 0x55e47fcb6680 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x55e480ab3320 tx=0x55e4809ec120 comp rx=0 tx=0).stop 2024-05-08T21:23:11.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac31f4e700 1 -- 172.21.15.3:0/1294965793 reap_dead start 2024-05-08T21:23:11.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 shutdown_connections 2024-05-08T21:23:11.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 >> 172.21.15.3:0/1294965793 conn(0x55e47fc20400 msgr2=0x55e4809a1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 shutdown_connections 2024-05-08T21:23:11.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.319+0000 7fac33fa0940 1 -- 172.21.15.3:0/1294965793 wait complete. 2024-05-08T21:23:11.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.320+0000 7fac33fa0940 1 -- 172.21.15.3:0/1755035574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47fc20800 msgr2=0x55e47fcb5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.320+0000 7fac33fa0940 1 --2- 172.21.15.3:0/1755035574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47fc20800 0x55e47fcb5080 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x55e48098b2f0 tx=0x55e4809ec000 comp rx=0 tx=0).stop 2024-05-08T21:23:11.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.320+0000 7fac33fa0940 1 -- 172.21.15.3:0/1755035574 shutdown_connections 2024-05-08T21:23:11.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.320+0000 7fac33fa0940 1 --2- 172.21.15.3:0/1755035574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e47fc20800 0x55e47fcb5080 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.320+0000 7fac33fa0940 1 --2- 172.21.15.3:0/1755035574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e47fc21000 0x55e47fcb4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.320+0000 7fac33fa0940 1 --2- 172.21.15.3:0/1755035574 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e47fc20c00 0x55e47fcb4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.320+0000 7fac33fa0940 1 -- 172.21.15.3:0/1755035574 >> 172.21.15.3:0/1755035574 conn(0x55e47fc21400 msgr2=0x55e4809a1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.320+0000 7fac33fa0940 1 -- 172.21.15.3:0/1755035574 shutdown_connections 2024-05-08T21:23:11.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.320+0000 7fac33fa0940 1 -- 172.21.15.3:0/1755035574 wait complete. 2024-05-08T21:23:11.327 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_inodes', '--alternate-pool', 'recovery_fs_metadata', '--filesystem', 'cephfs', '--force-corrupt', '--force-init', 'cephfs_data'] in time 0:00:00.125616, result: 2024-05-08T21:23:11.327 DEBUG:teuthology.orchestra.run.smithi003:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_links --filesystem recovery_fs 2024-05-08T21:23:11.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.375+0000 7fc801fa7940 1 -- start start 2024-05-08T21:23:11.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.375+0000 7fc801fa7940 1 Processor -- start 2024-05-08T21:23:11.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc801fa7940 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a88890b000 0x55a88899e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc801fa7940 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a88890ac00 0x55a88899eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc801fa7940 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a88890a800 0x55a88899f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc801fa7940 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a88898e780 con 0x55a88890b000 2024-05-08T21:23:11.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc801fa7940 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a88898ea00 con 0x55a88890a800 2024-05-08T21:23:11.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc801fa7940 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a88898e640 con 0x55a88890ac00 2024-05-08T21:23:11.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc7fff55700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a88890b000 0x55a88899e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc800756700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a88890a800 0x55a88899f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc7ff754700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a88890ac00 0x55a88899eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc7fff55700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a88890b000 0x55a88899e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57460/0 (socket says 172.21.15.3:57460) 2024-05-08T21:23:11.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc7fff55700 1 -- 172.21.15.3:0/3072582739 learned_addr learned my addr 172.21.15.3:0/3072582739 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc7ff754700 1 -- 172.21.15.3:0/3072582739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a88890a800 msgr2=0x55a88899f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc7ff754700 1 --2- 172.21.15.3:0/3072582739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a88890a800 0x55a88899f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc7ff754700 1 -- 172.21.15.3:0/3072582739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a88890b000 msgr2=0x55a88899e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc7ff754700 1 --2- 172.21.15.3:0/3072582739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a88890b000 0x55a88899e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc7ff754700 1 -- 172.21.15.3:0/3072582739 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a88892e9c0 con 0x55a88890ac00 2024-05-08T21:23:11.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.376+0000 7fc7ff754700 1 --2- 172.21.15.3:0/3072582739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a88890ac00 0x55a88899eb00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55a889675350 tx=0x55a8896da360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8fbf362c6638317 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.377+0000 7fc7fd750700 1 -- 172.21.15.3:0/3072582739 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55a88893b600 con 0x55a88890ac00 2024-05-08T21:23:11.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.377+0000 7fc7fd750700 1 -- 172.21.15.3:0/3072582739 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a889782000 con 0x55a88890ac00 2024-05-08T21:23:11.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.377+0000 7fc7fd750700 1 -- 172.21.15.3:0/3072582739 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55a88893b1e0 con 0x55a88890ac00 2024-05-08T21:23:11.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.377+0000 7fc801fa7940 1 -- 172.21.15.3:0/3072582739 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a88892eb60 con 0x55a88890ac00 2024-05-08T21:23:11.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.377+0000 7fc801fa7940 1 -- 172.21.15.3:0/3072582739 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55a88892e340 con 0x55a88890ac00 2024-05-08T21:23:11.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.377+0000 7fc801fa7940 4 waiting for MDS map... 2024-05-08T21:23:11.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.377+0000 7fc7fd750700 1 -- 172.21.15.3:0/3072582739 <== mon.2 v2:172.21.15.186:3301/0 4 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x55a888948c40 con 0x55a88890ac00 2024-05-08T21:23:11.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.377+0000 7fc7fd750700 1 -- 172.21.15.3:0/3072582739 <== mon.2 v2:172.21.15.186:3301/0 5 ==== fsmap(e 21) v1 ==== 3691+0+0 (secure 0 0 0) 0x55a889786000 con 0x55a88890ac00 2024-05-08T21:23:11.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.377+0000 7fc801fa7940 4 Got MDS map 21 2024-05-08T21:23:11.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.377+0000 7fc801fa7940 4 datascan.main: Using metadata pool output 2024-05-08T21:23:11.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.377+0000 7fc801fa7940 4 datascan.main: connecting to RADOS... 2024-05-08T21:23:11.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.378+0000 7fc801fa7940 1 Processor -- start 2024-05-08T21:23:11.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.378+0000 7fc801fa7940 1 -- start start 2024-05-08T21:23:11.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc801fa7940 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a88890bc00 0x55a88899f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc801fa7940 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a888c2c000 0x55a88899fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc801fa7940 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a888c2c400 0x55a8889a0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc801fa7940 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a888947e00 con 0x55a88890bc00 2024-05-08T21:23:11.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc801fa7940 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a88898e780 con 0x55a888c2c400 2024-05-08T21:23:11.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc801fa7940 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a88898e500 con 0x55a888c2c000 2024-05-08T21:23:11.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc800756700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a88890bc00 0x55a88899f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc800756700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a88890bc00 0x55a88899f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57464/0 (socket says 172.21.15.3:57464) 2024-05-08T21:23:11.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc800756700 1 -- 172.21.15.3:0/2445796729 learned_addr learned my addr 172.21.15.3:0/2445796729 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc7ff754700 1 --2- 172.21.15.3:0/2445796729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a888c2c400 0x55a8889a0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc7fff55700 1 --2- 172.21.15.3:0/2445796729 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a888c2c000 0x55a88899fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc800756700 1 -- 172.21.15.3:0/2445796729 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a888c2c000 msgr2=0x55a88899fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc800756700 1 --2- 172.21.15.3:0/2445796729 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a888c2c000 0x55a88899fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc800756700 1 -- 172.21.15.3:0/2445796729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a888c2c400 msgr2=0x55a8889a0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc800756700 1 --2- 172.21.15.3:0/2445796729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a888c2c400 0x55a8889a0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc800756700 1 -- 172.21.15.3:0/2445796729 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a88892e000 con 0x55a88890bc00 2024-05-08T21:23:11.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.379+0000 7fc800756700 1 --2- 172.21.15.3:0/2445796729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a88890bc00 0x55a88899f600 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x55a8897a41b0 tx=0x55a8896da060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe48347073ca012 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.380+0000 7fc7fb74c700 1 -- 172.21.15.3:0/2445796729 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55a88893b080 con 0x55a88890bc00 2024-05-08T21:23:11.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.380+0000 7fc7fb74c700 1 -- 172.21.15.3:0/2445796729 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a889782480 con 0x55a88890bc00 2024-05-08T21:23:11.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.380+0000 7fc7fb74c700 1 -- 172.21.15.3:0/2445796729 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55a88893b600 con 0x55a88890bc00 2024-05-08T21:23:11.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.380+0000 7fc801fa7940 1 -- 172.21.15.3:0/2445796729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a88890bc00 msgr2=0x55a88899f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.380+0000 7fc801fa7940 1 --2- 172.21.15.3:0/2445796729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a88890bc00 0x55a88899f600 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x55a8897a41b0 tx=0x55a8896da060 comp rx=0 tx=0).stop 2024-05-08T21:23:11.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.380+0000 7fc801fa7940 1 -- 172.21.15.3:0/2445796729 shutdown_connections 2024-05-08T21:23:11.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.380+0000 7fc801fa7940 1 --2- 172.21.15.3:0/2445796729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a888c2c400 0x55a8889a0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.380+0000 7fc801fa7940 1 --2- 172.21.15.3:0/2445796729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a88890bc00 0x55a88899f600 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.380+0000 7fc801fa7940 1 --2- 172.21.15.3:0/2445796729 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a888c2c000 0x55a88899fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.380+0000 7fc801fa7940 1 -- 172.21.15.3:0/2445796729 >> 172.21.15.3:0/2445796729 conn(0x55a88890a400 msgr2=0x55a88968b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.380+0000 7fc801fa7940 1 -- 172.21.15.3:0/2445796729 shutdown_connections 2024-05-08T21:23:11.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.380+0000 7fc801fa7940 1 -- 172.21.15.3:0/2445796729 wait complete. 2024-05-08T21:23:11.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.381+0000 7fc801fa7940 1 Processor -- start 2024-05-08T21:23:11.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.381+0000 7fc801fa7940 1 -- start start 2024-05-08T21:23:11.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc801fa7940 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a88890a400 0x55a8889a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc801fa7940 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a888c3a000 0x55a8889a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc801fa7940 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a888c3a400 0x55a8889a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc801fa7940 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a88898e3c0 con 0x55a888c3a400 2024-05-08T21:23:11.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc801fa7940 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a88898e280 con 0x55a888c3a000 2024-05-08T21:23:11.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc801fa7940 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a88898e140 con 0x55a88890a400 2024-05-08T21:23:11.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc7ff754700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a888c3a400 0x55a8889a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc7ff754700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a888c3a400 0x55a8889a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57470/0 (socket says 172.21.15.3:57470) 2024-05-08T21:23:11.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc7fff55700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a888c3a000 0x55a8889a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 learned_addr learned my addr 172.21.15.3:0/371058130 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc800756700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a88890a400 0x55a8889a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a88890a400 msgr2=0x55a8889a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc7ff754700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a88890a400 0x55a8889a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a888c3a000 msgr2=0x55a8889a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc7ff754700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a888c3a000 0x55a8889a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a88892e1a0 con 0x55a888c3a400 2024-05-08T21:23:11.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc7fff55700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a888c3a000 0x55a8889a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:11.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc7ff754700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a888c3a400 0x55a8889a0680 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x55a8896754a0 tx=0x55a8896a3a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc7f9f49700 1 -- 172.21.15.3:0/371058130 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55a88893af20 con 0x55a888c3a400 2024-05-08T21:23:11.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc7f9f49700 1 -- 172.21.15.3:0/371058130 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a889782c00 con 0x55a888c3a400 2024-05-08T21:23:11.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a88892e000 con 0x55a888c3a400 2024-05-08T21:23:11.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.382+0000 7fc7f9f49700 1 -- 172.21.15.3:0/371058130 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55a88893b080 con 0x55a888c3a400 2024-05-08T21:23:11.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.383+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a88892e9c0 con 0x55a888c3a400 2024-05-08T21:23:11.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.383+0000 7fc801fa7940 4 datascan.init: resolving metadata pool 6 2024-05-08T21:23:11.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.384+0000 7fc7f9f49700 1 -- 172.21.15.3:0/371058130 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x55a88896c840 con 0x55a888c3a400 2024-05-08T21:23:11.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.384+0000 7fc7f9f49700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55a888c40c00 0x55a8889a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.384+0000 7fc7f9f49700 1 -- 172.21.15.3:0/371058130 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x55a888948a80 con 0x55a888c3a400 2024-05-08T21:23:11.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.384+0000 7fc801fa7940 4 datascan.init: found metadata pool 'recovery_fs_metadata' 2024-05-08T21:23:11.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.384+0000 7fc801fa7940 4 datascan.main: resolving metadata pool 6 2024-05-08T21:23:11.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.384+0000 7fc801fa7940 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x55a888c3a800 0x55a8889a0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.384+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:1 6.0 6:00000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3ac00 con 0x55a888c3a800 2024-05-08T21:23:11.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.384+0000 7fc800756700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55a888c40c00 0x55a8889a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.385+0000 7fc7fff55700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x55a888c3a800 0x55a8889a0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.385+0000 7fc800756700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55a888c40c00 0x55a8889a1700 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x55a8897a41b0 tx=0x55a8896da300 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.385+0000 7fc7fff55700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x55a888c3a800 0x55a8889a0100 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.385+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.0 v2:172.21.15.3:6824/47120 1 ==== osd_op_reply(1 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c3a800 2024-05-08T21:23:11.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.385+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:2 6.10 6:08000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3ac00 con 0x55a888c3a800 2024-05-08T21:23:11.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.386+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.0 v2:172.21.15.3:6824/47120 2 ==== osd_op_reply(2 [pgnls start_epoch 42 out=81b] v42'2 uv2 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c3a800 2024-05-08T21:23:11.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.386+0000 7fc7fff55700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55a888c3ac00 0x55a888d40000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.386+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:3 6.8 6:10000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c41000 con 0x55a888c3ac00 2024-05-08T21:23:11.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.386+0000 7fc800756700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55a888c3ac00 0x55a888d40000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.386+0000 7fc800756700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55a888c3ac00 0x55a888d40000 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.387+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.7 v2:172.21.15.186:6809/47341 1 ==== osd_op_reply(3 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c3ac00 2024-05-08T21:23:11.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.387+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:4 6.18 6:18000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c41000 con 0x55a888c3a800 2024-05-08T21:23:11.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.387+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.0 v2:172.21.15.3:6824/47120 3 ==== osd_op_reply(4 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c3a800 2024-05-08T21:23:11.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.387+0000 7fc7fff55700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x55a888c41000 0x55a888d40580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.387+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:5 6.4 6:20000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c41400 con 0x55a888c41000 2024-05-08T21:23:11.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.387+0000 7fc7fff55700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x55a888c41000 0x55a888d40580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.387+0000 7fc7fff55700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x55a888c41000 0x55a888d40580 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.388+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.1 v2:172.21.15.3:6816/47119 1 ==== osd_op_reply(5 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c41000 2024-05-08T21:23:11.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.388+0000 7fc7fff55700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55a888c41400 0x55a888d40b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.388+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:6 6.14 6:28000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c41800 con 0x55a888c41400 2024-05-08T21:23:11.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.388+0000 7fc7ff754700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55a888c41400 0x55a888d40b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.388+0000 7fc7ff754700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55a888c41400 0x55a888d40b00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.389+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 <== osd.2 v2:172.21.15.3:6808/47117 1 ==== osd_op_reply(6 [pgnls start_epoch 42 out=77b] v42'1 uv1 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x55a888c2b680 con 0x55a888c41400 2024-05-08T21:23:11.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.389+0000 7fc7ff754700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55a888c41800 0x55a888d41080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.389+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:7 6.c 6:30000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c40400 con 0x55a888c41800 2024-05-08T21:23:11.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.389+0000 7fc800756700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55a888c41800 0x55a888d41080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.389+0000 7fc800756700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55a888c41800 0x55a888d41080 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.389+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 1 ==== osd_op_reply(7 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.389+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:8 6.1c 6:38000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c40400 con 0x55a888c3ac00 2024-05-08T21:23:11.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.390+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.7 v2:172.21.15.186:6809/47341 2 ==== osd_op_reply(8 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c3ac00 2024-05-08T21:23:11.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.390+0000 7fc800756700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x55a888c40400 0x55a888d41600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.390+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:9 6.2 6:40000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56000 con 0x55a888c40400 2024-05-08T21:23:11.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.390+0000 7fc7fff55700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x55a888c40400 0x55a888d41600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.390+0000 7fc7fff55700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x55a888c40400 0x55a888d41600 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.391+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.4 v2:172.21.15.186:6810/47344 1 ==== osd_op_reply(9 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c40400 2024-05-08T21:23:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.391+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:10 6.12 6:48000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56000 con 0x55a888c3ac00 2024-05-08T21:23:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.391+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.7 v2:172.21.15.186:6809/47341 3 ==== osd_op_reply(10 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c3ac00 2024-05-08T21:23:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.391+0000 7fc800756700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55a888d56000 0x55a888d41b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.391+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:11 6.a 6:50000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888d56000 2024-05-08T21:23:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.391+0000 7fc7ff754700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55a888d56000 0x55a888d41b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.392+0000 7fc7ff754700 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55a888d56000 0x55a888d41b80 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.392+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 <== osd.5 v2:172.21.15.186:6800/47340 1 ==== osd_op_reply(11 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b680 con 0x55a888d56000 2024-05-08T21:23:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.392+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:12 6.1a 6:58000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c40400 2024-05-08T21:23:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.392+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.4 v2:172.21.15.186:6810/47344 2 ==== osd_op_reply(12 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c40400 2024-05-08T21:23:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.392+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:13 6.6 6:60000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c41800 2024-05-08T21:23:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.392+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 2 ==== osd_op_reply(13 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.392+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:14 6.16 6:68000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c3a800 2024-05-08T21:23:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.393+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.0 v2:172.21.15.3:6824/47120 4 ==== osd_op_reply(14 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c3a800 2024-05-08T21:23:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.393+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:15 6.e 6:70000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c40400 2024-05-08T21:23:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.393+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.4 v2:172.21.15.186:6810/47344 3 ==== osd_op_reply(15 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c40400 2024-05-08T21:23:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.393+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:16 6.1e 6:78000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c40400 2024-05-08T21:23:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.393+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.4 v2:172.21.15.186:6810/47344 4 ==== osd_op_reply(16 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c40400 2024-05-08T21:23:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.393+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:17 6.1 6:80000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c41000 2024-05-08T21:23:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.394+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.1 v2:172.21.15.3:6816/47119 2 ==== osd_op_reply(17 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c41000 2024-05-08T21:23:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.394+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:18 6.11 6:88000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c41800 2024-05-08T21:23:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.394+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 3 ==== osd_op_reply(18 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.394+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:19 6.9 6:90000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c3a800 2024-05-08T21:23:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.394+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.0 v2:172.21.15.3:6824/47120 5 ==== osd_op_reply(19 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c3a800 2024-05-08T21:23:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.394+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:20 6.19 6:98000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888d56000 2024-05-08T21:23:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.395+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 <== osd.5 v2:172.21.15.186:6800/47340 2 ==== osd_op_reply(20 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b680 con 0x55a888d56000 2024-05-08T21:23:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.395+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:21 6.5 6:a0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c3ac00 2024-05-08T21:23:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.395+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.7 v2:172.21.15.186:6809/47341 4 ==== osd_op_reply(21 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c3ac00 2024-05-08T21:23:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.395+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:22 6.15 6:a8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c3ac00 2024-05-08T21:23:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.395+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.7 v2:172.21.15.186:6809/47341 5 ==== osd_op_reply(22 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c3ac00 2024-05-08T21:23:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.395+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:23 6.d 6:b0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888d56000 2024-05-08T21:23:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.395+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 <== osd.5 v2:172.21.15.186:6800/47340 3 ==== osd_op_reply(23 [pgnls start_epoch 42 out=74b] v42'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x55a888c2b680 con 0x55a888d56000 2024-05-08T21:23:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.395+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:24 6.1d 6:b8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c41000 2024-05-08T21:23:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.396+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.1 v2:172.21.15.3:6816/47119 3 ==== osd_op_reply(24 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c41000 2024-05-08T21:23:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.396+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:25 6.3 6:c0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c3ac00 2024-05-08T21:23:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.396+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.7 v2:172.21.15.186:6809/47341 6 ==== osd_op_reply(25 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c3ac00 2024-05-08T21:23:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.396+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:26 6.13 6:c8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c41800 2024-05-08T21:23:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.397+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 4 ==== osd_op_reply(26 [pgnls start_epoch 42 out=98b] v42'2 uv2 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.397+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:27 6.b 6:d0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c41800 2024-05-08T21:23:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.397+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 5 ==== osd_op_reply(27 [pgnls start_epoch 42 out=76b] v42'1 uv1 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.397+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:28 6.1b 6:d8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c41800 2024-05-08T21:23:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.397+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 6 ==== osd_op_reply(28 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.397+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:29 6.7 6:e0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888d56000 2024-05-08T21:23:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.397+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 <== osd.5 v2:172.21.15.186:6800/47340 4 ==== osd_op_reply(29 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b680 con 0x55a888d56000 2024-05-08T21:23:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.398+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:30 6.17 6:e8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c40400 2024-05-08T21:23:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.398+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.4 v2:172.21.15.186:6810/47344 5 ==== osd_op_reply(30 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c40400 2024-05-08T21:23:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.398+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:31 6.f 6:f0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c41400 2024-05-08T21:23:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.398+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 <== osd.2 v2:172.21.15.3:6808/47117 2 ==== osd_op_reply(31 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b680 con 0x55a888c41400 2024-05-08T21:23:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.398+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:32 6.1f 6:f8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c41800 2024-05-08T21:23:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 7 ==== osd_op_reply(32 [pgnls start_epoch 42 out=101b] v42'3 uv3 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 10 datascan.scan_links: step 1: handling object 10000000000.00000000 2024-05-08T21:23:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:33 6.10 6:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3b000 con 0x55a888c3a800 2024-05-08T21:23:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.0 v2:172.21.15.3:6824/47120 6 ==== osd_op_reply(33 10000000000.00000000 [omap-get-vals out=509b] v0'0 uv2 ondisk = 0) v8 ==== 164+0+509 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c3a800 2024-05-08T21:23:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 20 datascan.scan_links: injected ino detected 2024-05-08T21:23:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 10 datascan.scan_links: step 1: handling object 1.00000000.inode 2024-05-08T21:23:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-05-08T21:23:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 10 datascan.scan_links: step 1: handling object mds_snaptable 2024-05-08T21:23:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-05-08T21:23:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 10 datascan.scan_links: step 1: handling object mds0_inotable 2024-05-08T21:23:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-05-08T21:23:11.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 10 datascan.scan_links: step 1: handling object 100.00000000 2024-05-08T21:23:11.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:34 6.13 6:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3b400 con 0x55a888c41800 2024-05-08T21:23:11.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 8 ==== osd_op_reply(34 100.00000000 [omap-get-vals out=5b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 10 datascan.scan_links: step 1: handling object mds0_sessionmap 2024-05-08T21:23:11.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-05-08T21:23:11.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 10 datascan.scan_links: step 1: handling object 100.00000000.inode 2024-05-08T21:23:11.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-05-08T21:23:11.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 10 datascan.scan_links: step 1: handling object 1.00000000 2024-05-08T21:23:11.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.399+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:35 6.1f 6:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3b800 con 0x55a888c41800 2024-05-08T21:23:11.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.400+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 9 ==== osd_op_reply(35 1.00000000 [omap-get-vals out=508b] v0'0 uv3 ondisk = 0) v8 ==== 154+0+508 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.400+0000 7fc801fa7940 20 datascan.scan_links: injected ino detected 2024-05-08T21:23:11.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.400+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:36 6.0 6:00000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c3a800 2024-05-08T21:23:11.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.400+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.0 v2:172.21.15.3:6824/47120 7 ==== osd_op_reply(36 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c3a800 2024-05-08T21:23:11.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.400+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:37 6.10 6:08000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c3a800 2024-05-08T21:23:11.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.400+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.0 v2:172.21.15.3:6824/47120 8 ==== osd_op_reply(37 [pgnls start_epoch 42 out=81b] v42'2 uv2 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c3a800 2024-05-08T21:23:11.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.400+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:38 6.8 6:10000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c3ac00 2024-05-08T21:23:11.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.401+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.7 v2:172.21.15.186:6809/47341 7 ==== osd_op_reply(38 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c3ac00 2024-05-08T21:23:11.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.401+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:39 6.18 6:18000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c3a800 2024-05-08T21:23:11.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.401+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.0 v2:172.21.15.3:6824/47120 9 ==== osd_op_reply(39 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c3a800 2024-05-08T21:23:11.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.401+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:40 6.4 6:20000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c41000 2024-05-08T21:23:11.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.401+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.1 v2:172.21.15.3:6816/47119 4 ==== osd_op_reply(40 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c41000 2024-05-08T21:23:11.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.401+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:41 6.14 6:28000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c41400 2024-05-08T21:23:11.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.401+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 <== osd.2 v2:172.21.15.3:6808/47117 3 ==== osd_op_reply(41 [pgnls start_epoch 42 out=77b] v42'1 uv1 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x55a888c2b680 con 0x55a888c41400 2024-05-08T21:23:11.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.401+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:42 6.c 6:30000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c41800 2024-05-08T21:23:11.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.402+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 10 ==== osd_op_reply(42 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.402+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:43 6.1c 6:38000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c3ac00 2024-05-08T21:23:11.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.402+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.7 v2:172.21.15.186:6809/47341 8 ==== osd_op_reply(43 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c3ac00 2024-05-08T21:23:11.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.402+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:44 6.2 6:40000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c40400 2024-05-08T21:23:11.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.403+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.4 v2:172.21.15.186:6810/47344 6 ==== osd_op_reply(44 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c40400 2024-05-08T21:23:11.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.403+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:45 6.12 6:48000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c3ac00 2024-05-08T21:23:11.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.403+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.7 v2:172.21.15.186:6809/47341 9 ==== osd_op_reply(45 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c3ac00 2024-05-08T21:23:11.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.403+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:46 6.a 6:50000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888d56000 2024-05-08T21:23:11.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.403+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 <== osd.5 v2:172.21.15.186:6800/47340 5 ==== osd_op_reply(46 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b680 con 0x55a888d56000 2024-05-08T21:23:11.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.403+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:47 6.1a 6:58000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c40400 2024-05-08T21:23:11.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.403+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.4 v2:172.21.15.186:6810/47344 7 ==== osd_op_reply(47 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c40400 2024-05-08T21:23:11.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.403+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:48 6.6 6:60000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c41800 2024-05-08T21:23:11.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.404+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 11 ==== osd_op_reply(48 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.404+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:49 6.16 6:68000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c3a800 2024-05-08T21:23:11.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.404+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.0 v2:172.21.15.3:6824/47120 10 ==== osd_op_reply(49 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c3a800 2024-05-08T21:23:11.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.404+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:50 6.e 6:70000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c40400 2024-05-08T21:23:11.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.404+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.4 v2:172.21.15.186:6810/47344 8 ==== osd_op_reply(50 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c40400 2024-05-08T21:23:11.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.404+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:51 6.1e 6:78000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c40400 2024-05-08T21:23:11.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.405+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.4 v2:172.21.15.186:6810/47344 9 ==== osd_op_reply(51 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c40400 2024-05-08T21:23:11.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.405+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:52 6.1 6:80000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c41000 2024-05-08T21:23:11.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.405+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.1 v2:172.21.15.3:6816/47119 5 ==== osd_op_reply(52 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c41000 2024-05-08T21:23:11.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.405+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:53 6.11 6:88000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c41800 2024-05-08T21:23:11.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.405+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 12 ==== osd_op_reply(53 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.405+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:54 6.9 6:90000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c3a800 2024-05-08T21:23:11.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.405+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.0 v2:172.21.15.3:6824/47120 11 ==== osd_op_reply(54 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c3a800 2024-05-08T21:23:11.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.405+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:55 6.19 6:98000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888d56000 2024-05-08T21:23:11.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.406+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 <== osd.5 v2:172.21.15.186:6800/47340 6 ==== osd_op_reply(55 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b680 con 0x55a888d56000 2024-05-08T21:23:11.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.406+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:56 6.5 6:a0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c3ac00 2024-05-08T21:23:11.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.406+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.7 v2:172.21.15.186:6809/47341 10 ==== osd_op_reply(56 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c3ac00 2024-05-08T21:23:11.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.406+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:57 6.15 6:a8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c3ac00 2024-05-08T21:23:11.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.406+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.7 v2:172.21.15.186:6809/47341 11 ==== osd_op_reply(57 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c3ac00 2024-05-08T21:23:11.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.406+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:58 6.d 6:b0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888d56000 2024-05-08T21:23:11.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.407+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 <== osd.5 v2:172.21.15.186:6800/47340 7 ==== osd_op_reply(58 [pgnls start_epoch 42 out=74b] v42'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x55a888c2b680 con 0x55a888d56000 2024-05-08T21:23:11.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.407+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- osd_op(unknown.0.0:59 6.1d 6:b8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c41000 2024-05-08T21:23:11.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.407+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.1 v2:172.21.15.3:6816/47119 6 ==== osd_op_reply(59 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c41000 2024-05-08T21:23:11.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.407+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:60 6.3 6:c0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c3ac00 2024-05-08T21:23:11.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.407+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.7 v2:172.21.15.186:6809/47341 12 ==== osd_op_reply(60 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c3ac00 2024-05-08T21:23:11.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.407+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:61 6.13 6:c8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c41800 2024-05-08T21:23:11.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.408+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 13 ==== osd_op_reply(61 [pgnls start_epoch 42 out=98b] v42'2 uv2 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.408+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:62 6.b 6:d0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c41800 2024-05-08T21:23:11.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.408+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 14 ==== osd_op_reply(62 [pgnls start_epoch 42 out=76b] v42'1 uv1 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.408+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:63 6.1b 6:d8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c41800 2024-05-08T21:23:11.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.408+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 15 ==== osd_op_reply(63 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.408+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:64 6.7 6:e0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888d56000 2024-05-08T21:23:11.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.409+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 <== osd.5 v2:172.21.15.186:6800/47340 8 ==== osd_op_reply(64 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b680 con 0x55a888d56000 2024-05-08T21:23:11.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.409+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:65 6.17 6:e8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c40400 2024-05-08T21:23:11.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.409+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.4 v2:172.21.15.186:6810/47344 10 ==== osd_op_reply(65 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c40400 2024-05-08T21:23:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.409+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:66 6.f 6:f0000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c41400 2024-05-08T21:23:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.409+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 <== osd.2 v2:172.21.15.3:6808/47117 4 ==== osd_op_reply(66 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a888c2b680 con 0x55a888c41400 2024-05-08T21:23:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.409+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:67 6.1f 6:f8000000::::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3bc00 con 0x55a888c41800 2024-05-08T21:23:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.409+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 16 ==== osd_op_reply(67 [pgnls start_epoch 42 out=101b] v42'3 uv3 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.409+0000 7fc801fa7940 10 datascan.scan_links: step 2: handling object 10000000000.00000000 2024-05-08T21:23:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.409+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:68 6.10 6:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56800 con 0x55a888c3a800 2024-05-08T21:23:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.0 v2:172.21.15.3:6824/47120 12 ==== osd_op_reply(68 10000000000.00000000 [omap-get-vals out=509b] v0'0 uv2 ondisk = 0) v8 ==== 164+0+509 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c3a800 2024-05-08T21:23:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc801fa7940 20 datascan.scan_links: injected ino detected 2024-05-08T21:23:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc801fa7940 20 datascan.scan_links: adding 0x10000000001 for future processing to fix dnfirst 2024-05-08T21:23:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc801fa7940 10 datascan.scan_links: step 2: handling object 1.00000000.inode 2024-05-08T21:23:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc801fa7940 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-05-08T21:23:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc801fa7940 10 datascan.scan_links: step 2: handling object mds_snaptable 2024-05-08T21:23:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc801fa7940 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-05-08T21:23:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc801fa7940 10 datascan.scan_links: step 2: handling object mds0_inotable 2024-05-08T21:23:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc801fa7940 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-05-08T21:23:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc801fa7940 10 datascan.scan_links: step 2: handling object 100.00000000 2024-05-08T21:23:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:69 6.13 6:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56400 con 0x55a888c41800 2024-05-08T21:23:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 17 ==== osd_op_reply(69 100.00000000 [omap-get-vals out=5b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc801fa7940 10 datascan.scan_links: step 2: handling object mds0_sessionmap 2024-05-08T21:23:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc801fa7940 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-05-08T21:23:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc801fa7940 10 datascan.scan_links: step 2: handling object 100.00000000.inode 2024-05-08T21:23:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc801fa7940 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-05-08T21:23:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc801fa7940 10 datascan.scan_links: step 2: handling object 1.00000000 2024-05-08T21:23:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.410+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:70 6.1f 6:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56c00 con 0x55a888c41800 2024-05-08T21:23:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.411+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 18 ==== osd_op_reply(70 1.00000000 [omap-get-vals out=508b] v0'0 uv3 ondisk = 0) v8 ==== 154+0+508 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.411+0000 7fc801fa7940 20 datascan.scan_links: injected ino detected 2024-05-08T21:23:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.411+0000 7fc801fa7940 20 datascan.scan_links: adding 0x10000000000 for future processing to fix dnfirst 2024-05-08T21:23:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.411+0000 7fc801fa7940 10 datascan.scan_links: processing 0 dup_primaries, 0 remote_links 2024-05-08T21:23:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.411+0000 7fc801fa7940 10 datascan.scan_links: removing dup dentries from 0 objects 2024-05-08T21:23:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.411+0000 7fc801fa7940 10 datascan.scan_links: processing 0 bad_nlink_inos 2024-05-08T21:23:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.411+0000 7fc801fa7940 10 datascan.scan_links: processing 2 injected_inos 2024-05-08T21:23:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.411+0000 7fc801fa7940 10 datascan.scan_links: handling injected_ino 0x10000000000 2024-05-08T21:23:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.411+0000 7fc801fa7940 20 datascan.scan_links: fixing linkage (dnfirst) of 0x1:subdir 2024-05-08T21:23:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.411+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:71 6.1f 6:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d57000 con 0x55a888c41800 2024-05-08T21:23:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.411+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 19 ==== osd_op_reply(71 1.00000000 [omap-get-vals-by-keys out=507b] v0'0 uv3 ondisk = 0) v8 ==== 154+0+507 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.411+0000 7fc801fa7940 20 datascan.read_dentry: oid=1.00000000 dname=subdir frag=*, r=0 2024-05-08T21:23:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.411+0000 7fc801fa7940 20 datascan.scan_links: first is now 2 2024-05-08T21:23:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.411+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:72 6.1f 6:ff5b34d6:::1.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d57400 con 0x55a888c41800 2024-05-08T21:23:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.412+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 20 ==== osd_op_reply(72 1.00000000 [omap-set-vals] v42'4 uv4 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.412+0000 7fc801fa7940 20 datascan.inject_linkage: Injected dentry 0x0x1/subdir pointing to 0x0x10000000000 2024-05-08T21:23:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.412+0000 7fc801fa7940 10 datascan.scan_links: handling injected_ino 0x10000000001 2024-05-08T21:23:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.412+0000 7fc801fa7940 20 datascan.scan_links: fixing linkage (dnfirst) of 0x10000000000:sixmegs 2024-05-08T21:23:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.412+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:73 6.10 6:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d57800 con 0x55a888c3a800 2024-05-08T21:23:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.413+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.0 v2:172.21.15.3:6824/47120 13 ==== osd_op_reply(73 10000000000.00000000 [omap-get-vals-by-keys out=508b] v0'0 uv2 ondisk = 0) v8 ==== 164+0+508 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c3a800 2024-05-08T21:23:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.413+0000 7fc801fa7940 20 datascan.read_dentry: oid=10000000000.00000000 dname=sixmegs frag=*, r=0 2024-05-08T21:23:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.413+0000 7fc801fa7940 20 datascan.scan_links: first is now 2 2024-05-08T21:23:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.413+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- osd_op(unknown.0.0:74 6.10 6:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=508b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d57c00 con 0x55a888c3a800 2024-05-08T21:23:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.414+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 <== osd.0 v2:172.21.15.3:6824/47120 14 ==== osd_op_reply(74 10000000000.00000000 [omap-set-vals] v42'3 uv3 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55a888c2b200 con 0x55a888c3a800 2024-05-08T21:23:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.414+0000 7fc801fa7940 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/sixmegs pointing to 0x0x10000000001 2024-05-08T21:23:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.414+0000 7fc801fa7940 10 datascan.scan_links: updating inotable 2024-05-08T21:23:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.414+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:75 6.13 6:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3b400 con 0x55a888c41800 2024-05-08T21:23:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.414+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 21 ==== osd_op_reply(75 mds0_inotable [read 0~34 out=34b] v0'0 uv2 ondisk = 0) v8 ==== 157+0+34 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.414+0000 7fc801fa7940 10 mds.0.inotable: skip_inos was [0x10000000000~0x10000000000] 2024-05-08T21:23:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.414+0000 7fc801fa7940 10 mds.0.inotable: skip_inos now [0x10000000002~0xfffffffffe] 2024-05-08T21:23:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.414+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:76 6.13 6:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888c3b800 con 0x55a888c41800 2024-05-08T21:23:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.416+0000 7fc800756700 1 -- 172.21.15.3:0/371058130 <== osd.3 v2:172.21.15.3:6800/47118 22 ==== osd_op_reply(76 mds0_inotable [writefull 0~34] v42'3 uv3 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x55a888c2b440 con 0x55a888c41800 2024-05-08T21:23:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.416+0000 7fc801fa7940 10 datascan.scan_links: updating snaptable 2024-05-08T21:23:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.416+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- osd_op(unknown.0.0:77 6.d 6:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55a888d56800 con 0x55a888d56000 2024-05-08T21:23:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.416+0000 7fc7ff754700 1 -- 172.21.15.3:0/371058130 <== osd.5 v2:172.21.15.186:6800/47340 9 ==== osd_op_reply(77 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) v8 ==== 157+0+70 (crc 0 0 0) 0x55a888c2b680 con 0x55a888d56000 2024-05-08T21:23:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.416+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x55a888c3a800 msgr2=0x55a8889a0100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.416+0000 7fc801fa7940 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x55a888c3a800 0x55a8889a0100 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x55a888c41000 msgr2=0x55a888d40580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x55a888c41000 0x55a888d40580 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55a888c41400 msgr2=0x55a888d40b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55a888c41400 0x55a888d40b00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55a888c41800 msgr2=0x55a888d41080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x55a888c41800 0x55a888d41080 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x55a888c40400 msgr2=0x55a888d41600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x55a888c40400 0x55a888d41600 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55a888d56000 msgr2=0x55a888d41b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x55a888d56000 0x55a888d41b80 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55a888c3ac00 msgr2=0x55a888d40000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55a888c3ac00 0x55a888d40000 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 reap_dead start 2024-05-08T21:23:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 reap_dead start 2024-05-08T21:23:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc7fff55700 1 -- 172.21.15.3:0/371058130 reap_dead start 2024-05-08T21:23:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55a888c40c00 msgr2=0x55a8889a1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55a888c40c00 0x55a8889a1700 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x55a8897a41b0 tx=0x55a8896da300 comp rx=0 tx=0).stop 2024-05-08T21:23:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a888c3a400 msgr2=0x55a8889a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a888c3a400 0x55a8889a0680 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x55a8896754a0 tx=0x55a8896a3a40 comp rx=0 tx=0).stop 2024-05-08T21:23:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 shutdown_connections 2024-05-08T21:23:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55a888c40c00 0x55a8889a1700 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 --2- 172.21.15.3:0/371058130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a888c3a400 0x55a8889a0680 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 >> 172.21.15.3:0/371058130 conn(0x55a88890bc00 msgr2=0x55a88968b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 shutdown_connections 2024-05-08T21:23:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 -- 172.21.15.3:0/371058130 wait complete. 2024-05-08T21:23:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 -- 172.21.15.3:0/3072582739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a88890ac00 msgr2=0x55a88899eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 --2- 172.21.15.3:0/3072582739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a88890ac00 0x55a88899eb00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55a889675350 tx=0x55a8896da360 comp rx=0 tx=0).stop 2024-05-08T21:23:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 -- 172.21.15.3:0/3072582739 shutdown_connections 2024-05-08T21:23:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 --2- 172.21.15.3:0/3072582739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a88890a800 0x55a88899f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 --2- 172.21.15.3:0/3072582739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a88890b000 0x55a88899e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 --2- 172.21.15.3:0/3072582739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a88890ac00 0x55a88899eb00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 -- 172.21.15.3:0/3072582739 >> 172.21.15.3:0/3072582739 conn(0x55a88890b400 msgr2=0x55a88968b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.417+0000 7fc801fa7940 1 -- 172.21.15.3:0/3072582739 shutdown_connections 2024-05-08T21:23:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.418+0000 7fc801fa7940 1 -- 172.21.15.3:0/3072582739 wait complete. 2024-05-08T21:23:11.430 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_links', '--filesystem', 'recovery_fs'] in time 0:00:00.102937, result: 2024-05-08T21:23:11.430 DEBUG:teuthology.orchestra.run.smithi003:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank recovery_fs:0 event recover_dentries list --alternate-pool recovery_fs_metadata 2024-05-08T21:23:11.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.481+0000 7f7567ce4440 1 -- start start 2024-05-08T21:23:11.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.481+0000 7f7567ce4440 1 Processor -- start 2024-05-08T21:23:11.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f7567ce4440 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d6b37000 0x55b6d6bca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f7567ce4440 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6b36c00 0x55b6d6bcab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f7567ce4440 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b6d6b36800 0x55b6d6bcb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f7567ce4440 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6d6b73e00 con 0x55b6d6b36c00 2024-05-08T21:23:11.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f7567ce4440 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6d6bbaa00 con 0x55b6d6b37000 2024-05-08T21:23:11.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f7567ce4440 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6d6bba640 con 0x55b6d6b36800 2024-05-08T21:23:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f756548f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6b36c00 0x55b6d6bcab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f756548f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6b36c00 0x55b6d6bcab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57484/0 (socket says 172.21.15.3:57484) 2024-05-08T21:23:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f756548f700 1 -- 172.21.15.3:0/3230040451 learned_addr learned my addr 172.21.15.3:0/3230040451 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f7566491700 1 --2- 172.21.15.3:0/3230040451 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b6d6b36800 0x55b6d6bcb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f7565c90700 1 --2- 172.21.15.3:0/3230040451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d6b37000 0x55b6d6bca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f756548f700 1 -- 172.21.15.3:0/3230040451 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b6d6b36800 msgr2=0x55b6d6bcb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f756548f700 1 --2- 172.21.15.3:0/3230040451 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b6d6b36800 0x55b6d6bcb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f756548f700 1 -- 172.21.15.3:0/3230040451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d6b37000 msgr2=0x55b6d6bca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f756548f700 1 --2- 172.21.15.3:0/3230040451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d6b37000 0x55b6d6bca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f756548f700 1 -- 172.21.15.3:0/3230040451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6d6b5a9c0 con 0x55b6d6b36c00 2024-05-08T21:23:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f7565c90700 1 --2- 172.21.15.3:0/3230040451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d6b37000 0x55b6d6bca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f756548f700 1 --2- 172.21.15.3:0/3230040451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6b36c00 0x55b6d6bcab00 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x55b6d6c96d20 tx=0x55b6d79b2240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f1a1087b3ff7567 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.482+0000 7f756348b700 1 -- 172.21.15.3:0/3230040451 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b6d6b67600 con 0x55b6d6b36c00 2024-05-08T21:23:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.483+0000 7f7567ce4440 1 -- 172.21.15.3:0/3230040451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b6d6b5a1a0 con 0x55b6d6b36c00 2024-05-08T21:23:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.483+0000 7f756348b700 1 -- 172.21.15.3:0/3230040451 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b6d6b89080 con 0x55b6d6b36c00 2024-05-08T21:23:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.483+0000 7f756348b700 1 -- 172.21.15.3:0/3230040451 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b6d6b671e0 con 0x55b6d6b36c00 2024-05-08T21:23:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.483+0000 7f7567ce4440 1 -- 172.21.15.3:0/3230040451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55b6d6b5ab60 con 0x55b6d6b36c00 2024-05-08T21:23:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.483+0000 7f7567ce4440 4 waiting for MDS map... 2024-05-08T21:23:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.483+0000 7f756348b700 1 -- 172.21.15.3:0/3230040451 <== mon.0 v2:172.21.15.3:3300/0 4 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x55b6d6b74c40 con 0x55b6d6b36c00 2024-05-08T21:23:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.483+0000 7f756348b700 1 -- 172.21.15.3:0/3230040451 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 21) v1 ==== 3691+0+0 (secure 0 0 0) 0x55b6d79b5200 con 0x55b6d6b36c00 2024-05-08T21:23:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.483+0000 7f7567ce4440 4 Got MDS map 21 2024-05-08T21:23:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.483+0000 7f7567ce4440 10 main: JournalTool::main 2024-05-08T21:23:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.483+0000 7f7567ce4440 4 main: JournalTool: connecting to RADOS... 2024-05-08T21:23:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.484+0000 7f7567ce4440 1 Processor -- start 2024-05-08T21:23:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.484+0000 7f7567ce4440 1 -- start start 2024-05-08T21:23:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.484+0000 7f7567ce4440 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6b37c00 0x55b6d6bcb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.484+0000 7f7567ce4440 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d6e4a000 0x55b6d6bcbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.484+0000 7f7567ce4440 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b6d6e4a400 0x55b6d6bcc100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.484+0000 7f7567ce4440 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6d6bba500 con 0x55b6d6b37c00 2024-05-08T21:23:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.484+0000 7f7567ce4440 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6d6bba3c0 con 0x55b6d6e4a000 2024-05-08T21:23:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.484+0000 7f7567ce4440 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6d6bbaa00 con 0x55b6d6e4a400 2024-05-08T21:23:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.484+0000 7f7566491700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6b37c00 0x55b6d6bcb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.484+0000 7f7566491700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6b37c00 0x55b6d6bcb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57494/0 (socket says 172.21.15.3:57494) 2024-05-08T21:23:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.484+0000 7f7566491700 1 -- 172.21.15.3:0/3465140044 learned_addr learned my addr 172.21.15.3:0/3465140044 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.485+0000 7f756548f700 1 --2- 172.21.15.3:0/3465140044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b6d6e4a400 0x55b6d6bcc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.485+0000 7f7565c90700 1 --2- 172.21.15.3:0/3465140044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d6e4a000 0x55b6d6bcbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.485+0000 7f7566491700 1 -- 172.21.15.3:0/3465140044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b6d6e4a400 msgr2=0x55b6d6bcc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.485+0000 7f7566491700 1 --2- 172.21.15.3:0/3465140044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b6d6e4a400 0x55b6d6bcc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.485+0000 7f7566491700 1 -- 172.21.15.3:0/3465140044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d6e4a000 msgr2=0x55b6d6bcbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.485+0000 7f7566491700 1 --2- 172.21.15.3:0/3465140044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d6e4a000 0x55b6d6bcbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.485+0000 7f7566491700 1 -- 172.21.15.3:0/3465140044 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6d6b5a340 con 0x55b6d6b37c00 2024-05-08T21:23:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.485+0000 7f7566491700 1 --2- 172.21.15.3:0/3465140044 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6b37c00 0x55b6d6bcb600 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x55b6d6c97d10 tx=0x55b6d78cf200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d13b104bb477c65 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.485+0000 7f7561487700 1 -- 172.21.15.3:0/3465140044 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b6d6b67080 con 0x55b6d6b37c00 2024-05-08T21:23:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.485+0000 7f7561487700 1 -- 172.21.15.3:0/3465140044 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b6d6ca0780 con 0x55b6d6b37c00 2024-05-08T21:23:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.485+0000 7f7561487700 1 -- 172.21.15.3:0/3465140044 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b6d6b67600 con 0x55b6d6b37c00 2024-05-08T21:23:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.486+0000 7f7567ce4440 1 -- 172.21.15.3:0/3465140044 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6b37c00 msgr2=0x55b6d6bcb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.486+0000 7f7567ce4440 1 --2- 172.21.15.3:0/3465140044 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6b37c00 0x55b6d6bcb600 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x55b6d6c97d10 tx=0x55b6d78cf200 comp rx=0 tx=0).stop 2024-05-08T21:23:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.486+0000 7f7567ce4440 1 -- 172.21.15.3:0/3465140044 shutdown_connections 2024-05-08T21:23:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.486+0000 7f7567ce4440 1 --2- 172.21.15.3:0/3465140044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b6d6e4a400 0x55b6d6bcc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.486+0000 7f7567ce4440 1 --2- 172.21.15.3:0/3465140044 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6b37c00 0x55b6d6bcb600 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.486+0000 7f7567ce4440 1 --2- 172.21.15.3:0/3465140044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d6e4a000 0x55b6d6bcbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.486+0000 7f7567ce4440 1 -- 172.21.15.3:0/3465140044 >> 172.21.15.3:0/3465140044 conn(0x55b6d6b36400 msgr2=0x55b6d6c25800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.486+0000 7f7567ce4440 1 -- 172.21.15.3:0/3465140044 shutdown_connections 2024-05-08T21:23:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.486+0000 7f7567ce4440 1 -- 172.21.15.3:0/3465140044 wait complete. 2024-05-08T21:23:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.486+0000 7f7567ce4440 1 Processor -- start 2024-05-08T21:23:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.486+0000 7f7567ce4440 1 -- start start 2024-05-08T21:23:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.487+0000 7f7567ce4440 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d6b36400 0x55b6d6bcd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.487+0000 7f7567ce4440 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b6d6e64000 0x55b6d6bccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.487+0000 7f7567ce4440 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6e64400 0x55b6d6bcc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.487+0000 7f7567ce4440 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6d6bba280 con 0x55b6d6e64400 2024-05-08T21:23:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.487+0000 7f7567ce4440 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6d6bba140 con 0x55b6d6b36400 2024-05-08T21:23:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.487+0000 7f7567ce4440 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6d6bba000 con 0x55b6d6e64000 2024-05-08T21:23:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.487+0000 7f756548f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6e64400 0x55b6d6bcc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.487+0000 7f756548f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6e64400 0x55b6d6bcc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57510/0 (socket says 172.21.15.3:57510) 2024-05-08T21:23:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.487+0000 7f756548f700 1 -- 172.21.15.3:0/2351399364 learned_addr learned my addr 172.21.15.3:0/2351399364 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.487+0000 7f7566491700 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d6b36400 0x55b6d6bcd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.487+0000 7f7565c90700 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b6d6e64000 0x55b6d6bccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.487+0000 7f756548f700 1 -- 172.21.15.3:0/2351399364 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b6d6e64000 msgr2=0x55b6d6bccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.487+0000 7f756548f700 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b6d6e64000 0x55b6d6bccc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.487+0000 7f756548f700 1 -- 172.21.15.3:0/2351399364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d6b36400 msgr2=0x55b6d6bcd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.487+0000 7f756548f700 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d6b36400 0x55b6d6bcd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.487+0000 7f756548f700 1 -- 172.21.15.3:0/2351399364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6d6b5a9c0 con 0x55b6d6e64400 2024-05-08T21:23:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.488+0000 7f756548f700 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6e64400 0x55b6d6bcc680 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x55b6d6c97c50 tx=0x55b6d79b2060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.488+0000 7f755fc84700 1 -- 172.21.15.3:0/2351399364 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b6d6b66f20 con 0x55b6d6e64400 2024-05-08T21:23:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.488+0000 7f755fc84700 1 -- 172.21.15.3:0/2351399364 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b6d6ca0f00 con 0x55b6d6e64400 2024-05-08T21:23:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.488+0000 7f7567ce4440 1 -- 172.21.15.3:0/2351399364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b6d6b5a1a0 con 0x55b6d6e64400 2024-05-08T21:23:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.488+0000 7f755fc84700 1 -- 172.21.15.3:0/2351399364 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b6d6b67080 con 0x55b6d6e64400 2024-05-08T21:23:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.488+0000 7f7567ce4440 1 -- 172.21.15.3:0/2351399364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b6d6b5ab60 con 0x55b6d6e64400 2024-05-08T21:23:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.488+0000 7f7567ce4440 4 main: JournalTool: resolving pool 6 2024-05-08T21:23:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.489+0000 7f755fc84700 1 -- 172.21.15.3:0/2351399364 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x55b6d6b98840 con 0x55b6d6e64400 2024-05-08T21:23:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.490+0000 7f755fc84700 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55b6d6e66800 0x55b6d6bcd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.490+0000 7f755fc84700 1 -- 172.21.15.3:0/2351399364 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x55b6d6b74a80 con 0x55b6d6e64400 2024-05-08T21:23:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.490+0000 7f7567ce4440 4 main: JournalTool: creating IoCtx.. 2024-05-08T21:23:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.490+0000 7f7567ce4440 4 main: Executing for rank 0 2024-05-08T21:23:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.490+0000 7f7567ce4440 1 main_event: Using alternate pool recovery_fs_metadata 2024-05-08T21:23:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.490+0000 7f7567ce4440 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55b6d6e64800 0x55b6d6bcc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.490+0000 7f7567ce4440 1 -- 172.21.15.3:0/2351399364 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:1 6.f 6:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55b6d6e64c00 con 0x55b6d6e64800 2024-05-08T21:23:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.490+0000 7f7566491700 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55b6d6e66800 0x55b6d6bcd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.490+0000 7f7565c90700 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55b6d6e64800 0x55b6d6bcc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.490+0000 7f7566491700 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55b6d6e66800 0x55b6d6bcd700 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x55b6d6c97d10 tx=0x55b6d78cf1a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.490+0000 7f7565c90700 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55b6d6e64800 0x55b6d6bcc100 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.491+0000 7f7565c90700 1 -- 172.21.15.3:0/2351399364 <== osd.2 v2:172.21.15.3:6808/47117 1 ==== osd_op_reply(1 400.00000000 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x55b6d799ab40 con 0x55b6d6e64800 2024-05-08T21:23:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.491+0000 7f7567ce4440 -1 Pointer 400.00000000 is absent 2024-05-08T21:23:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.491+0000 7f7567ce4440 4 main_event: consumed 0 inodes 2024-05-08T21:23:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.491+0000 7f7567ce4440 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55b6d6e65000 0x55b6d6bcbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.491+0000 7f7567ce4440 1 -- 172.21.15.3:0/2351399364 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- osd_op(unknown.0.0:2 6.3 6:c0ce80d4:::4.00000000:head [omap-rm-keys in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55b6d6e65400 con 0x55b6d6e65000 2024-05-08T21:23:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.491+0000 7f7566491700 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55b6d6e65000 0x55b6d6bcbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.491+0000 7f7566491700 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55b6d6e65000 0x55b6d6bcbb80 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.492+0000 7f7566491700 1 -- 172.21.15.3:0/2351399364 <== osd.7 v2:172.21.15.186:6809/47341 1 ==== osd_op_reply(2 4.00000000 [omap-rm-keys] v42'1 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x55b6d799b200 con 0x55b6d6e65000 2024-05-08T21:23:11.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.492+0000 7f7567ce4440 4 0 events 2024-05-08T21:23:11.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.492+0000 7f7567ce4440 1 -- 172.21.15.3:0/2351399364 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55b6d6e64800 msgr2=0x55b6d6bcc100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.492+0000 7f7567ce4440 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x55b6d6e64800 0x55b6d6bcc100 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.492+0000 7f7567ce4440 1 -- 172.21.15.3:0/2351399364 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55b6d6e65000 msgr2=0x55b6d6bcbb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.492+0000 7f7567ce4440 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x55b6d6e65000 0x55b6d6bcbb80 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 -- 172.21.15.3:0/2351399364 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55b6d6e66800 msgr2=0x55b6d6bcd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x55b6d6e66800 0x55b6d6bcd700 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x55b6d6c97d10 tx=0x55b6d78cf1a0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 -- 172.21.15.3:0/2351399364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6e64400 msgr2=0x55b6d6bcc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6e64400 0x55b6d6bcc680 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x55b6d6c97c50 tx=0x55b6d79b2060 comp rx=0 tx=0).stop 2024-05-08T21:23:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7565c90700 1 -- 172.21.15.3:0/2351399364 reap_dead start 2024-05-08T21:23:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 -- 172.21.15.3:0/2351399364 shutdown_connections 2024-05-08T21:23:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 --2- 172.21.15.3:0/2351399364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6e64400 0x55b6d6bcc680 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 -- 172.21.15.3:0/2351399364 >> 172.21.15.3:0/2351399364 conn(0x55b6d6b37c00 msgr2=0x55b6d6c25800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 -- 172.21.15.3:0/2351399364 shutdown_connections 2024-05-08T21:23:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 -- 172.21.15.3:0/2351399364 wait complete. 2024-05-08T21:23:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 -- 172.21.15.3:0/3230040451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6b36c00 msgr2=0x55b6d6bcab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 --2- 172.21.15.3:0/3230040451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6b36c00 0x55b6d6bcab00 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x55b6d6c96d20 tx=0x55b6d79b2240 comp rx=0 tx=0).stop 2024-05-08T21:23:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 -- 172.21.15.3:0/3230040451 shutdown_connections 2024-05-08T21:23:11.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 --2- 172.21.15.3:0/3230040451 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b6d6b36800 0x55b6d6bcb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 --2- 172.21.15.3:0/3230040451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d6b37000 0x55b6d6bca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 --2- 172.21.15.3:0/3230040451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6d6b36c00 0x55b6d6bcab00 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 -- 172.21.15.3:0/3230040451 >> 172.21.15.3:0/3230040451 conn(0x55b6d6b37400 msgr2=0x55b6d6c25000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 -- 172.21.15.3:0/3230040451 shutdown_connections 2024-05-08T21:23:11.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.493+0000 7f7567ce4440 1 -- 172.21.15.3:0/3230040451 wait complete. 2024-05-08T21:23:11.502 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'recovery_fs:0', 'event', 'recover_dentries', 'list', '--alternate-pool', 'recovery_fs_metadata'] in time 0:00:00.071845, result: 2024-05-08T21:23:11.502 DEBUG:teuthology.orchestra.run.smithi003:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank recovery_fs:0 journal reset --force 2024-05-08T21:23:11.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.550+0000 7fdca1a91440 1 -- start start 2024-05-08T21:23:11.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.550+0000 7fdca1a91440 1 Processor -- start 2024-05-08T21:23:11.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdca1a91440 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579636c5000 0x557963758580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdca1a91440 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5579636c4c00 0x557963758b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdca1a91440 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5579636c4800 0x557963759080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdca1a91440 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557963701e00 con 0x5579636c4800 2024-05-08T21:23:11.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdca1a91440 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x557963748a00 con 0x5579636c5000 2024-05-08T21:23:11.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdca1a91440 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x557963748640 con 0x5579636c4c00 2024-05-08T21:23:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdca023e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5579636c4800 0x557963759080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdca023e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5579636c4800 0x557963759080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57516/0 (socket says 172.21.15.3:57516) 2024-05-08T21:23:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdca023e700 1 -- 172.21.15.3:0/1088322775 learned_addr learned my addr 172.21.15.3:0/1088322775 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdc9fa3d700 1 --2- 172.21.15.3:0/1088322775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579636c5000 0x557963758580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdc9f23c700 1 --2- 172.21.15.3:0/1088322775 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5579636c4c00 0x557963758b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdca023e700 1 -- 172.21.15.3:0/1088322775 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5579636c4c00 msgr2=0x557963758b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdca023e700 1 --2- 172.21.15.3:0/1088322775 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5579636c4c00 0x557963758b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdca023e700 1 -- 172.21.15.3:0/1088322775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579636c5000 msgr2=0x557963758580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdca023e700 1 --2- 172.21.15.3:0/1088322775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579636c5000 0x557963758580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdca023e700 1 -- 172.21.15.3:0/1088322775 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5579636e89c0 con 0x5579636c4800 2024-05-08T21:23:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdca023e700 1 --2- 172.21.15.3:0/1088322775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5579636c4800 0x557963759080 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x557963824c30 tx=0x55796452e240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9827415d91ce2524 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdc9d238700 1 -- 172.21.15.3:0/1088322775 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5579636f5600 con 0x5579636c4800 2024-05-08T21:23:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdc9d238700 1 -- 172.21.15.3:0/1088322775 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55796382e300 con 0x5579636c4800 2024-05-08T21:23:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.551+0000 7fdc9d238700 1 -- 172.21.15.3:0/1088322775 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5579636f51e0 con 0x5579636c4800 2024-05-08T21:23:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.552+0000 7fdca1a91440 1 -- 172.21.15.3:0/1088322775 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5579636e8b60 con 0x5579636c4800 2024-05-08T21:23:11.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.552+0000 7fdca1a91440 1 -- 172.21.15.3:0/1088322775 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5579636e8340 con 0x5579636c4800 2024-05-08T21:23:11.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.552+0000 7fdca1a91440 4 waiting for MDS map... 2024-05-08T21:23:11.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.552+0000 7fdc9d238700 1 -- 172.21.15.3:0/1088322775 <== mon.0 v2:172.21.15.3:3300/0 4 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x557963702c40 con 0x5579636c4800 2024-05-08T21:23:11.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.552+0000 7fdc9d238700 1 -- 172.21.15.3:0/1088322775 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 21) v1 ==== 3691+0+0 (secure 0 0 0) 0x557964530000 con 0x5579636c4800 2024-05-08T21:23:11.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.552+0000 7fdca1a91440 4 Got MDS map 21 2024-05-08T21:23:11.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.552+0000 7fdca1a91440 10 main: JournalTool::main 2024-05-08T21:23:11.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.552+0000 7fdca1a91440 4 main: JournalTool: connecting to RADOS... 2024-05-08T21:23:11.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.552+0000 7fdca1a91440 1 Processor -- start 2024-05-08T21:23:11.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.553+0000 7fdca1a91440 1 -- start start 2024-05-08T21:23:11.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.553+0000 7fdca1a91440 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5579636c5c00 0x557963759600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.553+0000 7fdca1a91440 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579639d6000 0x557963759b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.553+0000 7fdca1a91440 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5579639d6400 0x55796375a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.553+0000 7fdca1a91440 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557963748500 con 0x5579636c5c00 2024-05-08T21:23:11.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.553+0000 7fdca1a91440 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579637483c0 con 0x5579639d6000 2024-05-08T21:23:11.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.553+0000 7fdca1a91440 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x557963748a00 con 0x5579639d6400 2024-05-08T21:23:11.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.553+0000 7fdc9f23c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5579636c5c00 0x557963759600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.553+0000 7fdc9f23c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5579636c5c00 0x557963759600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57532/0 (socket says 172.21.15.3:57532) 2024-05-08T21:23:11.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.553+0000 7fdc9f23c700 1 -- 172.21.15.3:0/1170071526 learned_addr learned my addr 172.21.15.3:0/1170071526 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.553+0000 7fdc9fa3d700 1 --2- 172.21.15.3:0/1170071526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579639d6000 0x557963759b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.553+0000 7fdc9f23c700 1 --2- 172.21.15.3:0/1170071526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5579639d6400 0x55796375a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.554+0000 7fdc9f23c700 1 -- 172.21.15.3:0/1170071526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5579639d6400 msgr2=0x55796375a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.554+0000 7fdc9f23c700 1 --2- 172.21.15.3:0/1170071526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5579639d6400 0x55796375a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.554+0000 7fdc9f23c700 1 -- 172.21.15.3:0/1170071526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579639d6000 msgr2=0x557963759b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.554+0000 7fdc9f23c700 1 --2- 172.21.15.3:0/1170071526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579639d6000 0x557963759b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.554+0000 7fdc9f23c700 1 -- 172.21.15.3:0/1170071526 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5579636e8000 con 0x5579636c5c00 2024-05-08T21:23:11.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.554+0000 7fdc9fa3d700 1 --2- 172.21.15.3:0/1170071526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579639d6000 0x557963759b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:11.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.554+0000 7fdc9f23c700 1 --2- 172.21.15.3:0/1170071526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5579636c5c00 0x557963759600 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x5579639dc120 tx=0x55796445d2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b97339b791e2d76 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.554+0000 7fdc9b234700 1 -- 172.21.15.3:0/1170071526 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5579636f5080 con 0x5579636c5c00 2024-05-08T21:23:11.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.554+0000 7fdc9b234700 1 -- 172.21.15.3:0/1170071526 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55796382e900 con 0x5579636c5c00 2024-05-08T21:23:11.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.554+0000 7fdc9b234700 1 -- 172.21.15.3:0/1170071526 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5579636f5600 con 0x5579636c5c00 2024-05-08T21:23:11.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.554+0000 7fdca1a91440 1 -- 172.21.15.3:0/1170071526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5579636c5c00 msgr2=0x557963759600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.554+0000 7fdca1a91440 1 --2- 172.21.15.3:0/1170071526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5579636c5c00 0x557963759600 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x5579639dc120 tx=0x55796445d2c0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.555+0000 7fdca1a91440 1 -- 172.21.15.3:0/1170071526 shutdown_connections 2024-05-08T21:23:11.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.555+0000 7fdca1a91440 1 --2- 172.21.15.3:0/1170071526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5579639d6400 0x55796375a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.555+0000 7fdca1a91440 1 --2- 172.21.15.3:0/1170071526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5579636c5c00 0x557963759600 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.555+0000 7fdca1a91440 1 --2- 172.21.15.3:0/1170071526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579639d6000 0x557963759b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.555+0000 7fdca1a91440 1 -- 172.21.15.3:0/1170071526 >> 172.21.15.3:0/1170071526 conn(0x5579636c4400 msgr2=0x5579637b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.555+0000 7fdca1a91440 1 -- 172.21.15.3:0/1170071526 shutdown_connections 2024-05-08T21:23:11.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.555+0000 7fdca1a91440 1 -- 172.21.15.3:0/1170071526 wait complete. 2024-05-08T21:23:11.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.555+0000 7fdca1a91440 1 Processor -- start 2024-05-08T21:23:11.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.555+0000 7fdca1a91440 1 -- start start 2024-05-08T21:23:11.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.556+0000 7fdca1a91440 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5579636c4400 0x55796375b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.556+0000 7fdca1a91440 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557964542000 0x55796375ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.556+0000 7fdca1a91440 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557964542400 0x55796375a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.556+0000 7fdca1a91440 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557963748280 con 0x557964542400 2024-05-08T21:23:11.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.556+0000 7fdca1a91440 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x557963748140 con 0x557964542000 2024-05-08T21:23:11.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.556+0000 7fdca1a91440 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x557963748000 con 0x5579636c4400 2024-05-08T21:23:11.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.556+0000 7fdc9f23c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557964542400 0x55796375a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.556+0000 7fdc9f23c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557964542400 0x55796375a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57540/0 (socket says 172.21.15.3:57540) 2024-05-08T21:23:11.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.556+0000 7fdc9f23c700 1 -- 172.21.15.3:0/1800473870 learned_addr learned my addr 172.21.15.3:0/1800473870 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.556+0000 7fdc9f23c700 1 --2- 172.21.15.3:0/1800473870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5579636c4400 0x55796375b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.556+0000 7fdc9fa3d700 1 --2- 172.21.15.3:0/1800473870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557964542000 0x55796375ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.557+0000 7fdc9f23c700 1 -- 172.21.15.3:0/1800473870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5579636c4400 msgr2=0x55796375b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.557+0000 7fdc9f23c700 1 --2- 172.21.15.3:0/1800473870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5579636c4400 0x55796375b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.557+0000 7fdc9f23c700 1 -- 172.21.15.3:0/1800473870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557964542000 msgr2=0x55796375ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.557+0000 7fdc9f23c700 1 --2- 172.21.15.3:0/1800473870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557964542000 0x55796375ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.557+0000 7fdc9f23c700 1 -- 172.21.15.3:0/1800473870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5579636e81a0 con 0x557964542400 2024-05-08T21:23:11.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.557+0000 7fdc9fa3d700 1 --2- 172.21.15.3:0/1800473870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557964542000 0x55796375ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:11.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.557+0000 7fdc9f23c700 1 --2- 172.21.15.3:0/1800473870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557964542400 0x55796375a680 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x5579639dc3c0 tx=0x55796445d080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.557+0000 7fdc99a31700 1 -- 172.21.15.3:0/1800473870 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5579636f51e0 con 0x557964542400 2024-05-08T21:23:11.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.557+0000 7fdc99a31700 1 -- 172.21.15.3:0/1800473870 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55796382e600 con 0x557964542400 2024-05-08T21:23:11.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.557+0000 7fdca1a91440 1 -- 172.21.15.3:0/1800473870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5579636e8000 con 0x557964542400 2024-05-08T21:23:11.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.557+0000 7fdc99a31700 1 -- 172.21.15.3:0/1800473870 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5579636f4dc0 con 0x557964542400 2024-05-08T21:23:11.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.557+0000 7fdca1a91440 1 -- 172.21.15.3:0/1800473870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5579636e89c0 con 0x557964542400 2024-05-08T21:23:11.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.557+0000 7fdca1a91440 4 main: JournalTool: resolving pool 6 2024-05-08T21:23:11.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.558+0000 7fdc99a31700 1 -- 172.21.15.3:0/1800473870 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x557963726840 con 0x557964542400 2024-05-08T21:23:11.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.559+0000 7fdc99a31700 1 --2- 172.21.15.3:0/1800473870 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x5579639f8400 0x55796375b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.559+0000 7fdc99a31700 1 -- 172.21.15.3:0/1800473870 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x557963702a80 con 0x557964542400 2024-05-08T21:23:11.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.559+0000 7fdca1a91440 4 main: JournalTool: creating IoCtx.. 2024-05-08T21:23:11.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.559+0000 7fdca1a91440 4 main: Executing for rank 0 2024-05-08T21:23:11.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.559+0000 7fdc9fa3d700 1 --2- 172.21.15.3:0/1800473870 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x5579639f8400 0x55796375b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.560+0000 7fdc9fa3d700 1 --2- 172.21.15.3:0/1800473870 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x5579639f8400 0x55796375b700 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x5579638242a0 tx=0x557963af4c00 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.560+0000 7fdca1a91440 1 -- start start 2024-05-08T21:23:11.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.560+0000 7fdca1a91440 1 Processor -- start 2024-05-08T21:23:11.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.560+0000 7fdca1a91440 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557964542c00 0x55796375a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.560+0000 7fdca1a91440 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557964543000 0x557963759b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.560+0000 7fdca1a91440 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x557964543400 0x557963759600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdca1a91440 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557963749040 con 0x557964542c00 2024-05-08T21:23:11.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdca1a91440 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x557963748500 con 0x557964543000 2024-05-08T21:23:11.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdca1a91440 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x557963748000 con 0x557964543400 2024-05-08T21:23:11.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdca023e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557964542c00 0x55796375a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdc9fa3d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557964543000 0x557963759b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdca023e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557964542c00 0x55796375a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57554/0 (socket says 172.21.15.3:57554) 2024-05-08T21:23:11.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdca023e700 1 -- 172.21.15.3:0/4081876050 learned_addr learned my addr 172.21.15.3:0/4081876050 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdc9f23c700 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x557964543400 0x557963759600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdca023e700 1 -- 172.21.15.3:0/4081876050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x557964543400 msgr2=0x557963759600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdca023e700 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x557964543400 0x557963759600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdca023e700 1 -- 172.21.15.3:0/4081876050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557964543000 msgr2=0x557963759b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdca023e700 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557964543000 0x557963759b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdca023e700 1 -- 172.21.15.3:0/4081876050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557963a2c4e0 con 0x557964542c00 2024-05-08T21:23:11.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdc9fa3d700 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557964543000 0x557963759b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:11.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdc9f23c700 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x557964543400 0x557963759600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:11.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdca023e700 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557964542c00 0x55796375a100 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x557963abab40 tx=0x55796452e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ee69f6af272c94 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdc96a2b700 1 -- 172.21.15.3:0/4081876050 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5579636f4f20 con 0x557964542c00 2024-05-08T21:23:11.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdc96a2b700 1 -- 172.21.15.3:0/4081876050 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55796382f500 con 0x557964542c00 2024-05-08T21:23:11.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdc96a2b700 1 -- 172.21.15.3:0/4081876050 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5579636f5600 con 0x557964542c00 2024-05-08T21:23:11.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.561+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557963a2c340 con 0x557964542c00 2024-05-08T21:23:11.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.562+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x557963a2bba0 con 0x557964542c00 2024-05-08T21:23:11.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.562+0000 7fdca1a91440 4 waiting for MDS map... 2024-05-08T21:23:11.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.562+0000 7fdc96a2b700 1 -- 172.21.15.3:0/4081876050 <== mon.0 v2:172.21.15.3:3300/0 4 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x5579637028c0 con 0x557964542c00 2024-05-08T21:23:11.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.562+0000 7fdc96a2b700 1 -- 172.21.15.3:0/4081876050 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 21) v1 ==== 3691+0+0 (secure 0 0 0) 0x557963b15b00 con 0x557964542c00 2024-05-08T21:23:11.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.562+0000 7fdca1a91440 4 Got MDS map 21 2024-05-08T21:23:11.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.562+0000 7fdca1a91440 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x557964543800 0x557963b66000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.562+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:1 6.f 6:f1f69726:::400.00000000:head [writefull 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x557964543c00 con 0x557964543800 2024-05-08T21:23:11.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.562+0000 7fdc9fa3d700 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x557964543800 0x557963b66000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.562+0000 7fdc9fa3d700 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x557964543800 0x557963b66000 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=a4616ae95de0d827 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.564+0000 7fdc9fa3d700 1 -- 172.21.15.3:0/4081876050 <== osd.2 v2:172.21.15.3:6808/47117 1 ==== osd_op_reply(1 400.00000000 [writefull 0~22] v42'1 uv1 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x557964524b40 con 0x557964543800 2024-05-08T21:23:11.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.564+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:2 6.14 6:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x557963b70000 con 0x557964543800 2024-05-08T21:23:11.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.566+0000 7fdc9fa3d700 1 -- 172.21.15.3:0/4081876050 <== osd.2 v2:172.21.15.3:6808/47117 2 ==== osd_op_reply(2 200.00000000 [writefull 0~90] v42'2 uv2 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x557964524b40 con 0x557964543800 2024-05-08T21:23:11.571 INFO:teuthology.orchestra.run.smithi003.stdout:writing EResetJournal entry 2024-05-08T21:23:11.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.566+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:3 6.14 6:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x557963b70400 con 0x557964543800 2024-05-08T21:23:11.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.566+0000 7fdca1a91440 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x557963b70800 0x557963b66580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.566+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:4 6.1b 6:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x557963b70c00 con 0x557963b70800 2024-05-08T21:23:11.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.566+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:5 6.b 6:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x557963b71000 con 0x557963b70800 2024-05-08T21:23:11.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.566+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- osd_op(unknown.0.0:6 6.1f 6:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x557963b71400 con 0x557963b70800 2024-05-08T21:23:11.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.566+0000 7fdca1a91440 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x557963b71800 0x557963b66b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.566+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:7 6.1a 6:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x557963b71c00 con 0x557963b71800 2024-05-08T21:23:11.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.566+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- osd_op(unknown.0.0:8 6.1e 6:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x557963b70000 con 0x557963b71800 2024-05-08T21:23:11.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.566+0000 7fdc9f23c700 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x557963b70800 0x557963b66580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.566+0000 7fdca023e700 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x557963b71800 0x557963b66b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.566+0000 7fdc9f23c700 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x557963b70800 0x557963b66580 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=ce2a3db4b5cc97a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.566+0000 7fdca023e700 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x557963b71800 0x557963b66b00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=53b55e028189d0af server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.567+0000 7fdc9fa3d700 1 -- 172.21.15.3:0/4081876050 <== osd.2 v2:172.21.15.3:6808/47117 3 ==== osd_op_reply(3 200.00000001 [delete] v42'3 uv2 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557964524b40 con 0x557964543800 2024-05-08T21:23:11.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.568+0000 7fdc9f23c700 1 -- 172.21.15.3:0/4081876050 <== osd.3 v2:172.21.15.3:6800/47118 1 ==== osd_op_reply(4 200.00000002 [delete] v42'1 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557964525200 con 0x557963b70800 2024-05-08T21:23:11.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.568+0000 7fdc95228700 1 -- 172.21.15.3:0/4081876050 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- osd_op(unknown.0.0:9 6.14 6:2e2fa760:::200.00000001:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x5579639f9400 con 0x557964543800 2024-05-08T21:23:11.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.568+0000 7fdca023e700 1 -- 172.21.15.3:0/4081876050 <== osd.4 v2:172.21.15.186:6810/47344 1 ==== osd_op_reply(7 200.00000005 [delete] v42'1 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5579645258c0 con 0x557963b71800 2024-05-08T21:23:11.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.568+0000 7fdca023e700 1 -- 172.21.15.3:0/4081876050 <== osd.4 v2:172.21.15.186:6810/47344 2 ==== osd_op_reply(8 200.00000006 [delete] v42'1 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5579645258c0 con 0x557963b71800 2024-05-08T21:23:11.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdc9fa3d700 1 -- 172.21.15.3:0/4081876050 <== osd.2 v2:172.21.15.3:6808/47117 4 ==== osd_op_reply(9 200.00000001 [write 0~48] v42'4 uv4 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x557964524b40 con 0x557964543800 2024-05-08T21:23:11.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdc9f23c700 1 -- 172.21.15.3:0/4081876050 <== osd.3 v2:172.21.15.3:6800/47118 2 ==== osd_op_reply(5 200.00000003 [delete] v42'2 uv1 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557964525200 con 0x557963b70800 2024-05-08T21:23:11.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdc9f23c700 1 -- 172.21.15.3:0/4081876050 <== osd.3 v2:172.21.15.3:6800/47118 3 ==== osd_op_reply(6 200.00000004 [delete] v42'5 uv4 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557964525200 con 0x557963b70800 2024-05-08T21:23:11.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 4 Successfully wrote new journal pointer and header for rank 3:0 2024-05-08T21:23:11.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x557964543800 msgr2=0x557963b66000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x557964543800 0x557963b66000 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x557963b70800 msgr2=0x557963b66580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x557963b70800 0x557963b66580 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x557963b71800 msgr2=0x557963b66b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x557963b71800 0x557963b66b00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557964542c00 msgr2=0x55796375a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdc9f23c700 1 -- 172.21.15.3:0/4081876050 reap_dead start 2024-05-08T21:23:11.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557964542c00 0x55796375a100 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x557963abab40 tx=0x55796452e060 comp rx=0 tx=0).stop 2024-05-08T21:23:11.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 shutdown_connections 2024-05-08T21:23:11.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 --2- 172.21.15.3:0/4081876050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557964542c00 0x55796375a100 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 >> 172.21.15.3:0/4081876050 conn(0x557964542800 msgr2=0x5579637b4000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 shutdown_connections 2024-05-08T21:23:11.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 -- 172.21.15.3:0/4081876050 wait complete. 2024-05-08T21:23:11.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 -- 172.21.15.3:0/1800473870 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x5579639f8400 msgr2=0x55796375b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 --2- 172.21.15.3:0/1800473870 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x5579639f8400 0x55796375b700 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x5579638242a0 tx=0x557963af4c00 comp rx=0 tx=0).stop 2024-05-08T21:23:11.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 -- 172.21.15.3:0/1800473870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557964542400 msgr2=0x55796375a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 --2- 172.21.15.3:0/1800473870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557964542400 0x55796375a680 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x5579639dc3c0 tx=0x55796445d080 comp rx=0 tx=0).stop 2024-05-08T21:23:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 -- 172.21.15.3:0/1800473870 shutdown_connections 2024-05-08T21:23:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 --2- 172.21.15.3:0/1800473870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5579636c4400 0x55796375b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 --2- 172.21.15.3:0/1800473870 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x5579639f8400 0x55796375b700 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 --2- 172.21.15.3:0/1800473870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557964542400 0x55796375a680 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 --2- 172.21.15.3:0/1800473870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557964542000 0x55796375ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.570+0000 7fdca1a91440 1 -- 172.21.15.3:0/1800473870 >> 172.21.15.3:0/1800473870 conn(0x5579636c5c00 msgr2=0x5579637b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.571+0000 7fdca1a91440 1 -- 172.21.15.3:0/1800473870 shutdown_connections 2024-05-08T21:23:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.571+0000 7fdca1a91440 1 -- 172.21.15.3:0/1800473870 wait complete. 2024-05-08T21:23:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.571+0000 7fdca1a91440 1 -- 172.21.15.3:0/1088322775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5579636c4800 msgr2=0x557963759080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.571+0000 7fdca1a91440 1 --2- 172.21.15.3:0/1088322775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5579636c4800 0x557963759080 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x557963824c30 tx=0x55796452e240 comp rx=0 tx=0).stop 2024-05-08T21:23:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.571+0000 7fdca1a91440 1 -- 172.21.15.3:0/1088322775 shutdown_connections 2024-05-08T21:23:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.571+0000 7fdca1a91440 1 --2- 172.21.15.3:0/1088322775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5579636c4800 0x557963759080 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.571+0000 7fdca1a91440 1 --2- 172.21.15.3:0/1088322775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5579636c5000 0x557963758580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.571+0000 7fdca1a91440 1 --2- 172.21.15.3:0/1088322775 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5579636c4c00 0x557963758b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.571+0000 7fdca1a91440 1 -- 172.21.15.3:0/1088322775 >> 172.21.15.3:0/1088322775 conn(0x5579636c5400 msgr2=0x5579637b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.571+0000 7fdca1a91440 1 -- 172.21.15.3:0/1088322775 shutdown_connections 2024-05-08T21:23:11.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.571+0000 7fdca1a91440 1 -- 172.21.15.3:0/1088322775 wait complete. 2024-05-08T21:23:11.580 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'recovery_fs:0', 'journal', 'reset', '--force'] in time 0:00:00.077552, result: writing EResetJournal entry 2024-05-08T21:23:11.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set recovery_fs joinable true 2024-05-08T21:23:11.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.721+0000 7f2422624700 1 Processor -- start 2024-05-08T21:23:11.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.721+0000 7f2422624700 1 -- start start 2024-05-08T21:23:11.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.722+0000 7f2422624700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f241c140760 0x7f241c06e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.722+0000 7f2422624700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15ecc0 0x7f241c06eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.722+0000 7f2422624700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241c140b40 con 0x7f241c15aa50 2024-05-08T21:23:11.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.722+0000 7f2422624700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241c0cc2b0 con 0x7f241c140760 2024-05-08T21:23:11.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.722+0000 7f2422624700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241c15ca30 con 0x7f241c15ecc0 2024-05-08T21:23:11.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.722+0000 7f241bfff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f241c140760 0x7f241c06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.722+0000 7f241b7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15ecc0 0x7f241c06eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.722+0000 7f241bfff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f241c140760 0x7f241c06e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:59128/0 (socket says 172.21.15.3:59128) 2024-05-08T21:23:11.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.722+0000 7f241b7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15ecc0 0x7f241c06eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.3:56318/0 (socket says 172.21.15.3:56318) 2024-05-08T21:23:11.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.722+0000 7f241bfff700 1 -- 172.21.15.3:0/3840749866 learned_addr learned my addr 172.21.15.3:0/3840749866 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.722+0000 7f241affd700 1 -- 172.21.15.3:0/3840749866 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616084064 0 0) 0x7f241c140b40 con 0x7f241c15aa50 2024-05-08T21:23:11.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.722+0000 7f241affd700 1 -- 172.21.15.3:0/3840749866 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24080035b0 con 0x7f241c15aa50 2024-05-08T21:23:11.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.722+0000 7f241bfff700 1 -- 172.21.15.3:0/3840749866 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15ecc0 msgr2=0x7f241c06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:11.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.722+0000 7f241bfff700 1 --2- 172.21.15.3:0/3840749866 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15ecc0 0x7f241c06eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.723+0000 7f241bfff700 1 -- 172.21.15.3:0/3840749866 >> v1:172.21.15.186:6789/0 conn(0x7f241c15aa50 legacy=0x7f241c15aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.723+0000 7f241bfff700 1 -- 172.21.15.3:0/3840749866 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c15b740 con 0x7f241c140760 2024-05-08T21:23:11.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.723+0000 7f241b7fe700 1 --2- 172.21.15.3:0/3840749866 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15ecc0 0x7f241c06eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:11.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.723+0000 7f241bfff700 1 --2- 172.21.15.3:0/3840749866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f241c140760 0x7f241c06e650 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f240c0098f0 tx=0x7f240c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de4dc18a0c87460e server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.723+0000 7f241affd700 1 -- 172.21.15.3:0/3840749866 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c011830 con 0x7f241c140760 2024-05-08T21:23:11.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.723+0000 7f241affd700 1 -- 172.21.15.3:0/3840749866 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240c011990 con 0x7f241c140760 2024-05-08T21:23:11.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.723+0000 7f241affd700 1 -- 172.21.15.3:0/3840749866 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c011d20 con 0x7f241c140760 2024-05-08T21:23:11.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.724+0000 7f2422624700 1 -- 172.21.15.3:0/3840749866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f241c140760 msgr2=0x7f241c06e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.724+0000 7f2422624700 1 --2- 172.21.15.3:0/3840749866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f241c140760 0x7f241c06e650 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f240c0098f0 tx=0x7f240c009c00 comp rx=0 tx=0).stop 2024-05-08T21:23:11.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.724+0000 7f2422624700 1 -- 172.21.15.3:0/3840749866 shutdown_connections 2024-05-08T21:23:11.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.724+0000 7f2422624700 1 --2- 172.21.15.3:0/3840749866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f241c140760 0x7f241c06e650 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.724+0000 7f2422624700 1 --2- 172.21.15.3:0/3840749866 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15ecc0 0x7f241c06eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.724+0000 7f2422624700 1 -- 172.21.15.3:0/3840749866 >> 172.21.15.3:0/3840749866 conn(0x7f241c0722d0 msgr2=0x7f241c06a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:11.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.724+0000 7f2422624700 1 -- 172.21.15.3:0/3840749866 shutdown_connections 2024-05-08T21:23:11.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.724+0000 7f2422624700 1 -- 172.21.15.3:0/3840749866 wait complete. 2024-05-08T21:23:11.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.725+0000 7f2422624700 1 Processor -- start 2024-05-08T21:23:11.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.725+0000 7f2422624700 1 -- start start 2024-05-08T21:23:11.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.726+0000 7f2422624700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f241c140760 0x7f241c0cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.726+0000 7f2422624700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f241c15aa50 0x7f241c0b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.726+0000 7f2422624700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15ecc0 0x7f241c151660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.726+0000 7f2422624700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0cbb10 con 0x7f241c15aa50 2024-05-08T21:23:11.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.726+0000 7f2422624700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0b3320 con 0x7f241c140760 2024-05-08T21:23:11.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.726+0000 7f2422624700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0b3600 con 0x7f241c15ecc0 2024-05-08T21:23:11.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.726+0000 7f241b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f241c15aa50 0x7f241c0b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.726+0000 7f241b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f241c15aa50 0x7f241c0b2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57558/0 (socket says 172.21.15.3:57558) 2024-05-08T21:23:11.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.726+0000 7f241b7fe700 1 -- 172.21.15.3:0/3656401711 learned_addr learned my addr 172.21.15.3:0/3656401711 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:11.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.726+0000 7f241bfff700 1 --2- 172.21.15.3:0/3656401711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f241c140760 0x7f241c0cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.726+0000 7f2420bc1700 1 --2- 172.21.15.3:0/3656401711 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15ecc0 0x7f241c151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.727+0000 7f241b7fe700 1 -- 172.21.15.3:0/3656401711 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15ecc0 msgr2=0x7f241c151660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.727+0000 7f241b7fe700 1 --2- 172.21.15.3:0/3656401711 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15ecc0 0x7f241c151660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.727+0000 7f241b7fe700 1 -- 172.21.15.3:0/3656401711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f241c140760 msgr2=0x7f241c0cb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:11.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.727+0000 7f241b7fe700 1 --2- 172.21.15.3:0/3656401711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f241c140760 0x7f241c0cb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:11.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.727+0000 7f241b7fe700 1 -- 172.21.15.3:0/3656401711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24040094a0 con 0x7f241c15aa50 2024-05-08T21:23:11.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.727+0000 7f241b7fe700 1 --2- 172.21.15.3:0/3656401711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f241c15aa50 0x7f241c0b2af0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f240400ebe0 tx=0x7f240400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.727+0000 7f2418ff9700 1 -- 172.21.15.3:0/3656401711 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2404009760 con 0x7f241c15aa50 2024-05-08T21:23:11.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.727+0000 7f2418ff9700 1 -- 172.21.15.3:0/3656401711 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240400c850 con 0x7f241c15aa50 2024-05-08T21:23:11.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.727+0000 7f2422624700 1 -- 172.21.15.3:0/3656401711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240c009550 con 0x7f241c15aa50 2024-05-08T21:23:11.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.727+0000 7f2418ff9700 1 -- 172.21.15.3:0/3656401711 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240400cbe0 con 0x7f241c15aa50 2024-05-08T21:23:11.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.727+0000 7f2422624700 1 -- 172.21.15.3:0/3656401711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241c0b3bc0 con 0x7f241c15aa50 2024-05-08T21:23:11.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.728+0000 7f2422624700 1 -- 172.21.15.3:0/3656401711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241c074bc0 con 0x7f241c15aa50 2024-05-08T21:23:11.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.728+0000 7f2418ff9700 1 -- 172.21.15.3:0/3656401711 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f240400cd40 con 0x7f241c15aa50 2024-05-08T21:23:11.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.729+0000 7f2418ff9700 1 --2- 172.21.15.3:0/3656401711 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f23fc06ac90 0x7f23fc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:11.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.729+0000 7f2418ff9700 1 -- 172.21.15.3:0/3656401711 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f2404089470 con 0x7f241c15aa50 2024-05-08T21:23:11.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.729+0000 7f241bfff700 1 --2- 172.21.15.3:0/3656401711 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f23fc06ac90 0x7f23fc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:11.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.730+0000 7f241bfff700 1 --2- 172.21.15.3:0/3656401711 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f23fc06ac90 0x7f23fc06d150 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f240c009c40 tx=0x7f240c008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:11.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:11.734+0000 7f2418ff9700 1 -- 172.21.15.3:0/3656401711 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f24040544c0 con 0x7f241c15aa50 2024-05-08T21:23:12.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.003+0000 7f2422624700 1 -- 172.21.15.3:0/3656401711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "recovery_fs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f241c13fac0 con 0x7f241c15aa50 2024-05-08T21:23:12.085 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-08T21:23:12.087 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-08T21:23:12.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.656+0000 7f2418ff9700 1 -- 172.21.15.3:0/3656401711 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "recovery_fs", "var": "joinable", "val": "true"}]=0 recovery_fs marked joinable; MDS may join as newly active. v22) v1 ==== 172+0+0 (secure 0 0 0) 0x7f2404057ae0 con 0x7f241c15aa50 2024-05-08T21:23:12.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.659+0000 7f2422624700 1 -- 172.21.15.3:0/3656401711 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f23fc06ac90 msgr2=0x7f23fc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:12.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.659+0000 7f2422624700 1 --2- 172.21.15.3:0/3656401711 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f23fc06ac90 0x7f23fc06d150 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f240c009c40 tx=0x7f240c008030 comp rx=0 tx=0).stop 2024-05-08T21:23:12.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.659+0000 7f2422624700 1 -- 172.21.15.3:0/3656401711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f241c15aa50 msgr2=0x7f241c0b2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:12.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.659+0000 7f2422624700 1 --2- 172.21.15.3:0/3656401711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f241c15aa50 0x7f241c0b2af0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f240400ebe0 tx=0x7f240400eef0 comp rx=0 tx=0).stop 2024-05-08T21:23:12.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.659+0000 7f2422624700 1 -- 172.21.15.3:0/3656401711 shutdown_connections 2024-05-08T21:23:12.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.659+0000 7f2422624700 1 --2- 172.21.15.3:0/3656401711 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15ecc0 0x7f241c151660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:12.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.659+0000 7f2422624700 1 --2- 172.21.15.3:0/3656401711 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f23fc06ac90 0x7f23fc06d150 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:12.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.659+0000 7f2422624700 1 --2- 172.21.15.3:0/3656401711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f241c140760 0x7f241c0cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:12.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.659+0000 7f2422624700 1 --2- 172.21.15.3:0/3656401711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f241c15aa50 0x7f241c0b2af0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:12.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.659+0000 7f2422624700 1 -- 172.21.15.3:0/3656401711 >> 172.21.15.3:0/3656401711 conn(0x7f241c0722d0 msgr2=0x7f241c15bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:12.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.660+0000 7f2422624700 1 -- 172.21.15.3:0/3656401711 shutdown_connections 2024-05-08T21:23:12.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.660+0000 7f2422624700 1 -- 172.21.15.3:0/3656401711 wait complete. 2024-05-08T21:23:12.662 INFO:teuthology.orchestra.run.smithi003.stderr:recovery_fs marked joinable; MDS may join as newly active. 2024-05-08T21:23:12.663 INFO:tasks.ceph.mds.d.smithi186.stderr:2024-05-08T21:23:12.662+0000 7f6cfa1b6700 -1 mds.0.openfiles _load_finish got (2) No such file or directory 2024-05-08T21:23:12.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:23:12.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.816+0000 7fc56367c700 1 Processor -- start 2024-05-08T21:23:12.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.816+0000 7fc56367c700 1 -- start start 2024-05-08T21:23:12.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.817+0000 7fc56367c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc55c06a960 0x7fc55c0ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:12.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.817+0000 7fc56367c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc55c0cf760 0x7fc55c0cfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.817+0000 7fc56367c700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c0cdbb0 con 0x7fc55c06b140 2024-05-08T21:23:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.817+0000 7fc56367c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc55c0d00f0 con 0x7fc55c0cf760 2024-05-08T21:23:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.817+0000 7fc56367c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c0cd310 con 0x7fc55c06a960 2024-05-08T21:23:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.817+0000 7fc561c19700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc55c0cf760 0x7fc55c0cfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.817+0000 7fc561c19700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc55c0cf760 0x7fc55c0cfbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57568/0 (socket says 172.21.15.3:57568) 2024-05-08T21:23:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.817+0000 7fc561c19700 1 -- 172.21.15.3:0/4113308716 learned_addr learned my addr 172.21.15.3:0/4113308716 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.817+0000 7fc560c17700 1 --2- 172.21.15.3:0/4113308716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc55c06a960 0x7fc55c0ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.817+0000 7fc561c19700 1 -- 172.21.15.3:0/4113308716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc55c06a960 msgr2=0x7fc55c0ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.818+0000 7fc561c19700 1 --2- 172.21.15.3:0/4113308716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc55c06a960 0x7fc55c0ccd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.818+0000 7fc561c19700 1 -- 172.21.15.3:0/4113308716 >> v1:172.21.15.186:6789/0 conn(0x7fc55c06b140 legacy=0x7fc55c15bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.818+0000 7fc561c19700 1 -- 172.21.15.3:0/4113308716 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c0cd5f0 con 0x7fc55c0cf760 2024-05-08T21:23:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.818+0000 7fc560c17700 1 --2- 172.21.15.3:0/4113308716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc55c06a960 0x7fc55c0ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.818+0000 7fc561c19700 1 --2- 172.21.15.3:0/4113308716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc55c0cf760 0x7fc55c0cfbb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc554009e50 tx=0x7fc55400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d4e286ad77a4aa server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.818+0000 7fc553fff700 1 -- 172.21.15.3:0/4113308716 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55400e070 con 0x7fc55c0cf760 2024-05-08T21:23:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.818+0000 7fc553fff700 1 -- 172.21.15.3:0/4113308716 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55400b9c0 con 0x7fc55c0cf760 2024-05-08T21:23:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.818+0000 7fc553fff700 1 -- 172.21.15.3:0/4113308716 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55400bd70 con 0x7fc55c0cf760 2024-05-08T21:23:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.818+0000 7fc56367c700 1 -- 172.21.15.3:0/4113308716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc55c0cf760 msgr2=0x7fc55c0cfbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.818+0000 7fc56367c700 1 --2- 172.21.15.3:0/4113308716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc55c0cf760 0x7fc55c0cfbb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc554009e50 tx=0x7fc55400b330 comp rx=0 tx=0).stop 2024-05-08T21:23:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.819+0000 7fc56367c700 1 -- 172.21.15.3:0/4113308716 shutdown_connections 2024-05-08T21:23:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.819+0000 7fc56367c700 1 --2- 172.21.15.3:0/4113308716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc55c0cf760 0x7fc55c0cfbb0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.819+0000 7fc56367c700 1 --2- 172.21.15.3:0/4113308716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc55c06a960 0x7fc55c0ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.819+0000 7fc56367c700 1 -- 172.21.15.3:0/4113308716 >> 172.21.15.3:0/4113308716 conn(0x7fc55c0d0320 msgr2=0x7fc55c0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.819+0000 7fc56367c700 1 -- 172.21.15.3:0/4113308716 shutdown_connections 2024-05-08T21:23:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.819+0000 7fc56367c700 1 -- 172.21.15.3:0/4113308716 wait complete. 2024-05-08T21:23:12.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.820+0000 7fc56367c700 1 Processor -- start 2024-05-08T21:23:12.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.820+0000 7fc56367c700 1 -- start start 2024-05-08T21:23:12.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc56367c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc55c06a960 0x7fc55c0c6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:12.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc56367c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc55c06b140 0x7fc55c06f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:12.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc56367c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc55c0cf760 0x7fc55c1f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc56367c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c06fc20 con 0x7fc55c0cf760 2024-05-08T21:23:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc56367c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c06e6e0 con 0x7fc55c06b140 2024-05-08T21:23:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc56367c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c06e9a0 con 0x7fc55c06a960 2024-05-08T21:23:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc561c19700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc55c0cf760 0x7fc55c1f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc561c19700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc55c0cf760 0x7fc55c1f3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57580/0 (socket says 172.21.15.3:57580) 2024-05-08T21:23:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc561c19700 1 -- 172.21.15.3:0/2806199761 learned_addr learned my addr 172.21.15.3:0/2806199761 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc561418700 1 --2- 172.21.15.3:0/2806199761 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc55c06a960 0x7fc55c0c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc560c17700 1 --2- 172.21.15.3:0/2806199761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc55c06b140 0x7fc55c06f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc561c19700 1 -- 172.21.15.3:0/2806199761 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc55c06a960 msgr2=0x7fc55c0c6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc561c19700 1 --2- 172.21.15.3:0/2806199761 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc55c06a960 0x7fc55c0c6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc561c19700 1 -- 172.21.15.3:0/2806199761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc55c06b140 msgr2=0x7fc55c06f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc561c19700 1 --2- 172.21.15.3:0/2806199761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc55c06b140 0x7fc55c06f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc561c19700 1 -- 172.21.15.3:0/2806199761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc554009b30 con 0x7fc55c0cf760 2024-05-08T21:23:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.821+0000 7fc561c19700 1 --2- 172.21.15.3:0/2806199761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc55c0cf760 0x7fc55c1f3210 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0718a0 tx=0x7fc554004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.822+0000 7fc551ffb700 1 -- 172.21.15.3:0/2806199761 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc554004800 con 0x7fc55c0cf760 2024-05-08T21:23:12.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.822+0000 7fc551ffb700 1 -- 172.21.15.3:0/2806199761 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5540092e0 con 0x7fc55c0cf760 2024-05-08T21:23:12.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.822+0000 7fc56367c700 1 -- 172.21.15.3:0/2806199761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc55c0c3cc0 con 0x7fc55c0cf760 2024-05-08T21:23:12.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.822+0000 7fc551ffb700 1 -- 172.21.15.3:0/2806199761 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55401c650 con 0x7fc55c0cf760 2024-05-08T21:23:12.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.822+0000 7fc56367c700 1 -- 172.21.15.3:0/2806199761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55c06ef60 con 0x7fc55c0cf760 2024-05-08T21:23:12.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.823+0000 7fc56367c700 1 -- 172.21.15.3:0/2806199761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc55c068000 con 0x7fc55c0cf760 2024-05-08T21:23:12.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.824+0000 7fc551ffb700 1 -- 172.21.15.3:0/2806199761 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc55401c7b0 con 0x7fc55c0cf760 2024-05-08T21:23:12.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.824+0000 7fc551ffb700 1 --2- 172.21.15.3:0/2806199761 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc54006ac90 0x7fc54006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:12.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.824+0000 7fc551ffb700 1 -- 172.21.15.3:0/2806199761 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7fc55408ae20 con 0x7fc55c0cf760 2024-05-08T21:23:12.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.824+0000 7fc561418700 1 --2- 172.21.15.3:0/2806199761 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc54006ac90 0x7fc54006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:12.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.825+0000 7fc561418700 1 --2- 172.21.15.3:0/2806199761 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc54006ac90 0x7fc54006d150 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc558007fd0 tx=0x7fc558009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:12.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:12.829+0000 7fc551ffb700 1 -- 172.21.15.3:0/2806199761 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc554055dd0 con 0x7fc55c0cf760 2024-05-08T21:23:13.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:13.098+0000 7fc56367c700 1 -- 172.21.15.3:0/2806199761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc55c15c4b0 con 0x7fc55c0cf760 2024-05-08T21:23:13.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:13.098+0000 7fc551ffb700 1 -- 172.21.15.3:0/2806199761 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5936 (secure 0 0 0) 0x7fc5540593f0 con 0x7fc55c0cf760 2024-05-08T21:23:13.100 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:13.100 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6833/2104267343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2104267343},{"type":"v1","addr":"172.21.15.3:6833","nonce":2104267343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/208321708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":208321708},{"type":"v1","addr":"172.21.15.3:6835","nonce":208321708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/1591386131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1591386131},{"type":"v1","addr":"172.21.15.186:6836","nonce":1591386131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:22:50.184245+0000","modified":"2024-05-08T21:23:01.563810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2},{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:23:05.475355+0000","modified":"2024-05-08T21:23:12.657462+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"d","rank":0,"incarnation":23,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/2000072997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":2000072997},{"type":"v1","addr":"172.21.15.186:6837","nonce":2000072997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":6,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-08T21:23:13.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:13.100+0000 7fc56367c700 1 -- 172.21.15.3:0/2806199761 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc54006ac90 msgr2=0x7fc54006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:13.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:13.100+0000 7fc56367c700 1 --2- 172.21.15.3:0/2806199761 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc54006ac90 0x7fc54006d150 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc558007fd0 tx=0x7fc558009040 comp rx=0 tx=0).stop 2024-05-08T21:23:13.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:13.100+0000 7fc56367c700 1 -- 172.21.15.3:0/2806199761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc55c0cf760 msgr2=0x7fc55c1f3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:13.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:13.100+0000 7fc56367c700 1 --2- 172.21.15.3:0/2806199761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc55c0cf760 0x7fc55c1f3210 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0718a0 tx=0x7fc554004ad0 comp rx=0 tx=0).stop 2024-05-08T21:23:13.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:13.100+0000 7fc56367c700 1 -- 172.21.15.3:0/2806199761 shutdown_connections 2024-05-08T21:23:13.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:13.100+0000 7fc56367c700 1 --2- 172.21.15.3:0/2806199761 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc55c06a960 0x7fc55c0c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:13.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:13.101+0000 7fc56367c700 1 --2- 172.21.15.3:0/2806199761 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc54006ac90 0x7fc54006d150 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:13.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:13.101+0000 7fc56367c700 1 --2- 172.21.15.3:0/2806199761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc55c0cf760 0x7fc55c1f3210 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:13.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:13.101+0000 7fc56367c700 1 --2- 172.21.15.3:0/2806199761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc55c06b140 0x7fc55c06f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:13.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:13.101+0000 7fc56367c700 1 -- 172.21.15.3:0/2806199761 >> 172.21.15.3:0/2806199761 conn(0x7fc55c0d0320 msgr2=0x7fc55c0ad0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:13.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:13.101+0000 7fc56367c700 1 -- 172.21.15.3:0/2806199761 shutdown_connections 2024-05-08T21:23:13.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:13.101+0000 7fc56367c700 1 -- 172.21.15.3:0/2806199761 wait complete. 2024-05-08T21:23:13.103 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 23 2024-05-08T21:23:13.262 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-08T21:23:13.261+0000 7f74f9e57700 -1 log_channel(cluster) log [ERR] : Health check update: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-08T21:23:13.518 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-08T21:23:05.475355+0000', 'modified': '2024-05-08T21:23:12.657462+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5191}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5191': {'gid': 5191, 'name': 'd', 'rank': 0, 'incarnation': 23, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.186:6837/2000072997', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6835', 'nonce': 2000072997}, {'type': 'v1', 'addr': '172.21.15.186:6837', 'nonce': 2000072997}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [5], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'recovery_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-05-08T21:23:13.519 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5191:up:replay 2024-05-08T21:23:14.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:23:14.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.664+0000 7fd876421700 1 Processor -- start 2024-05-08T21:23:14.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.664+0000 7fd876421700 1 -- start start 2024-05-08T21:23:14.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.665+0000 7fd876421700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd8700d13e0 0x7fd8700d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:14.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.665+0000 7fd876421700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd87015db60 0x7fd8700d2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:14.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.665+0000 7fd876421700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8700b4430 con 0x7fd87015d780 2024-05-08T21:23:14.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.665+0000 7fd876421700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8700d45c0 con 0x7fd8700d13e0 2024-05-08T21:23:14.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.665+0000 7fd876421700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8700b3820 con 0x7fd87015db60 2024-05-08T21:23:14.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.666+0000 7fd86f7fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd8700d13e0 0x7fd8700d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:14.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.666+0000 7fd8749be700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd87015db60 0x7fd8700d2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:14.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.666+0000 7fd86f7fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd8700d13e0 0x7fd8700d1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:59168/0 (socket says 172.21.15.3:59168) 2024-05-08T21:23:14.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.666+0000 7fd86f7fe700 1 -- 172.21.15.3:0/2374246434 learned_addr learned my addr 172.21.15.3:0/2374246434 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:14.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.666+0000 7fd86f7fe700 1 -- 172.21.15.3:0/2374246434 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd87015db60 msgr2=0x7fd8700d2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:14.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.666+0000 7fd86f7fe700 1 --2- 172.21.15.3:0/2374246434 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd87015db60 0x7fd8700d2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:14.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.666+0000 7fd86f7fe700 1 -- 172.21.15.3:0/2374246434 >> v1:172.21.15.186:6789/0 conn(0x7fd87015d780 legacy=0x7fd8700b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:14.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.666+0000 7fd86f7fe700 1 -- 172.21.15.3:0/2374246434 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8700b3b00 con 0x7fd8700d13e0 2024-05-08T21:23:14.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.666+0000 7fd8749be700 1 --2- 172.21.15.3:0/2374246434 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd87015db60 0x7fd8700d2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:14.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.666+0000 7fd86f7fe700 1 --2- 172.21.15.3:0/2374246434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8700d13e0 0x7fd8700d1be0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd8640098f0 tx=0x7fd864009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1808de857c6efd65 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:14.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.668+0000 7fd86effd700 1 -- 172.21.15.3:0/2374246434 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd864011830 con 0x7fd8700d13e0 2024-05-08T21:23:14.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.668+0000 7fd86effd700 1 -- 172.21.15.3:0/2374246434 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd864011990 con 0x7fd8700d13e0 2024-05-08T21:23:14.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.668+0000 7fd86effd700 1 -- 172.21.15.3:0/2374246434 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd864011d20 con 0x7fd8700d13e0 2024-05-08T21:23:14.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.669+0000 7fd876421700 1 -- 172.21.15.3:0/2374246434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8700d13e0 msgr2=0x7fd8700d1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:14.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.669+0000 7fd876421700 1 --2- 172.21.15.3:0/2374246434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8700d13e0 0x7fd8700d1be0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd8640098f0 tx=0x7fd864009c00 comp rx=0 tx=0).stop 2024-05-08T21:23:14.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.669+0000 7fd876421700 1 -- 172.21.15.3:0/2374246434 shutdown_connections 2024-05-08T21:23:14.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.669+0000 7fd876421700 1 --2- 172.21.15.3:0/2374246434 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd87015db60 0x7fd8700d2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:14.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.669+0000 7fd876421700 1 --2- 172.21.15.3:0/2374246434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8700d13e0 0x7fd8700d1be0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:14.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.669+0000 7fd876421700 1 -- 172.21.15.3:0/2374246434 >> 172.21.15.3:0/2374246434 conn(0x7fd87008afe0 msgr2=0x7fd8700d3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:14.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.669+0000 7fd876421700 1 -- 172.21.15.3:0/2374246434 shutdown_connections 2024-05-08T21:23:14.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.669+0000 7fd876421700 1 -- 172.21.15.3:0/2374246434 wait complete. 2024-05-08T21:23:14.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.670+0000 7fd876421700 1 Processor -- start 2024-05-08T21:23:14.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.670+0000 7fd876421700 1 -- start start 2024-05-08T21:23:14.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.671+0000 7fd876421700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8700d13e0 0x7fd8701e74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:14.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.671+0000 7fd876421700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd87015d780 0x7fd8700cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:14.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.671+0000 7fd876421700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd87015db60 0x7fd8700ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:14.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.671+0000 7fd876421700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8701e61b0 con 0x7fd8700d13e0 2024-05-08T21:23:14.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.671+0000 7fd876421700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8701e6400 con 0x7fd87015d780 2024-05-08T21:23:14.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.671+0000 7fd876421700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8701e66e0 con 0x7fd87015db60 2024-05-08T21:23:14.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.671+0000 7fd86ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8700d13e0 0x7fd8701e74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:14.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.671+0000 7fd86ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8700d13e0 0x7fd8701e74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57592/0 (socket says 172.21.15.3:57592) 2024-05-08T21:23:14.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.671+0000 7fd86ffff700 1 -- 172.21.15.3:0/952927484 learned_addr learned my addr 172.21.15.3:0/952927484 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:14.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.671+0000 7fd86f7fe700 1 --2- 172.21.15.3:0/952927484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd87015d780 0x7fd8700cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:14.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.671+0000 7fd8749be700 1 --2- 172.21.15.3:0/952927484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd87015db60 0x7fd8700ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:14.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.672+0000 7fd86ffff700 1 -- 172.21.15.3:0/952927484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd87015db60 msgr2=0x7fd8700ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:14.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.672+0000 7fd86ffff700 1 --2- 172.21.15.3:0/952927484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd87015db60 0x7fd8700ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:14.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.672+0000 7fd86ffff700 1 -- 172.21.15.3:0/952927484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd87015d780 msgr2=0x7fd8700cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:14.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.672+0000 7fd86ffff700 1 --2- 172.21.15.3:0/952927484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd87015d780 0x7fd8700cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:14.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.672+0000 7fd86ffff700 1 -- 172.21.15.3:0/952927484 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd868009a70 con 0x7fd8700d13e0 2024-05-08T21:23:14.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.672+0000 7fd8749be700 1 --2- 172.21.15.3:0/952927484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd87015db60 0x7fd8700ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:14.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.672+0000 7fd86ffff700 1 --2- 172.21.15.3:0/952927484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8700d13e0 0x7fd8701e74a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd86000c4d0 tx=0x7fd86000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:14.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.672+0000 7fd86cff9700 1 -- 172.21.15.3:0/952927484 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8600052d0 con 0x7fd8700d13e0 2024-05-08T21:23:14.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.672+0000 7fd876421700 1 -- 172.21.15.3:0/952927484 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd864009550 con 0x7fd8700d13e0 2024-05-08T21:23:14.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.672+0000 7fd86cff9700 1 -- 172.21.15.3:0/952927484 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd860005430 con 0x7fd8700d13e0 2024-05-08T21:23:14.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.672+0000 7fd86cff9700 1 -- 172.21.15.3:0/952927484 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8600038f0 con 0x7fd8700d13e0 2024-05-08T21:23:14.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.672+0000 7fd876421700 1 -- 172.21.15.3:0/952927484 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8701e6ca0 con 0x7fd8700d13e0 2024-05-08T21:23:14.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.673+0000 7fd876421700 1 -- 172.21.15.3:0/952927484 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8700d1a50 con 0x7fd8700d13e0 2024-05-08T21:23:14.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.674+0000 7fd86cff9700 1 -- 172.21.15.3:0/952927484 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd860003a50 con 0x7fd8700d13e0 2024-05-08T21:23:14.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.675+0000 7fd86cff9700 1 --2- 172.21.15.3:0/952927484 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd85c06ac90 0x7fd85c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:14.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.675+0000 7fd86cff9700 1 -- 172.21.15.3:0/952927484 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7fd860088e90 con 0x7fd8700d13e0 2024-05-08T21:23:14.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.675+0000 7fd86f7fe700 1 --2- 172.21.15.3:0/952927484 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd85c06ac90 0x7fd85c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:14.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.676+0000 7fd86f7fe700 1 --2- 172.21.15.3:0/952927484 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd85c06ac90 0x7fd85c06d150 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd864009c40 tx=0x7fd864008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:14.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.679+0000 7fd86cff9700 1 -- 172.21.15.3:0/952927484 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd860053e50 con 0x7fd8700d13e0 2024-05-08T21:23:14.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.946+0000 7fd876421700 1 -- 172.21.15.3:0/952927484 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8700678f0 con 0x7fd8700d13e0 2024-05-08T21:23:14.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.947+0000 7fd86cff9700 1 -- 172.21.15.3:0/952927484 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5936 (secure 0 0 0) 0x7fd860057470 con 0x7fd8700d13e0 2024-05-08T21:23:14.948 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:14.948 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":25,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6833/2104267343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2104267343},{"type":"v1","addr":"172.21.15.3:6833","nonce":2104267343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/208321708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":208321708},{"type":"v1","addr":"172.21.15.3:6835","nonce":208321708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/1591386131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1591386131},{"type":"v1","addr":"172.21.15.186:6836","nonce":1591386131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:22:50.184245+0000","modified":"2024-05-08T21:23:01.563810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2},{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:23:05.475355+0000","modified":"2024-05-08T21:23:14.660518+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"d","rank":0,"incarnation":23,"state":"up:rejoin","state_seq":5,"addr":"172.21.15.186:6837/2000072997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":2000072997},{"type":"v1","addr":"172.21.15.186:6837","nonce":2000072997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":6,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-08T21:23:14.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.949+0000 7fd876421700 1 -- 172.21.15.3:0/952927484 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd85c06ac90 msgr2=0x7fd85c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:14.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.949+0000 7fd876421700 1 --2- 172.21.15.3:0/952927484 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd85c06ac90 0x7fd85c06d150 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd864009c40 tx=0x7fd864008030 comp rx=0 tx=0).stop 2024-05-08T21:23:14.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.949+0000 7fd876421700 1 -- 172.21.15.3:0/952927484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8700d13e0 msgr2=0x7fd8701e74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:14.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.949+0000 7fd876421700 1 --2- 172.21.15.3:0/952927484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8700d13e0 0x7fd8701e74a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd86000c4d0 tx=0x7fd86000c7e0 comp rx=0 tx=0).stop 2024-05-08T21:23:14.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.949+0000 7fd876421700 1 -- 172.21.15.3:0/952927484 shutdown_connections 2024-05-08T21:23:14.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.949+0000 7fd876421700 1 --2- 172.21.15.3:0/952927484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd87015db60 0x7fd8700ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:14.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.949+0000 7fd876421700 1 --2- 172.21.15.3:0/952927484 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd85c06ac90 0x7fd85c06d150 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:14.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.949+0000 7fd876421700 1 --2- 172.21.15.3:0/952927484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8700d13e0 0x7fd8701e74a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:14.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.949+0000 7fd876421700 1 --2- 172.21.15.3:0/952927484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd87015d780 0x7fd8700cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:14.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.949+0000 7fd876421700 1 -- 172.21.15.3:0/952927484 >> 172.21.15.3:0/952927484 conn(0x7fd87008afe0 msgr2=0x7fd8700ced30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:14.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.949+0000 7fd876421700 1 -- 172.21.15.3:0/952927484 shutdown_connections 2024-05-08T21:23:14.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:14.949+0000 7fd876421700 1 -- 172.21.15.3:0/952927484 wait complete. 2024-05-08T21:23:14.952 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 25 2024-05-08T21:23:15.366 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-08T21:23:05.475355+0000', 'modified': '2024-05-08T21:23:14.660518+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5191}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5191': {'gid': 5191, 'name': 'd', 'rank': 0, 'incarnation': 23, 'state': 'up:rejoin', 'state_seq': 5, 'addr': '172.21.15.186:6837/2000072997', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6835', 'nonce': 2000072997}, {'type': 'v1', 'addr': '172.21.15.186:6837', 'nonce': 2000072997}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [5], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'recovery_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-05-08T21:23:15.367 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5191:up:rejoin 2024-05-08T21:23:16.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:23:16.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.513+0000 7f245f036700 1 Processor -- start 2024-05-08T21:23:16.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.513+0000 7f245f036700 1 -- start start 2024-05-08T21:23:16.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.514+0000 7f245f036700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f24580ccbd0 0x7f2458168580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.514+0000 7f245f036700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f245809a870 con 0x7f2458161030 2024-05-08T21:23:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.514+0000 7f245f036700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24580d0840 con 0x7f24580b2480 2024-05-08T21:23:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.514+0000 7f245f036700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f245806ccc0 con 0x7f24580ccbd0 2024-05-08T21:23:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.514+0000 7f245d5d3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f24580ccbd0 0x7f2458168580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.514+0000 7f245cdd2700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2458161030 0x7f2458099a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51122/0 (socket says 172.21.15.3:51122) 2024-05-08T21:23:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.514+0000 7f245d5d3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f24580ccbd0 0x7f2458168580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57602/0 (socket says 172.21.15.3:57602) 2024-05-08T21:23:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.514+0000 7f245cdd2700 1 -- 172.21.15.3:0/3564978999 learned_addr learned my addr 172.21.15.3:0/3564978999 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.515+0000 7f245d5d3700 1 -- 172.21.15.3:0/3564978999 >> v1:172.21.15.186:6789/0 conn(0x7f24580b2480 legacy=0x7f245809a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.515+0000 7f245d5d3700 1 -- 172.21.15.3:0/3564978999 >> v1:172.21.15.3:6789/0 conn(0x7f2458161030 legacy=0x7f2458099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.515+0000 7f24577fe700 1 -- 172.21.15.3:0/3564978999 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970027767 0 0) 0x7f245809a870 con 0x7f2458161030 2024-05-08T21:23:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.515+0000 7f245d5d3700 1 -- 172.21.15.3:0/3564978999 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245806c600 con 0x7f24580ccbd0 2024-05-08T21:23:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.515+0000 7f245d5d3700 1 --2- 172.21.15.3:0/3564978999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24580ccbd0 0x7f2458168580 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2450009d90 tx=0x7f245000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5127b2a6c5ea0a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.515+0000 7f24577fe700 1 -- 172.21.15.3:0/3564978999 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245000e070 con 0x7f24580ccbd0 2024-05-08T21:23:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.515+0000 7f24577fe700 1 -- 172.21.15.3:0/3564978999 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245000bbd0 con 0x7f24580ccbd0 2024-05-08T21:23:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.515+0000 7f24577fe700 1 -- 172.21.15.3:0/3564978999 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2450012430 con 0x7f24580ccbd0 2024-05-08T21:23:16.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.516+0000 7f245f036700 1 -- 172.21.15.3:0/3564978999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24580ccbd0 msgr2=0x7f2458168580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:16.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.516+0000 7f245f036700 1 --2- 172.21.15.3:0/3564978999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24580ccbd0 0x7f2458168580 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2450009d90 tx=0x7f245000b330 comp rx=0 tx=0).stop 2024-05-08T21:23:16.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.516+0000 7f245f036700 1 -- 172.21.15.3:0/3564978999 shutdown_connections 2024-05-08T21:23:16.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.516+0000 7f245f036700 1 --2- 172.21.15.3:0/3564978999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24580ccbd0 0x7f2458168580 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:16.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.516+0000 7f245f036700 1 -- 172.21.15.3:0/3564978999 >> 172.21.15.3:0/3564978999 conn(0x7f24580b2f20 msgr2=0x7f24580b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:16.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.516+0000 7f245f036700 1 -- 172.21.15.3:0/3564978999 shutdown_connections 2024-05-08T21:23:16.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.516+0000 7f245f036700 1 -- 172.21.15.3:0/3564978999 wait complete. 2024-05-08T21:23:16.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.518+0000 7f245f036700 1 Processor -- start 2024-05-08T21:23:16.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.518+0000 7f245f036700 1 -- start start 2024-05-08T21:23:16.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f245f036700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24580b2480 0x7f2458155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:16.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f245f036700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24580ccbd0 0x7f24581569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:16.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f245f036700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2458161030 0x7f245814f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:16.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f245f036700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24580d0660 con 0x7f24580ccbd0 2024-05-08T21:23:16.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f245f036700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f245814fcf0 con 0x7f2458161030 2024-05-08T21:23:16.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f245f036700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2458156ef0 con 0x7f24580b2480 2024-05-08T21:23:16.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f2457fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24580ccbd0 0x7f24581569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:16.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f2457fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24580ccbd0 0x7f24581569b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57610/0 (socket says 172.21.15.3:57610) 2024-05-08T21:23:16.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f2457fff700 1 -- 172.21.15.3:0/1285400895 learned_addr learned my addr 172.21.15.3:0/1285400895 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:16.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f245d5d3700 1 --2- 172.21.15.3:0/1285400895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2458161030 0x7f245814f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:16.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f245cdd2700 1 --2- 172.21.15.3:0/1285400895 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24580b2480 0x7f2458155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:16.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f2457fff700 1 -- 172.21.15.3:0/1285400895 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24580b2480 msgr2=0x7f2458155210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:16.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f2457fff700 1 --2- 172.21.15.3:0/1285400895 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24580b2480 0x7f2458155210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:16.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f2457fff700 1 -- 172.21.15.3:0/1285400895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2458161030 msgr2=0x7f245814f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:16.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f2457fff700 1 --2- 172.21.15.3:0/1285400895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2458161030 0x7f245814f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:16.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f2457fff700 1 -- 172.21.15.3:0/1285400895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2450009a70 con 0x7f24580ccbd0 2024-05-08T21:23:16.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f245d5d3700 1 --2- 172.21.15.3:0/1285400895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2458161030 0x7f245814f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:16.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.519+0000 7f245cdd2700 1 --2- 172.21.15.3:0/1285400895 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24580b2480 0x7f2458155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:16.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.520+0000 7f2457fff700 1 --2- 172.21.15.3:0/1285400895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24580ccbd0 0x7f24581569b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f244c00c4d0 tx=0x7f244c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:16.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.520+0000 7f24557fa700 1 -- 172.21.15.3:0/1285400895 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c0052d0 con 0x7f24580ccbd0 2024-05-08T21:23:16.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.520+0000 7f24557fa700 1 -- 172.21.15.3:0/1285400895 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f244c005430 con 0x7f24580ccbd0 2024-05-08T21:23:16.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.520+0000 7f245f036700 1 -- 172.21.15.3:0/1285400895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2458155910 con 0x7f24580ccbd0 2024-05-08T21:23:16.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.520+0000 7f24557fa700 1 -- 172.21.15.3:0/1285400895 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c003880 con 0x7f24580ccbd0 2024-05-08T21:23:16.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.520+0000 7f245f036700 1 -- 172.21.15.3:0/1285400895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2458155ed0 con 0x7f24580ccbd0 2024-05-08T21:23:16.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.521+0000 7f245f036700 1 -- 172.21.15.3:0/1285400895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2458068000 con 0x7f24580ccbd0 2024-05-08T21:23:16.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.521+0000 7f24557fa700 1 -- 172.21.15.3:0/1285400895 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f244c0039e0 con 0x7f24580ccbd0 2024-05-08T21:23:16.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.522+0000 7f24557fa700 1 --2- 172.21.15.3:0/1285400895 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f244406ac90 0x7f244406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:16.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.522+0000 7f24557fa700 1 -- 172.21.15.3:0/1285400895 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f244c088770 con 0x7f24580ccbd0 2024-05-08T21:23:16.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.522+0000 7f245cdd2700 1 --2- 172.21.15.3:0/1285400895 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f244406ac90 0x7f244406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:16.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.522+0000 7f245cdd2700 1 --2- 172.21.15.3:0/1285400895 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f244406ac90 0x7f244406d150 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2448000f60 tx=0x7f2448007ce0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:16.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.527+0000 7f24557fa700 1 -- 172.21.15.3:0/1285400895 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f244c053870 con 0x7f24580ccbd0 2024-05-08T21:23:16.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.797+0000 7f245f036700 1 -- 172.21.15.3:0/1285400895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24580c8270 con 0x7f24580ccbd0 2024-05-08T21:23:16.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.797+0000 7f24557fa700 1 -- 172.21.15.3:0/1285400895 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5936 (secure 0 0 0) 0x7f244c056e90 con 0x7f24580ccbd0 2024-05-08T21:23:16.800 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:16.800 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6833/2104267343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2104267343},{"type":"v1","addr":"172.21.15.3:6833","nonce":2104267343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/208321708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":208321708},{"type":"v1","addr":"172.21.15.3:6835","nonce":208321708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/1591386131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1591386131},{"type":"v1","addr":"172.21.15.186:6836","nonce":1591386131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:22:50.184245+0000","modified":"2024-05-08T21:23:01.563810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:23:05.475355+0000","modified":"2024-05-08T21:23:15.662101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6837/2000072997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":2000072997},{"type":"v1","addr":"172.21.15.186:6837","nonce":2000072997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":6,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-08T21:23:16.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.800+0000 7f245f036700 1 -- 172.21.15.3:0/1285400895 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f244406ac90 msgr2=0x7f244406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:16.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.800+0000 7f245f036700 1 --2- 172.21.15.3:0/1285400895 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f244406ac90 0x7f244406d150 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2448000f60 tx=0x7f2448007ce0 comp rx=0 tx=0).stop 2024-05-08T21:23:16.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.800+0000 7f245f036700 1 -- 172.21.15.3:0/1285400895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24580ccbd0 msgr2=0x7f24581569b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:16.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.800+0000 7f245f036700 1 --2- 172.21.15.3:0/1285400895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24580ccbd0 0x7f24581569b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f244c00c4d0 tx=0x7f244c00c890 comp rx=0 tx=0).stop 2024-05-08T21:23:16.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.800+0000 7f245f036700 1 -- 172.21.15.3:0/1285400895 shutdown_connections 2024-05-08T21:23:16.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.800+0000 7f245f036700 1 --2- 172.21.15.3:0/1285400895 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24580b2480 0x7f2458155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:16.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.800+0000 7f245f036700 1 --2- 172.21.15.3:0/1285400895 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f244406ac90 0x7f244406d150 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:16.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.800+0000 7f245f036700 1 --2- 172.21.15.3:0/1285400895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2458161030 0x7f245814f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:16.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.800+0000 7f245f036700 1 --2- 172.21.15.3:0/1285400895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24580ccbd0 0x7f24581569b0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:16.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.800+0000 7f245f036700 1 -- 172.21.15.3:0/1285400895 >> 172.21.15.3:0/1285400895 conn(0x7f24580b2f20 msgr2=0x7f245806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:16.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.800+0000 7f245f036700 1 -- 172.21.15.3:0/1285400895 shutdown_connections 2024-05-08T21:23:16.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:16.800+0000 7f245f036700 1 -- 172.21.15.3:0/1285400895 wait complete. 2024-05-08T21:23:16.805 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 26 2024-05-08T21:23:17.218 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-08T21:23:05.475355+0000', 'modified': '2024-05-08T21:23:15.662101+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5191}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5191': {'gid': 5191, 'name': 'd', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.186:6837/2000072997', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6835', 'nonce': 2000072997}, {'type': 'v1', 'addr': '172.21.15.186:6837', 'nonce': 2000072997}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [5], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'recovery_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-05-08T21:23:17.219 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-08T21:23:17.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-05-08T21:23:17.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.365+0000 7fcb2fce8700 1 Processor -- start 2024-05-08T21:23:17.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.365+0000 7fcb2fce8700 1 -- start start 2024-05-08T21:23:17.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.366+0000 7fcb2fce8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcb2815b900 0x7fcb280cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.366+0000 7fcb2fce8700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcb28099a50 0x7fcb28099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.366+0000 7fcb2fce8700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb2809a8d0 con 0x7fcb2809a440 2024-05-08T21:23:17.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.366+0000 7fcb2fce8700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb280b1410 con 0x7fcb2815b900 2024-05-08T21:23:17.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.366+0000 7fcb2fce8700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb280b0e30 con 0x7fcb28099a50 2024-05-08T21:23:17.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.366+0000 7fcb2da84700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcb2815b900 0x7fcb280cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.366+0000 7fcb2da84700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcb2815b900 0x7fcb280cf9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57624/0 (socket says 172.21.15.3:57624) 2024-05-08T21:23:17.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.366+0000 7fcb2da84700 1 -- 172.21.15.3:0/1340496227 learned_addr learned my addr 172.21.15.3:0/1340496227 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:17.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.367+0000 7fcb2d283700 1 --2- 172.21.15.3:0/1340496227 >> v2:172.21.15.186:3300/0 conn(0x7fcb28099a50 0x7fcb28099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.367+0000 7fcb2da84700 1 -- 172.21.15.3:0/1340496227 >> v2:172.21.15.186:3300/0 conn(0x7fcb28099a50 msgr2=0x7fcb28099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.367+0000 7fcb2da84700 1 --2- 172.21.15.3:0/1340496227 >> v2:172.21.15.186:3300/0 conn(0x7fcb28099a50 0x7fcb28099f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.367+0000 7fcb2da84700 1 -- 172.21.15.3:0/1340496227 >> v1:172.21.15.186:6789/0 conn(0x7fcb2809a440 legacy=0x7fcb280cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.367+0000 7fcb2da84700 1 -- 172.21.15.3:0/1340496227 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb280cffd0 con 0x7fcb2815b900 2024-05-08T21:23:17.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.367+0000 7fcb2da84700 1 --2- 172.21.15.3:0/1340496227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb2815b900 0x7fcb280cf9e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fcb280d0a70 tx=0x7fcb24009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed088185dec0c802 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:17.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.367+0000 7fcb2ca82700 1 -- 172.21.15.3:0/1340496227 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb24004230 con 0x7fcb2815b900 2024-05-08T21:23:17.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.367+0000 7fcb2ca82700 1 -- 172.21.15.3:0/1340496227 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb24004390 con 0x7fcb2815b900 2024-05-08T21:23:17.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.367+0000 7fcb2ca82700 1 -- 172.21.15.3:0/1340496227 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb240036a0 con 0x7fcb2815b900 2024-05-08T21:23:17.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.368+0000 7fcb2fce8700 1 -- 172.21.15.3:0/1340496227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb2815b900 msgr2=0x7fcb280cf9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.368+0000 7fcb2fce8700 1 --2- 172.21.15.3:0/1340496227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb2815b900 0x7fcb280cf9e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fcb280d0a70 tx=0x7fcb24009bd0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.368+0000 7fcb2fce8700 1 -- 172.21.15.3:0/1340496227 shutdown_connections 2024-05-08T21:23:17.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.368+0000 7fcb2fce8700 1 --2- 172.21.15.3:0/1340496227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb2815b900 0x7fcb280cf9e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.368+0000 7fcb2fce8700 1 --2- 172.21.15.3:0/1340496227 >> v2:172.21.15.186:3300/0 conn(0x7fcb28099a50 0x7fcb28099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.368+0000 7fcb2fce8700 1 -- 172.21.15.3:0/1340496227 >> 172.21.15.3:0/1340496227 conn(0x7fcb280b2f20 msgr2=0x7fcb280b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:17.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.368+0000 7fcb2fce8700 1 -- 172.21.15.3:0/1340496227 shutdown_connections 2024-05-08T21:23:17.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.368+0000 7fcb2fce8700 1 -- 172.21.15.3:0/1340496227 wait complete. 2024-05-08T21:23:17.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.369+0000 7fcb2fce8700 1 Processor -- start 2024-05-08T21:23:17.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.369+0000 7fcb2fce8700 1 -- start start 2024-05-08T21:23:17.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2fce8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb28099a50 0x7fcb280c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2fce8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb2809a440 0x7fcb280b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2fce8700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb2815b900 0x7fcb280cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2fce8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb280b59b0 con 0x7fcb2809a440 2024-05-08T21:23:17.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2fce8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb280cad10 con 0x7fcb28099a50 2024-05-08T21:23:17.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2fce8700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb280b53f0 con 0x7fcb2815b900 2024-05-08T21:23:17.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2d283700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb2809a440 0x7fcb280b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2d283700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb2809a440 0x7fcb280b4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57628/0 (socket says 172.21.15.3:57628) 2024-05-08T21:23:17.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2d283700 1 -- 172.21.15.3:0/412108250 learned_addr learned my addr 172.21.15.3:0/412108250 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:17.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2da84700 1 --2- 172.21.15.3:0/412108250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb28099a50 0x7fcb280c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2e285700 1 --2- 172.21.15.3:0/412108250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb2815b900 0x7fcb280cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2d283700 1 -- 172.21.15.3:0/412108250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb2815b900 msgr2=0x7fcb280cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2d283700 1 --2- 172.21.15.3:0/412108250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb2815b900 0x7fcb280cb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2d283700 1 -- 172.21.15.3:0/412108250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb28099a50 msgr2=0x7fcb280c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2d283700 1 --2- 172.21.15.3:0/412108250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb28099a50 0x7fcb280c9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2d283700 1 -- 172.21.15.3:0/412108250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb24009550 con 0x7fcb2809a440 2024-05-08T21:23:17.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2e285700 1 --2- 172.21.15.3:0/412108250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb2815b900 0x7fcb280cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:17.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.370+0000 7fcb2d283700 1 --2- 172.21.15.3:0/412108250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb2809a440 0x7fcb280b4cf0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fcb1800c7f0 tx=0x7fcb1800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:17.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.371+0000 7fcb1e7fc700 1 -- 172.21.15.3:0/412108250 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb18011800 con 0x7fcb2809a440 2024-05-08T21:23:17.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.371+0000 7fcb1e7fc700 1 -- 172.21.15.3:0/412108250 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb18011960 con 0x7fcb2809a440 2024-05-08T21:23:17.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.371+0000 7fcb2fce8700 1 -- 172.21.15.3:0/412108250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb280b56d0 con 0x7fcb2809a440 2024-05-08T21:23:17.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.371+0000 7fcb1e7fc700 1 -- 172.21.15.3:0/412108250 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb18011cd0 con 0x7fcb2809a440 2024-05-08T21:23:17.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.371+0000 7fcb2fce8700 1 -- 172.21.15.3:0/412108250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb280cb6e0 con 0x7fcb2809a440 2024-05-08T21:23:17.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.372+0000 7fcb1e7fc700 1 -- 172.21.15.3:0/412108250 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcb18010350 con 0x7fcb2809a440 2024-05-08T21:23:17.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.373+0000 7fcb1e7fc700 1 --2- 172.21.15.3:0/412108250 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb0c06ac90 0x7fcb0c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.373+0000 7fcb1e7fc700 1 -- 172.21.15.3:0/412108250 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7fcb18089080 con 0x7fcb2809a440 2024-05-08T21:23:17.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.373+0000 7fcb2da84700 1 --2- 172.21.15.3:0/412108250 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb0c06ac90 0x7fcb0c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.374+0000 7fcb2da84700 1 --2- 172.21.15.3:0/412108250 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb0c06ac90 0x7fcb0c06d150 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcb24009130 tx=0x7fcb24008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:17.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.375+0000 7fcb3bea2b80 1 Processor -- start 2024-05-08T21:23:17.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.375+0000 7fcb3bea2b80 1 -- start start 2024-05-08T21:23:17.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.376+0000 7fcb3bea2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e2326580c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.376+0000 7fcb3bea2b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23265a320 0x55e23265d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.376+0000 7fcb3bea2b80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e232686d80 0x55e232687200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.376+0000 7fcb3bea2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2325b3100 con 0x55e23267f750 2024-05-08T21:23:17.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.376+0000 7fcb3bea2b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2325b0ac0 con 0x55e23265a320 2024-05-08T21:23:17.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.376+0000 7fcb3bea2b80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e2325ac3b0 con 0x55e232686d80 2024-05-08T21:23:17.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.376+0000 7fcb2da84700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e2326580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.376+0000 7fcb2da84700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e2326580c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57632/0 (socket says 172.21.15.3:57632) 2024-05-08T21:23:17.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.376+0000 7fcb2da84700 1 -- 172.21.15.3:0/335898555 learned_addr learned my addr 172.21.15.3:0/335898555 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:17.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.376+0000 7fcb2d283700 1 --2- 172.21.15.3:0/335898555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23265a320 0x55e23265d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.376+0000 7fcb2e285700 1 --2- 172.21.15.3:0/335898555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e232686d80 0x55e232687200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.376+0000 7fcb2da84700 1 -- 172.21.15.3:0/335898555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e232686d80 msgr2=0x55e232687200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.376+0000 7fcb2da84700 1 --2- 172.21.15.3:0/335898555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e232686d80 0x55e232687200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.376+0000 7fcb2da84700 1 -- 172.21.15.3:0/335898555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23265a320 msgr2=0x55e23265d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.376+0000 7fcb2da84700 1 --2- 172.21.15.3:0/335898555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23265a320 0x55e23265d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.376+0000 7fcb2da84700 1 -- 172.21.15.3:0/335898555 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e23242d360 con 0x55e23267f750 2024-05-08T21:23:17.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.377+0000 7fcb2da84700 1 --2- 172.21.15.3:0/335898555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e2326580c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fcb28072530 tx=0x7fcb24010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5bd8705e50ee5e server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:17.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.377+0000 7fcb0b7fe700 1 -- 172.21.15.3:0/335898555 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb24012840 con 0x55e23267f750 2024-05-08T21:23:17.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.377+0000 7fcb0b7fe700 1 -- 172.21.15.3:0/335898555 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb240129a0 con 0x55e23267f750 2024-05-08T21:23:17.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.377+0000 7fcb0b7fe700 1 -- 172.21.15.3:0/335898555 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb24012d50 con 0x55e23267f750 2024-05-08T21:23:17.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.377+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/335898555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 msgr2=0x55e2326580c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.377+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/335898555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e2326580c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fcb28072530 tx=0x7fcb24010000 comp rx=0 tx=0).stop 2024-05-08T21:23:17.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.378+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/335898555 shutdown_connections 2024-05-08T21:23:17.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.378+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/335898555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e232686d80 0x55e232687200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.378+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/335898555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e2326580c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.378+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/335898555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23265a320 0x55e23265d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.378+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/335898555 >> 172.21.15.3:0/335898555 conn(0x55e2325ba1c0 msgr2=0x55e2326a1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:17.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.378+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/335898555 shutdown_connections 2024-05-08T21:23:17.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.378+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/335898555 wait complete. 2024-05-08T21:23:17.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.378+0000 7fcb3bea2b80 1 -- start start 2024-05-08T21:23:17.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.379+0000 7fcb3bea2b80 1 Processor -- start 2024-05-08T21:23:17.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.380+0000 7fcb3bea2b80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e23265a320 0x55e232683e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.380+0000 7fcb3bea2b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23267f750 0x55e23268a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.380+0000 7fcb3bea2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e232686d80 0x55e2326a2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.380+0000 7fcb3bea2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e232548510 con 0x55e232686d80 2024-05-08T21:23:17.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.380+0000 7fcb3bea2b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2325481b0 con 0x55e23267f750 2024-05-08T21:23:17.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.380+0000 7fcb3bea2b80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e23242f650 con 0x55e23265a320 2024-05-08T21:23:17.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.380+0000 7fcb3bea2b80 10 client.0 _ll_register_callbacks cb 0x55e232551ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-08T21:23:17.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.380+0000 7fcb2e285700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e232686d80 0x55e2326a2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.380+0000 7fcb2e285700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e232686d80 0x55e2326a2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57640/0 (socket says 172.21.15.3:57640) 2024-05-08T21:23:17.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.380+0000 7fcb2e285700 1 -- 172.21.15.3:0/638117420 learned_addr learned my addr 172.21.15.3:0/638117420 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:17.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.380+0000 7fcb2da84700 1 --2- 172.21.15.3:0/638117420 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e23265a320 0x55e232683e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.380+0000 7fcb2d283700 1 --2- 172.21.15.3:0/638117420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23267f750 0x55e23268a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb2e285700 1 -- 172.21.15.3:0/638117420 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e23265a320 msgr2=0x55e232683e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb2e285700 1 --2- 172.21.15.3:0/638117420 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e23265a320 0x55e232683e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb2e285700 1 -- 172.21.15.3:0/638117420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23267f750 msgr2=0x55e23268a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb2e285700 1 --2- 172.21.15.3:0/638117420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23267f750 0x55e23268a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb2e285700 1 -- 172.21.15.3:0/638117420 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb280cffd0 con 0x55e232686d80 2024-05-08T21:23:17.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb2d283700 1 --2- 172.21.15.3:0/638117420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23267f750 0x55e23268a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:17.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb2e285700 1 --2- 172.21.15.3:0/638117420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e232686d80 0x55e2326a2090 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fcb2000ac30 tx=0x7fcb2000ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c671933e7b30e79c server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:17.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb08ff9700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-05-08T21:23:17.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb08ff9700 1 -- 172.21.15.3:0/638117420 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2000aea0 con 0x55e232686d80 2024-05-08T21:23:17.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb08ff9700 1 -- 172.21.15.3:0/638117420 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7fcb20007b50 con 0x55e232686d80 2024-05-08T21:23:17.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/638117420 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e232505220 con 0x55e232686d80 2024-05-08T21:23:17.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb08ff9700 1 -- 172.21.15.3:0/638117420 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb20009b20 con 0x55e232686d80 2024-05-08T21:23:17.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb3bea2b80 10 client.5532 fetch_fsmap learned FSMap version 26 2024-05-08T21:23:17.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/638117420 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55e2323d9b10 con 0x55e232686d80 2024-05-08T21:23:17.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb08ff9700 1 -- 172.21.15.3:0/638117420 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e232505220 con 0x55e232686d80 2024-05-08T21:23:17.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb08ff9700 1 -- 172.21.15.3:0/638117420 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 26) v1 ==== 3687+0+0 (secure 0 0 0) 0x7fcb20007d10 con 0x55e232686d80 2024-05-08T21:23:17.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb3bea2b80 10 client.5532 fetch_fsmap finished waiting for FSMap version 26 2024-05-08T21:23:17.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb3bea2b80 10 client.5532 resolve_mds: resolved name 'd' to daemon mds.d 2024-05-08T21:23:17.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb3bea2b80 20 client.5532 populate_metadata read hostname 'smithi003' 2024-05-08T21:23:17.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/638117420 >> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] conn(0x55e2326a4f00 0x55e2326b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb3bea2b80 4 client.5532 mds_command: new command op to 5191 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-08T21:23:17.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.381+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/638117420 --> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e23262b790 con 0x55e2326a4f00 2024-05-08T21:23:17.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.382+0000 7fcb2da84700 1 --2- 172.21.15.3:0/638117420 >> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] conn(0x55e2326a4f00 0x55e2326b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.382+0000 7fcb08ff9700 10 client.5532 ms_handle_connect on v2:172.21.15.186:6835/2000072997 2024-05-08T21:23:17.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.382+0000 7fcb2da84700 1 --2- 172.21.15.3:0/638117420 >> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] conn(0x55e2326a4f00 0x55e2326b4d30 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=479d62313cbec538 server_cookie=12b628af1f79a46e in_seq=0 out_seq=0 2024-05-08T21:23:17.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.383+0000 7fcb08ff9700 1 -- 172.21.15.3:0/638117420 <== mds.0 v2:172.21.15.186:6835/2000072997 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e23262b790 con 0x55e2326a4f00 2024-05-08T21:23:17.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.383+0000 7fcb08ff9700 10 client.5532 handle_command_reply: tid=0 2024-05-08T21:23:17.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.383+0000 7fcb3bea2b80 1 client.5532 shutdown 2024-05-08T21:23:17.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.383+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/638117420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e232686d80 msgr2=0x55e2326a2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.383+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/638117420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e232686d80 0x55e2326a2090 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fcb2000ac30 tx=0x7fcb2000ac60 comp rx=0 tx=0).stop 2024-05-08T21:23:17.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.383+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/638117420 shutdown_connections 2024-05-08T21:23:17.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.383+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/638117420 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e23265a320 0x55e232683e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.383+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/638117420 >> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] conn(0x55e2326a4f00 0x55e2326b4d30 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.383+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/638117420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e232686d80 0x55e2326a2090 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.383+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/638117420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23267f750 0x55e23268a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.383+0000 7fcb08ff9700 0 client.5532 ms_handle_reset on v2:172.21.15.186:6835/2000072997 2024-05-08T21:23:17.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.383+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/638117420 >> 172.21.15.3:0/638117420 conn(0x55e2325ba1c0 msgr2=0x55e23265c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:17.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.383+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/638117420 shutdown_connections 2024-05-08T21:23:17.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.383+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/638117420 wait complete. 2024-05-08T21:23:17.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.383+0000 7fcb3bea2b80 20 client.5532 trim_cache size 0 max 16384 2024-05-08T21:23:17.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.403+0000 7fcb3bea2b80 1 Processor -- start 2024-05-08T21:23:17.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.403+0000 7fcb3bea2b80 1 -- start start 2024-05-08T21:23:17.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb3bea2b80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e23265a320 0x55e23265c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb3bea2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e232687770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb3bea2b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e232686d80 0x55e2326a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb3bea2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2326c42e0 con 0x55e23267f750 2024-05-08T21:23:17.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb3bea2b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2326a70c0 con 0x55e232686d80 2024-05-08T21:23:17.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb3bea2b80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e23265bf10 con 0x55e23265a320 2024-05-08T21:23:17.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb2d283700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e232687770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb2d283700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e232687770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57646/0 (socket says 172.21.15.3:57646) 2024-05-08T21:23:17.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb2d283700 1 -- 172.21.15.3:0/485659228 learned_addr learned my addr 172.21.15.3:0/485659228 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:17.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb2da84700 1 --2- 172.21.15.3:0/485659228 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e23265a320 0x55e23265c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb2e285700 1 --2- 172.21.15.3:0/485659228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e232686d80 0x55e2326a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb2d283700 1 -- 172.21.15.3:0/485659228 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e23265a320 msgr2=0x55e23265c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb2d283700 1 --2- 172.21.15.3:0/485659228 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e23265a320 0x55e23265c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb2d283700 1 -- 172.21.15.3:0/485659228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e232686d80 msgr2=0x55e2326a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb2d283700 1 --2- 172.21.15.3:0/485659228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e232686d80 0x55e2326a0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb2d283700 1 -- 172.21.15.3:0/485659228 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e232572510 con 0x55e23267f750 2024-05-08T21:23:17.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.404+0000 7fcb2d283700 1 --2- 172.21.15.3:0/485659228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e232687770 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fcb1805fbd0 tx=0x7fcb18057740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c0989c85c6e4d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:17.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.405+0000 7fcb0bfff700 1 -- 172.21.15.3:0/485659228 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1805bb40 con 0x55e23267f750 2024-05-08T21:23:17.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.405+0000 7fcb0bfff700 1 -- 172.21.15.3:0/485659228 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb18060100 con 0x55e23267f750 2024-05-08T21:23:17.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.405+0000 7fcb0bfff700 1 -- 172.21.15.3:0/485659228 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb18057200 con 0x55e23267f750 2024-05-08T21:23:17.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.405+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/485659228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 msgr2=0x55e232687770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.405+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/485659228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e232687770 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fcb1805fbd0 tx=0x7fcb18057740 comp rx=0 tx=0).stop 2024-05-08T21:23:17.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.405+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/485659228 shutdown_connections 2024-05-08T21:23:17.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.405+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/485659228 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e23265a320 0x55e23265c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.405+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/485659228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e232686d80 0x55e2326a0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.405+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/485659228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e232687770 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.405+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/485659228 >> 172.21.15.3:0/485659228 conn(0x55e2325ba1c0 msgr2=0x55e232653190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:17.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.405+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/485659228 shutdown_connections 2024-05-08T21:23:17.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.405+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/485659228 wait complete. 2024-05-08T21:23:17.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.406+0000 7fcb3bea2b80 1 -- start start 2024-05-08T21:23:17.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.406+0000 7fcb3bea2b80 1 Processor -- start 2024-05-08T21:23:17.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.407+0000 7fcb3bea2b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23265a320 0x55e2326a3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb3bea2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e23265b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb3bea2b80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e232686d80 0x55e23265b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb3bea2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2326a6a20 con 0x55e23267f750 2024-05-08T21:23:17.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb3bea2b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e23265bb90 con 0x55e23265a320 2024-05-08T21:23:17.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb3bea2b80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e2327100b0 con 0x55e232686d80 2024-05-08T21:23:17.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb3bea2b80 10 client.0 _ll_register_callbacks cb 0x55e232551ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-08T21:23:17.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb2d283700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e23265b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb2d283700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e23265b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57650/0 (socket says 172.21.15.3:57650) 2024-05-08T21:23:17.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb2d283700 1 -- 172.21.15.3:0/1346605219 learned_addr learned my addr 172.21.15.3:0/1346605219 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:17.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb2da84700 1 --2- 172.21.15.3:0/1346605219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23265a320 0x55e2326a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb2e285700 1 --2- 172.21.15.3:0/1346605219 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e232686d80 0x55e23265b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb2d283700 1 -- 172.21.15.3:0/1346605219 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e232686d80 msgr2=0x55e23265b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb2d283700 1 --2- 172.21.15.3:0/1346605219 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e232686d80 0x55e23265b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb2d283700 1 -- 172.21.15.3:0/1346605219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23265a320 msgr2=0x55e2326a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb2d283700 1 --2- 172.21.15.3:0/1346605219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23265a320 0x55e2326a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb2d283700 1 -- 172.21.15.3:0/1346605219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb280cb6e0 con 0x55e23267f750 2024-05-08T21:23:17.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb2da84700 1 --2- 172.21.15.3:0/1346605219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23265a320 0x55e2326a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:17.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb2d283700 1 --2- 172.21.15.3:0/1346605219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e23265b110 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcb18058370 tx=0x7fcb1800af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4e374506e483ba1 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:17.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb08ff9700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-05-08T21:23:17.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb08ff9700 1 -- 172.21.15.3:0/1346605219 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1805fe70 con 0x55e23267f750 2024-05-08T21:23:17.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb08ff9700 1 -- 172.21.15.3:0/1346605219 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7fcb1805bb40 con 0x55e23267f750 2024-05-08T21:23:17.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/1346605219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcb240129a0 con 0x55e23267f750 2024-05-08T21:23:17.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb08ff9700 1 -- 172.21.15.3:0/1346605219 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb18087180 con 0x55e23267f750 2024-05-08T21:23:17.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.408+0000 7fcb3bea2b80 10 client.5538 fetch_fsmap learned FSMap version 26 2024-05-08T21:23:17.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.409+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/1346605219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55e232431c80 con 0x55e23267f750 2024-05-08T21:23:17.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.409+0000 7fcb08ff9700 1 -- 172.21.15.3:0/1346605219 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb240129a0 con 0x55e23267f750 2024-05-08T21:23:17.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.409+0000 7fcb08ff9700 1 -- 172.21.15.3:0/1346605219 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 26) v1 ==== 3687+0+0 (secure 0 0 0) 0x7fcb18058d20 con 0x55e23267f750 2024-05-08T21:23:17.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.409+0000 7fcb3bea2b80 10 client.5538 fetch_fsmap finished waiting for FSMap version 26 2024-05-08T21:23:17.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.409+0000 7fcb3bea2b80 10 client.5538 resolve_mds: resolved name 'd' to daemon mds.d 2024-05-08T21:23:17.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.409+0000 7fcb3bea2b80 20 client.5538 populate_metadata read hostname 'smithi003' 2024-05-08T21:23:17.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.409+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/1346605219 >> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] conn(0x55e2327053b0 0x55e232705790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.409+0000 7fcb3bea2b80 4 client.5538 mds_command: new command op to 5191 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-08T21:23:17.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.409+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/1346605219 --> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e2325a4e70 con 0x55e2327053b0 2024-05-08T21:23:17.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.409+0000 7fcb2da84700 1 --2- 172.21.15.3:0/1346605219 >> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] conn(0x55e2327053b0 0x55e232705790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.409+0000 7fcb08ff9700 10 client.5538 ms_handle_connect on v2:172.21.15.186:6835/2000072997 2024-05-08T21:23:17.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.409+0000 7fcb2da84700 1 --2- 172.21.15.3:0/1346605219 >> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] conn(0x55e2327053b0 0x55e232705790 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a97469168d6f9f03 server_cookie=b3565ef489915552 in_seq=0 out_seq=0 2024-05-08T21:23:17.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.410+0000 7fcb08ff9700 1 -- 172.21.15.3:0/1346605219 <== mds.0 v2:172.21.15.186:6835/2000072997 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55e2325a4e70 con 0x55e2327053b0 2024-05-08T21:23:17.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.410+0000 7fcb08ff9700 10 client.5538 handle_command_reply: tid=0 2024-05-08T21:23:17.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.410+0000 7fcb3bea2b80 1 client.5538 shutdown 2024-05-08T21:23:17.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.410+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/1346605219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 msgr2=0x55e23265b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.410+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/1346605219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e23265b110 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcb18058370 tx=0x7fcb1800af90 comp rx=0 tx=0).stop 2024-05-08T21:23:17.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.410+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/1346605219 shutdown_connections 2024-05-08T21:23:17.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.410+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/1346605219 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55e232686d80 0x55e23265b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.410+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/1346605219 >> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] conn(0x55e2327053b0 0x55e232705790 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.410+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/1346605219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e23265a320 0x55e2326a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.410+0000 7fcb3bea2b80 1 --2- 172.21.15.3:0/1346605219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e23267f750 0x55e23265b110 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.410+0000 7fcb08ff9700 0 client.5538 ms_handle_reset on v2:172.21.15.186:6835/2000072997 2024-05-08T21:23:17.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.410+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/1346605219 >> 172.21.15.3:0/1346605219 conn(0x55e2325ba1c0 msgr2=0x55e23265c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:17.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.410+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/1346605219 shutdown_connections 2024-05-08T21:23:17.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.410+0000 7fcb3bea2b80 1 -- 172.21.15.3:0/1346605219 wait complete. 2024-05-08T21:23:17.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.410+0000 7fcb3bea2b80 20 client.5538 trim_cache size 0 max 16384 2024-05-08T21:23:17.422 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:23:17.422 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "d0f58275-758f-4470-b003-84d6abe25489", 2024-05-08T21:23:17.422 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 0, 2024-05-08T21:23:17.422 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 5191, 2024-05-08T21:23:17.422 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2024-05-08T21:23:17.422 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2024-05-08T21:23:17.422 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "recovery_fs", 2024-05-08T21:23:17.422 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 4.751588173, 2024-05-08T21:23:17.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 26, 2024-05-08T21:23:17.422 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 42, 2024-05-08T21:23:17.422 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 42, 2024-05-08T21:23:17.423 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 15.48402405 2024-05-08T21:23:17.423 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:23:17.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.411+0000 7fcb2fce8700 1 -- 172.21.15.3:0/412108250 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb0c06ac90 msgr2=0x7fcb0c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.411+0000 7fcb2fce8700 1 --2- 172.21.15.3:0/412108250 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb0c06ac90 0x7fcb0c06d150 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcb24009130 tx=0x7fcb24008030 comp rx=0 tx=0).stop 2024-05-08T21:23:17.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.411+0000 7fcb2fce8700 1 -- 172.21.15.3:0/412108250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb2809a440 msgr2=0x7fcb280b4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.411+0000 7fcb2fce8700 1 --2- 172.21.15.3:0/412108250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb2809a440 0x7fcb280b4cf0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fcb1800c7f0 tx=0x7fcb1800cb00 comp rx=0 tx=0).stop 2024-05-08T21:23:17.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.411+0000 7fcb2fce8700 1 -- 172.21.15.3:0/412108250 shutdown_connections 2024-05-08T21:23:17.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.411+0000 7fcb2fce8700 1 --2- 172.21.15.3:0/412108250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb2815b900 0x7fcb280cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.411+0000 7fcb2fce8700 1 --2- 172.21.15.3:0/412108250 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb0c06ac90 0x7fcb0c06d150 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.411+0000 7fcb2fce8700 1 --2- 172.21.15.3:0/412108250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb28099a50 0x7fcb280c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.411+0000 7fcb2fce8700 1 --2- 172.21.15.3:0/412108250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb2809a440 0x7fcb280b4cf0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.411+0000 7fcb2fce8700 1 -- 172.21.15.3:0/412108250 >> 172.21.15.3:0/412108250 conn(0x7fcb280b2f20 msgr2=0x7fcb280cbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:17.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.411+0000 7fcb2fce8700 1 -- 172.21.15.3:0/412108250 shutdown_connections 2024-05-08T21:23:17.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.411+0000 7fcb2fce8700 1 -- 172.21.15.3:0/412108250 wait complete. 2024-05-08T21:23:17.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds debug_mds 20 2024-05-08T21:23:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.564+0000 7f48d71a0700 1 Processor -- start 2024-05-08T21:23:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.564+0000 7f48d71a0700 1 -- start start 2024-05-08T21:23:17.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.565+0000 7f48d71a0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d00cdd40 0x7f48d0161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.565+0000 7f48d71a0700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f48d00d0e80 0x7f48d00ce120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.565+0000 7f48d71a0700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d00d1f30 con 0x7f48d015dc40 2024-05-08T21:23:17.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.565+0000 7f48d71a0700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d00cfab0 con 0x7f48d00d0e80 2024-05-08T21:23:17.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.565+0000 7f48d71a0700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d0162120 con 0x7f48d00cdd40 2024-05-08T21:23:17.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.566+0000 7f48d619e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d00cdd40 0x7f48d0161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.566+0000 7f48d599d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f48d00d0e80 0x7f48d00ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.566+0000 7f48d619e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d00cdd40 0x7f48d0161be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.3:56426/0 (socket says 172.21.15.3:56426) 2024-05-08T21:23:17.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.566+0000 7f48d619e700 1 -- 172.21.15.3:0/753004926 learned_addr learned my addr 172.21.15.3:0/753004926 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:17.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.566+0000 7f48d619e700 1 -- 172.21.15.3:0/753004926 >> v2:172.21.15.186:3300/0 conn(0x7f48d00d0e80 msgr2=0x7f48d00ce120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.566+0000 7f48d619e700 1 --2- 172.21.15.3:0/753004926 >> v2:172.21.15.186:3300/0 conn(0x7f48d00d0e80 0x7f48d00ce120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.566+0000 7f48d619e700 1 -- 172.21.15.3:0/753004926 >> v1:172.21.15.186:6789/0 conn(0x7f48d015dc40 legacy=0x7f48d015e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.566+0000 7f48d619e700 1 -- 172.21.15.3:0/753004926 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d00ce660 con 0x7f48d00cdd40 2024-05-08T21:23:17.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.566+0000 7f48d619e700 1 --2- 172.21.15.3:0/753004926 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d00cdd40 0x7f48d0161be0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f48cc0098f0 tx=0x7f48cc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40abf844b1c21a3c server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:17.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.567+0000 7f48d519c700 1 -- 172.21.15.3:0/753004926 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc011830 con 0x7f48d00cdd40 2024-05-08T21:23:17.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.567+0000 7f48d519c700 1 -- 172.21.15.3:0/753004926 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48cc011990 con 0x7f48d00cdd40 2024-05-08T21:23:17.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.567+0000 7f48d519c700 1 -- 172.21.15.3:0/753004926 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc011d20 con 0x7f48d00cdd40 2024-05-08T21:23:17.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.567+0000 7f48d71a0700 1 -- 172.21.15.3:0/753004926 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d00cdd40 msgr2=0x7f48d0161be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.567+0000 7f48d71a0700 1 --2- 172.21.15.3:0/753004926 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d00cdd40 0x7f48d0161be0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f48cc0098f0 tx=0x7f48cc009c00 comp rx=0 tx=0).stop 2024-05-08T21:23:17.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.567+0000 7f48d71a0700 1 -- 172.21.15.3:0/753004926 shutdown_connections 2024-05-08T21:23:17.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.567+0000 7f48d71a0700 1 --2- 172.21.15.3:0/753004926 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d00cdd40 0x7f48d0161be0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.567+0000 7f48d71a0700 1 --2- 172.21.15.3:0/753004926 >> v2:172.21.15.186:3300/0 conn(0x7f48d00d0e80 0x7f48d00ce120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.567+0000 7f48d71a0700 1 -- 172.21.15.3:0/753004926 >> 172.21.15.3:0/753004926 conn(0x7f48d006a960 msgr2=0x7f48d006ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:17.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.567+0000 7f48d71a0700 1 -- 172.21.15.3:0/753004926 shutdown_connections 2024-05-08T21:23:17.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.567+0000 7f48d71a0700 1 -- 172.21.15.3:0/753004926 wait complete. 2024-05-08T21:23:17.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.569+0000 7f48d71a0700 1 Processor -- start 2024-05-08T21:23:17.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.569+0000 7f48d71a0700 1 -- start start 2024-05-08T21:23:17.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.570+0000 7f48d71a0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d00cdd40 0x7f48d0156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.570+0000 7f48d71a0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48d00d0e80 0x7f48d00c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.570+0000 7f48d71a0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48d015dc40 0x7f48d00c7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.570+0000 7f48d71a0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d01e88d0 con 0x7f48d015dc40 2024-05-08T21:23:17.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.570+0000 7f48d71a0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d01e7a00 con 0x7f48d00d0e80 2024-05-08T21:23:17.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.570+0000 7f48d71a0700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d01e7c50 con 0x7f48d00cdd40 2024-05-08T21:23:17.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.570+0000 7f48d699f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48d015dc40 0x7f48d00c7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.570+0000 7f48d699f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48d015dc40 0x7f48d00c7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57654/0 (socket says 172.21.15.3:57654) 2024-05-08T21:23:17.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.570+0000 7f48d699f700 1 -- 172.21.15.3:0/4084101515 learned_addr learned my addr 172.21.15.3:0/4084101515 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.571+0000 7f48d619e700 1 --2- 172.21.15.3:0/4084101515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d00cdd40 0x7f48d0156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.571+0000 7f48d599d700 1 --2- 172.21.15.3:0/4084101515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48d00d0e80 0x7f48d00c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.571+0000 7f48d699f700 1 -- 172.21.15.3:0/4084101515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d00cdd40 msgr2=0x7f48d0156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.571+0000 7f48d699f700 1 --2- 172.21.15.3:0/4084101515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d00cdd40 0x7f48d0156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.571+0000 7f48d699f700 1 -- 172.21.15.3:0/4084101515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48d00d0e80 msgr2=0x7f48d00c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.571+0000 7f48d699f700 1 --2- 172.21.15.3:0/4084101515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48d00d0e80 0x7f48d00c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.571+0000 7f48d699f700 1 -- 172.21.15.3:0/4084101515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48cc009550 con 0x7f48d015dc40 2024-05-08T21:23:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.571+0000 7f48d699f700 1 --2- 172.21.15.3:0/4084101515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48d015dc40 0x7f48d00c7170 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f48c800cb50 tx=0x7f48c800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.571+0000 7f48c6ffd700 1 -- 172.21.15.3:0/4084101515 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c8003c00 con 0x7f48d015dc40 2024-05-08T21:23:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.571+0000 7f48c6ffd700 1 -- 172.21.15.3:0/4084101515 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c8003d60 con 0x7f48d015dc40 2024-05-08T21:23:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.571+0000 7f48c6ffd700 1 -- 172.21.15.3:0/4084101515 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c800aa20 con 0x7f48d015dc40 2024-05-08T21:23:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.571+0000 7f48d71a0700 1 -- 172.21.15.3:0/4084101515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d01e7f30 con 0x7f48d015dc40 2024-05-08T21:23:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.571+0000 7f48d71a0700 1 -- 172.21.15.3:0/4084101515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d01e84f0 con 0x7f48d015dc40 2024-05-08T21:23:17.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.572+0000 7f48d71a0700 1 -- 172.21.15.3:0/4084101515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d0074b50 con 0x7f48d015dc40 2024-05-08T21:23:17.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.573+0000 7f48c6ffd700 1 -- 172.21.15.3:0/4084101515 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f48c800ab80 con 0x7f48d015dc40 2024-05-08T21:23:17.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.573+0000 7f48c6ffd700 1 --2- 172.21.15.3:0/4084101515 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48b406ac90 0x7f48b406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.573+0000 7f48c6ffd700 1 -- 172.21.15.3:0/4084101515 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f48c80894f0 con 0x7f48d015dc40 2024-05-08T21:23:17.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.573+0000 7f48d619e700 1 --2- 172.21.15.3:0/4084101515 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48b406ac90 0x7f48b406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.574+0000 7f48d619e700 1 --2- 172.21.15.3:0/4084101515 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48b406ac90 0x7f48b406d150 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f48cc009c40 tx=0x7f48cc008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:17.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.578+0000 7f48c6ffd700 1 -- 172.21.15.3:0/4084101515 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f48c80544b0 con 0x7f48d015dc40 2024-05-08T21:23:17.617 INFO:tasks.ceph.mds.d.smithi186.stderr:2024-05-08T21:23:17.615+0000 7f6cfd9bd700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-08T21:23:17.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.797+0000 7f48d71a0700 1 -- 172.21.15.3:0/4084101515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) v1 -- 0x7f48d00b3800 con 0x7f48d015dc40 2024-05-08T21:23:17.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.801+0000 7f48c6ffd700 1 -- 172.21.15.3:0/4084101515 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v11)=0 v11) v1 ==== 108+0+0 (secure 0 0 0) 0x7f48c8057ad0 con 0x7f48d015dc40 2024-05-08T21:23:17.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.804+0000 7f48d71a0700 1 -- 172.21.15.3:0/4084101515 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48b406ac90 msgr2=0x7f48b406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.804+0000 7f48d71a0700 1 --2- 172.21.15.3:0/4084101515 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48b406ac90 0x7f48b406d150 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f48cc009c40 tx=0x7f48cc008030 comp rx=0 tx=0).stop 2024-05-08T21:23:17.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.804+0000 7f48d71a0700 1 -- 172.21.15.3:0/4084101515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48d015dc40 msgr2=0x7f48d00c7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.804+0000 7f48d71a0700 1 --2- 172.21.15.3:0/4084101515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48d015dc40 0x7f48d00c7170 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f48c800cb50 tx=0x7f48c800ce60 comp rx=0 tx=0).stop 2024-05-08T21:23:17.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.804+0000 7f48d71a0700 1 -- 172.21.15.3:0/4084101515 shutdown_connections 2024-05-08T21:23:17.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.804+0000 7f48d71a0700 1 --2- 172.21.15.3:0/4084101515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d00cdd40 0x7f48d0156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.804+0000 7f48d71a0700 1 --2- 172.21.15.3:0/4084101515 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f48b406ac90 0x7f48b406d150 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.804+0000 7f48d71a0700 1 --2- 172.21.15.3:0/4084101515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48d015dc40 0x7f48d00c7170 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.804+0000 7f48d71a0700 1 --2- 172.21.15.3:0/4084101515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48d00d0e80 0x7f48d00c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.804+0000 7f48d71a0700 1 -- 172.21.15.3:0/4084101515 >> 172.21.15.3:0/4084101515 conn(0x7f48d006a960 msgr2=0x7f48d00b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:17.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.804+0000 7f48d71a0700 1 -- 172.21.15.3:0/4084101515 shutdown_connections 2024-05-08T21:23:17.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.804+0000 7f48d71a0700 1 -- 172.21.15.3:0/4084101515 wait complete. 2024-05-08T21:23:17.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start / force,recursive,repair 2024-05-08T21:23:17.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.957+0000 7f76bc64e700 1 Processor -- start 2024-05-08T21:23:17.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.957+0000 7f76bc64e700 1 -- start start 2024-05-08T21:23:17.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.958+0000 7f76bc64e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76b4161320 0x7f76b40ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.958+0000 7f76bc64e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b40cf560 con 0x7f76b40d2230 2024-05-08T21:23:17.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.958+0000 7f76bc64e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b40ce950 con 0x7f76b40d1e50 2024-05-08T21:23:17.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.958+0000 7f76bc64e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b40cec30 con 0x7f76b4161320 2024-05-08T21:23:17.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.959+0000 7f76babeb700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f76b40d2230 0x7f76b406bad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51136/0 (socket says 172.21.15.3:51136) 2024-05-08T21:23:17.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.959+0000 7f76babeb700 1 -- 172.21.15.3:0/3696905940 learned_addr learned my addr 172.21.15.3:0/3696905940 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:17.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.959+0000 7f76b9be9700 1 --2- 172.21.15.3:0/3696905940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76b4161320 0x7f76b40ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.959+0000 7f76b93e8700 1 -- 172.21.15.3:0/3696905940 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858010356 0 0) 0x7f76b40cf560 con 0x7f76b40d2230 2024-05-08T21:23:17.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.959+0000 7f76b93e8700 1 -- 172.21.15.3:0/3696905940 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a00035b0 con 0x7f76b40d2230 2024-05-08T21:23:17.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.959+0000 7f76b93e8700 1 -- 172.21.15.3:0/3696905940 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577713725 0 0) 0x7f76b40ce950 con 0x7f76b40d1e50 2024-05-08T21:23:17.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.959+0000 7f76b93e8700 1 -- 172.21.15.3:0/3696905940 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b40cf560 con 0x7f76b40d1e50 2024-05-08T21:23:17.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.959+0000 7f76b93e8700 1 -- 172.21.15.3:0/3696905940 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1407881675 0 0) 0x7f76a00035b0 con 0x7f76b40d2230 2024-05-08T21:23:17.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.959+0000 7f76b9be9700 1 -- 172.21.15.3:0/3696905940 >> v1:172.21.15.186:6789/0 conn(0x7f76b40d1e50 legacy=0x7f76b4160c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.959+0000 7f76b9be9700 1 -- 172.21.15.3:0/3696905940 >> v1:172.21.15.3:6789/0 conn(0x7f76b40d2230 legacy=0x7f76b406bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.959+0000 7f76b9be9700 1 -- 172.21.15.3:0/3696905940 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b40cef10 con 0x7f76b4161320 2024-05-08T21:23:17.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.959+0000 7f76b9be9700 1 --2- 172.21.15.3:0/3696905940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76b4161320 0x7f76b40ce2b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f76a4009870 tx=0x7f76a4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1d30cc7068eddd4 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:17.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.960+0000 7f76b93e8700 1 -- 172.21.15.3:0/3696905940 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac0019f0 con 0x7f76b4161320 2024-05-08T21:23:17.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.960+0000 7f76b93e8700 1 -- 172.21.15.3:0/3696905940 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a4004030 con 0x7f76b4161320 2024-05-08T21:23:17.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.960+0000 7f76b93e8700 1 -- 172.21.15.3:0/3696905940 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a40043c0 con 0x7f76b4161320 2024-05-08T21:23:17.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.960+0000 7f76bc64e700 1 -- 172.21.15.3:0/3696905940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76b4161320 msgr2=0x7f76b40ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.960+0000 7f76bc64e700 1 --2- 172.21.15.3:0/3696905940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76b4161320 0x7f76b40ce2b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f76a4009870 tx=0x7f76a4009b80 comp rx=0 tx=0).stop 2024-05-08T21:23:17.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.960+0000 7f76bc64e700 1 -- 172.21.15.3:0/3696905940 shutdown_connections 2024-05-08T21:23:17.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.960+0000 7f76bc64e700 1 --2- 172.21.15.3:0/3696905940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76b4161320 0x7f76b40ce2b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.960+0000 7f76bc64e700 1 -- 172.21.15.3:0/3696905940 >> 172.21.15.3:0/3696905940 conn(0x7f76b408afe0 msgr2=0x7f76b40d35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:17.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.960+0000 7f76bc64e700 1 -- 172.21.15.3:0/3696905940 shutdown_connections 2024-05-08T21:23:17.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.960+0000 7f76bc64e700 1 -- 172.21.15.3:0/3696905940 wait complete. 2024-05-08T21:23:17.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.962+0000 7f76bc64e700 1 Processor -- start 2024-05-08T21:23:17.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.962+0000 7f76bc64e700 1 -- start start 2024-05-08T21:23:17.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76bc64e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76b40d1e50 0x7f76b40c7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76bc64e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76b40d2230 0x7f76b40cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76bc64e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b4161320 0x7f76b4153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76bc64e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b40c7700 con 0x7f76b4161320 2024-05-08T21:23:17.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76bc64e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b40c79e0 con 0x7f76b40d1e50 2024-05-08T21:23:17.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76bc64e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b40c7cc0 con 0x7f76b40d2230 2024-05-08T21:23:17.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76babeb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b4161320 0x7f76b4153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76babeb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b4161320 0x7f76b4153b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57660/0 (socket says 172.21.15.3:57660) 2024-05-08T21:23:17.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76babeb700 1 -- 172.21.15.3:0/3450520111 learned_addr learned my addr 172.21.15.3:0/3450520111 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:17.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76b9be9700 1 --2- 172.21.15.3:0/3450520111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76b40d2230 0x7f76b40cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76ba3ea700 1 --2- 172.21.15.3:0/3450520111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76b40d1e50 0x7f76b40c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76babeb700 1 -- 172.21.15.3:0/3450520111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76b40d2230 msgr2=0x7f76b40cb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76babeb700 1 --2- 172.21.15.3:0/3450520111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76b40d2230 0x7f76b40cb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76babeb700 1 -- 172.21.15.3:0/3450520111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76b40d1e50 msgr2=0x7f76b40c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76babeb700 1 --2- 172.21.15.3:0/3450520111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76b40d1e50 0x7f76b40c7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76babeb700 1 -- 172.21.15.3:0/3450520111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a40094a0 con 0x7f76b4161320 2024-05-08T21:23:17.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76babeb700 1 --2- 172.21.15.3:0/3450520111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b4161320 0x7f76b4153b70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f76ac001700 tx=0x7f76ac00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:17.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76aaffd700 1 -- 172.21.15.3:0/3450520111 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac00ab00 con 0x7f76b4161320 2024-05-08T21:23:17.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.963+0000 7f76aaffd700 1 -- 172.21.15.3:0/3450520111 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76ac00ac60 con 0x7f76b4161320 2024-05-08T21:23:17.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.964+0000 7f76bc64e700 1 -- 172.21.15.3:0/3450520111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b40c7fa0 con 0x7f76b4161320 2024-05-08T21:23:17.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.964+0000 7f76aaffd700 1 -- 172.21.15.3:0/3450520111 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac0070d0 con 0x7f76b4161320 2024-05-08T21:23:17.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.964+0000 7f76bc64e700 1 -- 172.21.15.3:0/3450520111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b41540b0 con 0x7f76b4161320 2024-05-08T21:23:17.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.966+0000 7f76aaffd700 1 -- 172.21.15.3:0/3450520111 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f76ac007250 con 0x7f76b4161320 2024-05-08T21:23:17.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.966+0000 7f76aaffd700 1 --2- 172.21.15.3:0/3450520111 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f769806ac90 0x7f769806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.966+0000 7f76aaffd700 1 -- 172.21.15.3:0/3450520111 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f76ac089340 con 0x7f76b4161320 2024-05-08T21:23:17.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.967+0000 7f76ba3ea700 1 --2- 172.21.15.3:0/3450520111 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f769806ac90 0x7f769806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.967+0000 7f76ba3ea700 1 --2- 172.21.15.3:0/3450520111 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f769806ac90 0x7f769806d150 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f76b00011e0 tx=0x7f76b0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:17.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.968+0000 7f76c8808b80 1 Processor -- start 2024-05-08T21:23:17.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.968+0000 7f76c8808b80 1 -- start start 2024-05-08T21:23:17.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76c8808b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd82840 0x5591edd5b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76c8808b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd5d410 0x5591edd60140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76c8808b80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd89e70 0x5591edd8a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76c8808b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591edcb61f0 con 0x5591edd82840 2024-05-08T21:23:17.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76c8808b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591edcb3bb0 con 0x5591edd5d410 2024-05-08T21:23:17.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76c8808b80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x5591edcaf4a0 con 0x5591edd89e70 2024-05-08T21:23:17.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76ba3ea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd82840 0x5591edd5b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76ba3ea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd82840 0x5591edd5b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57666/0 (socket says 172.21.15.3:57666) 2024-05-08T21:23:17.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76ba3ea700 1 -- 172.21.15.3:0/3573392662 learned_addr learned my addr 172.21.15.3:0/3573392662 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:17.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76b9be9700 1 --2- 172.21.15.3:0/3573392662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd5d410 0x5591edd60140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76babeb700 1 --2- 172.21.15.3:0/3573392662 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd89e70 0x5591edd8a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76ba3ea700 1 -- 172.21.15.3:0/3573392662 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd89e70 msgr2=0x5591edd8a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76ba3ea700 1 --2- 172.21.15.3:0/3573392662 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd89e70 0x5591edd8a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76ba3ea700 1 -- 172.21.15.3:0/3573392662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd5d410 msgr2=0x5591edd60140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76ba3ea700 1 --2- 172.21.15.3:0/3573392662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd5d410 0x5591edd60140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76ba3ea700 1 -- 172.21.15.3:0/3573392662 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5591edb30450 con 0x5591edd82840 2024-05-08T21:23:17.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.969+0000 7f76ba3ea700 1 --2- 172.21.15.3:0/3573392662 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd82840 0x5591edd5b1b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f76b0007fd0 tx=0x7f76b0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7786572b4b7237ec server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:17.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.970+0000 7f7697fff700 1 -- 172.21.15.3:0/3573392662 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b000fe10 con 0x5591edd82840 2024-05-08T21:23:17.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.970+0000 7f7697fff700 1 -- 172.21.15.3:0/3573392662 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b0014450 con 0x5591edd82840 2024-05-08T21:23:17.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.970+0000 7f7697fff700 1 -- 172.21.15.3:0/3573392662 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b00147c0 con 0x5591edd82840 2024-05-08T21:23:17.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.970+0000 7f76c8808b80 1 -- 172.21.15.3:0/3573392662 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd82840 msgr2=0x5591edd5b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.970+0000 7f76c8808b80 1 --2- 172.21.15.3:0/3573392662 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd82840 0x5591edd5b1b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f76b0007fd0 tx=0x7f76b0012000 comp rx=0 tx=0).stop 2024-05-08T21:23:17.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.970+0000 7f76c8808b80 1 -- 172.21.15.3:0/3573392662 shutdown_connections 2024-05-08T21:23:17.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.970+0000 7f76c8808b80 1 --2- 172.21.15.3:0/3573392662 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd89e70 0x5591edd8a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.970+0000 7f76c8808b80 1 --2- 172.21.15.3:0/3573392662 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd82840 0x5591edd5b1b0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.970+0000 7f76c8808b80 1 --2- 172.21.15.3:0/3573392662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd5d410 0x5591edd60140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.970+0000 7f76c8808b80 1 -- 172.21.15.3:0/3573392662 >> 172.21.15.3:0/3573392662 conn(0x5591edcbd2b0 msgr2=0x5591edda4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:17.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.970+0000 7f76c8808b80 1 -- 172.21.15.3:0/3573392662 shutdown_connections 2024-05-08T21:23:17.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.971+0000 7f76c8808b80 1 -- 172.21.15.3:0/3573392662 wait complete. 2024-05-08T21:23:17.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.971+0000 7f76c8808b80 1 -- start start 2024-05-08T21:23:17.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.971+0000 7f76c8808b80 1 Processor -- start 2024-05-08T21:23:17.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.972+0000 7f76c8808b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd5d410 0x5591eddaa150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.972+0000 7f76c8808b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd82840 0x5591edd814d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.972+0000 7f76c8808b80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd89e70 0x5591edd8c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.972+0000 7f76c8808b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591edc4b600 con 0x5591edd5d410 2024-05-08T21:23:17.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.972+0000 7f76c8808b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591edc4b2a0 con 0x5591edd82840 2024-05-08T21:23:17.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.972+0000 7f76c8808b80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x5591edb32740 con 0x5591edd89e70 2024-05-08T21:23:17.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.972+0000 7f76c8808b80 10 client.0 _ll_register_callbacks cb 0x5591edc54bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-08T21:23:17.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.972+0000 7f76ba3ea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd5d410 0x5591eddaa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.972+0000 7f76ba3ea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd5d410 0x5591eddaa150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57672/0 (socket says 172.21.15.3:57672) 2024-05-08T21:23:17.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.972+0000 7f76ba3ea700 1 -- 172.21.15.3:0/3077476143 learned_addr learned my addr 172.21.15.3:0/3077476143 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:17.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.973+0000 7f76b9be9700 1 --2- 172.21.15.3:0/3077476143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd82840 0x5591edd814d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.973+0000 7f76babeb700 1 --2- 172.21.15.3:0/3077476143 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd89e70 0x5591edd8c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.973+0000 7f76ba3ea700 1 -- 172.21.15.3:0/3077476143 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd89e70 msgr2=0x5591edd8c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.973+0000 7f76ba3ea700 1 --2- 172.21.15.3:0/3077476143 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd89e70 0x5591edd8c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.973+0000 7f76ba3ea700 1 -- 172.21.15.3:0/3077476143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd82840 msgr2=0x5591edd814d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.973+0000 7f76ba3ea700 1 --2- 172.21.15.3:0/3077476143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd82840 0x5591edd814d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.973+0000 7f76ba3ea700 1 -- 172.21.15.3:0/3077476143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b40cf560 con 0x5591edd5d410 2024-05-08T21:23:17.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.973+0000 7f76ba3ea700 1 --2- 172.21.15.3:0/3077476143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd5d410 0x5591eddaa150 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f76b000f4b0 tx=0x7f76b0001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b471119484b0d8b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:17.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.973+0000 7f76957fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-05-08T21:23:17.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.973+0000 7f76957fa700 1 -- 172.21.15.3:0/3077476143 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b002b620 con 0x5591edd5d410 2024-05-08T21:23:17.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.973+0000 7f76957fa700 1 -- 172.21.15.3:0/3077476143 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f76b001c040 con 0x5591edd5d410 2024-05-08T21:23:17.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.973+0000 7f76c8808b80 1 -- 172.21.15.3:0/3077476143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5591edc08310 con 0x5591edd5d410 2024-05-08T21:23:17.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.973+0000 7f76957fa700 1 -- 172.21.15.3:0/3077476143 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76b002b780 con 0x5591edd5d410 2024-05-08T21:23:17.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.973+0000 7f76c8808b80 10 client.5553 fetch_fsmap learned FSMap version 26 2024-05-08T21:23:17.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.973+0000 7f76c8808b80 1 -- 172.21.15.3:0/3077476143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x5591edadcc00 con 0x5591edd5d410 2024-05-08T21:23:17.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.974+0000 7f76957fa700 1 -- 172.21.15.3:0/3077476143 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5591edc08310 con 0x5591edd5d410 2024-05-08T21:23:17.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.974+0000 7f76957fa700 1 -- 172.21.15.3:0/3077476143 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 26) v1 ==== 3687+0+0 (secure 0 0 0) 0x7f76b002b8e0 con 0x5591edd5d410 2024-05-08T21:23:17.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.974+0000 7f76c8808b80 10 client.5553 fetch_fsmap finished waiting for FSMap version 26 2024-05-08T21:23:17.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.974+0000 7f76c8808b80 10 client.5553 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-05-08T21:23:17.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.974+0000 7f76c8808b80 20 client.5553 populate_metadata read hostname 'smithi003' 2024-05-08T21:23:17.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.974+0000 7f76c8808b80 1 --2- 172.21.15.3:0/3077476143 >> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] conn(0x5591edd80ff0 0x5591edd8d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.974+0000 7f76c8808b80 4 client.5553 mds_command: new command op to 5191 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-08T21:23:17.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.974+0000 7f76c8808b80 1 -- 172.21.15.3:0/3077476143 --> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5591edd2e880 con 0x5591edd80ff0 2024-05-08T21:23:17.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.974+0000 7f76b9be9700 1 --2- 172.21.15.3:0/3077476143 >> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] conn(0x5591edd80ff0 0x5591edd8d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.974+0000 7f76957fa700 10 client.5553 ms_handle_connect on v2:172.21.15.186:6835/2000072997 2024-05-08T21:23:17.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.974+0000 7f76b9be9700 1 --2- 172.21.15.3:0/3077476143 >> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] conn(0x5591edd80ff0 0x5591edd8d2b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45073f84299a358f server_cookie=e72b1580de78b7e6 in_seq=0 out_seq=0 2024-05-08T21:23:17.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.975+0000 7f76957fa700 1 -- 172.21.15.3:0/3077476143 <== mds.0 v2:172.21.15.186:6835/2000072997 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5591edd2e880 con 0x5591edd80ff0 2024-05-08T21:23:17.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.975+0000 7f76957fa700 10 client.5553 handle_command_reply: tid=0 2024-05-08T21:23:17.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.975+0000 7f76c8808b80 1 client.5553 shutdown 2024-05-08T21:23:17.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.975+0000 7f76c8808b80 1 -- 172.21.15.3:0/3077476143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd5d410 msgr2=0x5591eddaa150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:17.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.975+0000 7f76c8808b80 1 --2- 172.21.15.3:0/3077476143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd5d410 0x5591eddaa150 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f76b000f4b0 tx=0x7f76b0001150 comp rx=0 tx=0).stop 2024-05-08T21:23:17.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.975+0000 7f76c8808b80 1 -- 172.21.15.3:0/3077476143 shutdown_connections 2024-05-08T21:23:17.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.975+0000 7f76c8808b80 1 --2- 172.21.15.3:0/3077476143 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd89e70 0x5591edd8c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.975+0000 7f76c8808b80 1 --2- 172.21.15.3:0/3077476143 >> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] conn(0x5591edd80ff0 0x5591edd8d2b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.975+0000 7f76c8808b80 1 --2- 172.21.15.3:0/3077476143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd5d410 0x5591eddaa150 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.975+0000 7f76c8808b80 1 --2- 172.21.15.3:0/3077476143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd82840 0x5591edd814d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.975+0000 7f76957fa700 0 client.5553 ms_handle_reset on v2:172.21.15.186:6835/2000072997 2024-05-08T21:23:17.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.975+0000 7f76c8808b80 1 -- 172.21.15.3:0/3077476143 >> 172.21.15.3:0/3077476143 conn(0x5591edcbd2b0 msgr2=0x5591edd8a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:17.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.975+0000 7f76c8808b80 1 -- 172.21.15.3:0/3077476143 shutdown_connections 2024-05-08T21:23:17.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.975+0000 7f76c8808b80 1 -- 172.21.15.3:0/3077476143 wait complete. 2024-05-08T21:23:17.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.976+0000 7f76c8808b80 20 client.5553 trim_cache size 0 max 16384 2024-05-08T21:23:17.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.994+0000 7f76c8808b80 1 Processor -- start 2024-05-08T21:23:17.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.994+0000 7f76c8808b80 1 -- start start 2024-05-08T21:23:17.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.995+0000 7f76c8808b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd5d410 0x5591edd603b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.995+0000 7f76c8808b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd82840 0x5591eddacd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.995+0000 7f76c8808b80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd89e70 0x5591ede04650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:17.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.995+0000 7f76c8808b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591edda5490 con 0x5591edd82840 2024-05-08T21:23:17.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.995+0000 7f76c8808b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591edd5c720 con 0x5591edd5d410 2024-05-08T21:23:17.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.995+0000 7f76c8808b80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x5591eddc6f30 con 0x5591edd89e70 2024-05-08T21:23:17.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.995+0000 7f76b9be9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd82840 0x5591eddacd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.995+0000 7f76b9be9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd82840 0x5591eddacd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57674/0 (socket says 172.21.15.3:57674) 2024-05-08T21:23:17.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.995+0000 7f76b9be9700 1 -- 172.21.15.3:0/2245003502 learned_addr learned my addr 172.21.15.3:0/2245003502 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:17.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.995+0000 7f76babeb700 1 --2- 172.21.15.3:0/2245003502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd89e70 0x5591ede04650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.995+0000 7f76ba3ea700 1 --2- 172.21.15.3:0/2245003502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd5d410 0x5591edd603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:17.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.995+0000 7f76b9be9700 1 -- 172.21.15.3:0/2245003502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd89e70 msgr2=0x5591ede04650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.995+0000 7f76b9be9700 1 --2- 172.21.15.3:0/2245003502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd89e70 0x5591ede04650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.996+0000 7f76b9be9700 1 -- 172.21.15.3:0/2245003502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd5d410 msgr2=0x5591edd603b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:17.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.996+0000 7f76b9be9700 1 --2- 172.21.15.3:0/2245003502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd5d410 0x5591edd603b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:17.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.996+0000 7f76b9be9700 1 -- 172.21.15.3:0/2245003502 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5591edc75600 con 0x5591edd82840 2024-05-08T21:23:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.996+0000 7f76b9be9700 1 --2- 172.21.15.3:0/2245003502 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd82840 0x5591eddacd70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f76a40090b0 tx=0x7f76a4010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e0304e41862731e server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.996+0000 7f76a8ff9700 1 -- 172.21.15.3:0/2245003502 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a4022040 con 0x5591edd82840 2024-05-08T21:23:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.996+0000 7f76a8ff9700 1 -- 172.21.15.3:0/2245003502 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a4028610 con 0x5591edd82840 2024-05-08T21:23:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.996+0000 7f76a8ff9700 1 -- 172.21.15.3:0/2245003502 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a4004350 con 0x5591edd82840 2024-05-08T21:23:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.996+0000 7f76c8808b80 1 -- 172.21.15.3:0/2245003502 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd82840 msgr2=0x5591eddacd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.996+0000 7f76c8808b80 1 --2- 172.21.15.3:0/2245003502 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd82840 0x5591eddacd70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f76a40090b0 tx=0x7f76a4010000 comp rx=0 tx=0).stop 2024-05-08T21:23:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.996+0000 7f76c8808b80 1 -- 172.21.15.3:0/2245003502 shutdown_connections 2024-05-08T21:23:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.996+0000 7f76c8808b80 1 --2- 172.21.15.3:0/2245003502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd89e70 0x5591ede04650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.997+0000 7f76c8808b80 1 --2- 172.21.15.3:0/2245003502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd5d410 0x5591edd603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.997+0000 7f76c8808b80 1 --2- 172.21.15.3:0/2245003502 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd82840 0x5591eddacd70 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.997+0000 7f76c8808b80 1 -- 172.21.15.3:0/2245003502 >> 172.21.15.3:0/2245003502 conn(0x5591edcbd2b0 msgr2=0x5591edd5b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.997+0000 7f76c8808b80 1 -- 172.21.15.3:0/2245003502 shutdown_connections 2024-05-08T21:23:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.997+0000 7f76c8808b80 1 -- 172.21.15.3:0/2245003502 wait complete. 2024-05-08T21:23:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.997+0000 7f76c8808b80 1 -- start start 2024-05-08T21:23:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.997+0000 7f76c8808b80 1 Processor -- start 2024-05-08T21:23:18.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.998+0000 7f76c8808b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd5d410 0x5591ede017f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:18.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.998+0000 7f76c8808b80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd82840 0x5591edd5f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:18.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.998+0000 7f76c8808b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd89e70 0x5591edda8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:18.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.998+0000 7f76c8808b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591edd5fc70 con 0x5591edd89e70 2024-05-08T21:23:18.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.998+0000 7f76c8808b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591ede02030 con 0x5591edd5d410 2024-05-08T21:23:18.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.998+0000 7f76c8808b80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x5591edd7e210 con 0x5591edd82840 2024-05-08T21:23:18.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.998+0000 7f76c8808b80 10 client.0 _ll_register_callbacks cb 0x5591edc54bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-08T21:23:18.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.998+0000 7f76babeb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd89e70 0x5591edda8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:18.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76babeb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd89e70 0x5591edda8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57684/0 (socket says 172.21.15.3:57684) 2024-05-08T21:23:18.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76babeb700 1 -- 172.21.15.3:0/1936245776 learned_addr learned my addr 172.21.15.3:0/1936245776 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:18.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76b9be9700 1 --2- 172.21.15.3:0/1936245776 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd82840 0x5591edd5f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:18.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76ba3ea700 1 --2- 172.21.15.3:0/1936245776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd5d410 0x5591ede017f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:18.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76babeb700 1 -- 172.21.15.3:0/1936245776 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd82840 msgr2=0x5591edd5f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:18.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76babeb700 1 --2- 172.21.15.3:0/1936245776 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd82840 0x5591edd5f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76babeb700 1 -- 172.21.15.3:0/1936245776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd5d410 msgr2=0x5591ede017f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:18.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76babeb700 1 --2- 172.21.15.3:0/1936245776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd5d410 0x5591ede017f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76babeb700 1 -- 172.21.15.3:0/1936245776 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b40cef10 con 0x5591edd89e70 2024-05-08T21:23:18.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76babeb700 1 --2- 172.21.15.3:0/1936245776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd89e70 0x5591edda8ca0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f76ac054550 tx=0x7f76ac0544c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fccc7559b32acf76 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:18.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76957fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-05-08T21:23:18.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76957fa700 1 -- 172.21.15.3:0/1936245776 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac060440 con 0x5591edd89e70 2024-05-08T21:23:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76957fa700 1 -- 172.21.15.3:0/1936245776 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f76ac0890b0 con 0x5591edd89e70 2024-05-08T21:23:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76c8808b80 1 -- 172.21.15.3:0/1936245776 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f76b0014450 con 0x5591edd89e70 2024-05-08T21:23:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76957fa700 1 -- 172.21.15.3:0/1936245776 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76ac0872a0 con 0x5591edd89e70 2024-05-08T21:23:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76c8808b80 10 client.5559 fetch_fsmap learned FSMap version 26 2024-05-08T21:23:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:17.999+0000 7f76c8808b80 1 -- 172.21.15.3:0/1936245776 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x5591edb34d70 con 0x5591edd89e70 2024-05-08T21:23:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.000+0000 7f76957fa700 1 -- 172.21.15.3:0/1936245776 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b0014450 con 0x5591edd89e70 2024-05-08T21:23:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.000+0000 7f76957fa700 1 -- 172.21.15.3:0/1936245776 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 26) v1 ==== 3687+0+0 (secure 0 0 0) 0x7f76ac05ffb0 con 0x5591edd89e70 2024-05-08T21:23:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.000+0000 7f76c8808b80 10 client.5559 fetch_fsmap finished waiting for FSMap version 26 2024-05-08T21:23:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.000+0000 7f76c8808b80 10 client.5559 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-05-08T21:23:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.000+0000 7f76c8808b80 20 client.5559 populate_metadata read hostname 'smithi003' 2024-05-08T21:23:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.000+0000 7f76c8808b80 1 --2- 172.21.15.3:0/1936245776 >> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] conn(0x5591edda91e0 0x5591edda3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.000+0000 7f76c8808b80 4 client.5559 mds_command: new command op to 5191 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive", "repair"]}] 2024-05-08T21:23:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.000+0000 7f76c8808b80 1 -- 172.21.15.3:0/1936245776 --> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive", "repair"]}) v1 -- 0x5591edca7f60 con 0x5591edda91e0 2024-05-08T21:23:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.000+0000 7f76ba3ea700 1 --2- 172.21.15.3:0/1936245776 >> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] conn(0x5591edda91e0 0x5591edda3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.000+0000 7f76957fa700 10 client.5559 ms_handle_connect on v2:172.21.15.186:6835/2000072997 2024-05-08T21:23:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.000+0000 7f76ba3ea700 1 --2- 172.21.15.3:0/1936245776 >> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] conn(0x5591edda91e0 0x5591edda3f60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4b729fa77711f10 server_cookie=b7714e7901eb6355 in_seq=0 out_seq=0 2024-05-08T21:23:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.001+0000 7f76957fa700 1 -- 172.21.15.3:0/1936245776 <== mds.0 v2:172.21.15.186:6835/2000072997 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5591edca7f60 con 0x5591edda91e0 2024-05-08T21:23:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.001+0000 7f76957fa700 10 client.5559 handle_command_reply: tid=0 2024-05-08T21:23:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.001+0000 7f76c8808b80 1 client.5559 shutdown 2024-05-08T21:23:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.001+0000 7f76c8808b80 1 -- 172.21.15.3:0/1936245776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd89e70 msgr2=0x5591edda8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.001+0000 7f76c8808b80 1 --2- 172.21.15.3:0/1936245776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd89e70 0x5591edda8ca0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f76ac054550 tx=0x7f76ac0544c0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.001+0000 7f76c8808b80 1 -- 172.21.15.3:0/1936245776 shutdown_connections 2024-05-08T21:23:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.001+0000 7f76c8808b80 1 --2- 172.21.15.3:0/1936245776 >> [v2:172.21.15.186:6835/2000072997,v1:172.21.15.186:6837/2000072997] conn(0x5591edda91e0 0x5591edda3f60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.001+0000 7f76c8808b80 1 --2- 172.21.15.3:0/1936245776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591edd89e70 0x5591edda8ca0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.001+0000 7f76c8808b80 1 --2- 172.21.15.3:0/1936245776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591edd5d410 0x5591ede017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.001+0000 7f76c8808b80 1 --2- 172.21.15.3:0/1936245776 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5591edd82840 0x5591edd5f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.001+0000 7f76c8808b80 1 -- 172.21.15.3:0/1936245776 >> 172.21.15.3:0/1936245776 conn(0x5591edcbd2b0 msgr2=0x5591ede02560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.001+0000 7f76957fa700 0 client.5559 ms_handle_reset on v2:172.21.15.186:6835/2000072997 2024-05-08T21:23:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.002+0000 7f76c8808b80 1 -- 172.21.15.3:0/1936245776 shutdown_connections 2024-05-08T21:23:18.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.002+0000 7f76c8808b80 1 -- 172.21.15.3:0/1936245776 wait complete. 2024-05-08T21:23:18.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.002+0000 7f76c8808b80 20 client.5559 trim_cache size 0 max 16384 2024-05-08T21:23:18.008 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:23:18.008 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-05-08T21:23:18.008 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "10d196e7-2c5c-4833-b776-3c188f0e03f9", 2024-05-08T21:23:18.008 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-05-08T21:23:18.008 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:23:18.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.002+0000 7f76bc64e700 1 -- 172.21.15.3:0/3450520111 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f769806ac90 msgr2=0x7f769806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.002+0000 7f76bc64e700 1 --2- 172.21.15.3:0/3450520111 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f769806ac90 0x7f769806d150 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f76b00011e0 tx=0x7f76b0009040 comp rx=0 tx=0).stop 2024-05-08T21:23:18.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.002+0000 7f76bc64e700 1 -- 172.21.15.3:0/3450520111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b4161320 msgr2=0x7f76b4153b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.002+0000 7f76bc64e700 1 --2- 172.21.15.3:0/3450520111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b4161320 0x7f76b4153b70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f76ac001700 tx=0x7f76ac00cf70 comp rx=0 tx=0).stop 2024-05-08T21:23:18.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.003+0000 7f76bc64e700 1 -- 172.21.15.3:0/3450520111 shutdown_connections 2024-05-08T21:23:18.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.003+0000 7f76bc64e700 1 --2- 172.21.15.3:0/3450520111 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f769806ac90 0x7f769806d150 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.003+0000 7f76bc64e700 1 --2- 172.21.15.3:0/3450520111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b4161320 0x7f76b4153b70 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.003+0000 7f76bc64e700 1 --2- 172.21.15.3:0/3450520111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76b40d1e50 0x7f76b40c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.003+0000 7f76bc64e700 1 --2- 172.21.15.3:0/3450520111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76b40d2230 0x7f76b40cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.003+0000 7f76bc64e700 1 -- 172.21.15.3:0/3450520111 >> 172.21.15.3:0/3450520111 conn(0x7f76b408afe0 msgr2=0x7f76b40d0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:18.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.003+0000 7f76bc64e700 1 -- 172.21.15.3:0/3450520111 shutdown_connections 2024-05-08T21:23:18.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.003+0000 7f76bc64e700 1 -- 172.21.15.3:0/3450520111 wait complete. 2024-05-08T21:23:18.010 INFO:tasks.ceph.mds.d.smithi186.stderr:2024-05-08T21:23:18.002+0000 7f6cfa1b6700 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x10000000000 2024-05-08T21:23:18.010 INFO:tasks.ceph.mds.d.smithi186.stderr:2024-05-08T21:23:18.003+0000 7f6cfa1b6700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /subdir/ auth v1 pv4 ap=3 DIRTYPARENT f(v0 1=1+0) n() (inest lock dirty) (ifile lock->sync w=1 flushing) (iversion lock) | dirtyscattered=2 lock=1 dirfrag=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55d2cb12d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(6)0x10000000000:[<0x1/subdir v1>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 1=1+0)","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 1=1+0)","memory_value.rstat":"n()","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-61} 2024-05-08T21:23:18.010 INFO:tasks.ceph.mds.d.smithi186.stderr:2024-05-08T21:23:18.004+0000 7f6cfa1b6700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /subdir/sixmegs auth v1 ap=1 DIRTYPARENT s=6291456 n() (iversion lock) | dirtyparent=1 authpin=1 scrubqueue=0 0x55d2cb173b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(5)0x10000000001:[<0x10000000000/sixmegs v10>,<0x1/subdir v12>]//[]","memoryvalue":"(5)0x10000000001:[<0x10000000000/sixmegs v1>,<0x1/subdir v1>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-08T21:23:18.011 INFO:tasks.ceph.mds.d.smithi186.stderr:2024-05-08T21:23:18.008+0000 7f6cfa1b6700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1 [...2,head] / auth v4 ap=5 snaprealm=0x55d2cb128d80 DIRTYPARENT f(v1 1=0+1) n(v0 1=0+1)/n() (idft sync r=1) (inest sync r=1) (ifile sync->mix r=1) (iversion lock) | dirtyscattered=0 lock=3 dirfrag=1 openingsnapparents=0 dirtyparent=1 dirty=1 waiter=1 authpin=1 scrubqueue=0 0x55d2cb12cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(6)0x1:[]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 1=0+1)","ondisk_value.rstat":"n(v0 2=0+2)","memory_value.dirstat":"f(v1 1=0+1)","memory_value.rstat":"n(v0 1=0+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-61} 2024-05-08T21:23:18.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:23:18.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.153+0000 7f7b7bf68700 1 Processor -- start 2024-05-08T21:23:18.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.153+0000 7f7b7bf68700 1 -- start start 2024-05-08T21:23:18.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.154+0000 7f7b7bf68700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7b7415ef60 0x7f7b740b2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:18.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.154+0000 7f7b7bf68700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7b740cfb70 0x7f7b740b4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:18.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.154+0000 7f7b7bf68700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7409ad40 con 0x7f7b740cf000 2024-05-08T21:23:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.154+0000 7f7b7bf68700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b740cd400 con 0x7f7b7415ef60 2024-05-08T21:23:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.154+0000 7f7b7bf68700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b740ccb60 con 0x7f7b740cfb70 2024-05-08T21:23:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.154+0000 7f7b79d04700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7b7415ef60 0x7f7b740b2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.154+0000 7f7b79503700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7b740cf000 0x7f7b740cf460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51152/0 (socket says 172.21.15.3:51152) 2024-05-08T21:23:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.154+0000 7f7b79503700 1 -- 172.21.15.3:0/3837984676 learned_addr learned my addr 172.21.15.3:0/3837984676 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.155+0000 7f7b7a505700 1 --2- 172.21.15.3:0/3837984676 >> v2:172.21.15.186:3300/0 conn(0x7f7b740cfb70 0x7f7b740b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.155+0000 7f7b79d04700 1 -- 172.21.15.3:0/3837984676 >> v2:172.21.15.186:3300/0 conn(0x7f7b740cfb70 msgr2=0x7f7b740b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.155+0000 7f7b79d04700 1 --2- 172.21.15.3:0/3837984676 >> v2:172.21.15.186:3300/0 conn(0x7f7b740cfb70 0x7f7b740b4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.155+0000 7f7b79d04700 1 -- 172.21.15.3:0/3837984676 >> v1:172.21.15.3:6789/0 conn(0x7f7b740cf000 legacy=0x7f7b740cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.155+0000 7f7b78d02700 1 -- 172.21.15.3:0/3837984676 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314526960 0 0) 0x7f7b7409ad40 con 0x7f7b740cf000 2024-05-08T21:23:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.155+0000 7f7b79d04700 1 -- 172.21.15.3:0/3837984676 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b740cce40 con 0x7f7b7415ef60 2024-05-08T21:23:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.155+0000 7f7b7a505700 1 --2- 172.21.15.3:0/3837984676 >> v2:172.21.15.186:3300/0 conn(0x7f7b740cfb70 0x7f7b740b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.155+0000 7f7b79d04700 1 --2- 172.21.15.3:0/3837984676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b7415ef60 0x7f7b740b2340 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7b74072340 tx=0x7f7b70009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29f86fde430f9e85 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.155+0000 7f7b78d02700 1 -- 172.21.15.3:0/3837984676 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b70011830 con 0x7f7b7415ef60 2024-05-08T21:23:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.155+0000 7f7b78d02700 1 -- 172.21.15.3:0/3837984676 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b70011990 con 0x7f7b7415ef60 2024-05-08T21:23:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.155+0000 7f7b78d02700 1 -- 172.21.15.3:0/3837984676 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b70011d20 con 0x7f7b7415ef60 2024-05-08T21:23:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.156+0000 7f7b7bf68700 1 -- 172.21.15.3:0/3837984676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b7415ef60 msgr2=0x7f7b740b2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.156+0000 7f7b7bf68700 1 --2- 172.21.15.3:0/3837984676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b7415ef60 0x7f7b740b2340 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7b74072340 tx=0x7f7b70009b20 comp rx=0 tx=0).stop 2024-05-08T21:23:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.156+0000 7f7b7bf68700 1 -- 172.21.15.3:0/3837984676 shutdown_connections 2024-05-08T21:23:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.156+0000 7f7b7bf68700 1 --2- 172.21.15.3:0/3837984676 >> v2:172.21.15.186:3300/0 conn(0x7f7b740cfb70 0x7f7b740b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.156+0000 7f7b7bf68700 1 --2- 172.21.15.3:0/3837984676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b7415ef60 0x7f7b740b2340 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.156+0000 7f7b7bf68700 1 -- 172.21.15.3:0/3837984676 >> 172.21.15.3:0/3837984676 conn(0x7f7b740b2f20 msgr2=0x7f7b740b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.156+0000 7f7b7bf68700 1 -- 172.21.15.3:0/3837984676 shutdown_connections 2024-05-08T21:23:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.156+0000 7f7b7bf68700 1 -- 172.21.15.3:0/3837984676 wait complete. 2024-05-08T21:23:18.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.157+0000 7f7b7bf68700 1 Processor -- start 2024-05-08T21:23:18.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.157+0000 7f7b7bf68700 1 -- start start 2024-05-08T21:23:18.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b7bf68700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b740cf000 0x7f7b7409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:18.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b7bf68700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7b740cfb70 0x7f7b74153730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:18.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b7bf68700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b7415ef60 0x7f7b741f3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:18.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b7bf68700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7409d210 con 0x7f7b7415ef60 2024-05-08T21:23:18.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b7bf68700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7409d4f0 con 0x7f7b740cf000 2024-05-08T21:23:18.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b7bf68700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7409d7d0 con 0x7f7b740cfb70 2024-05-08T21:23:18.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b7a505700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b7415ef60 0x7f7b741f3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:18.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b7a505700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b7415ef60 0x7f7b741f3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57698/0 (socket says 172.21.15.3:57698) 2024-05-08T21:23:18.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b7a505700 1 -- 172.21.15.3:0/1999995377 learned_addr learned my addr 172.21.15.3:0/1999995377 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:18.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b79d04700 1 --2- 172.21.15.3:0/1999995377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b740cf000 0x7f7b7409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:18.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b79503700 1 --2- 172.21.15.3:0/1999995377 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7b740cfb70 0x7f7b74153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:18.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b7a505700 1 -- 172.21.15.3:0/1999995377 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7b740cfb70 msgr2=0x7f7b74153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b7a505700 1 --2- 172.21.15.3:0/1999995377 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7b740cfb70 0x7f7b74153730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b7a505700 1 -- 172.21.15.3:0/1999995377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b740cf000 msgr2=0x7f7b7409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b7a505700 1 --2- 172.21.15.3:0/1999995377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b740cf000 0x7f7b7409ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b7a505700 1 -- 172.21.15.3:0/1999995377 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b700094a0 con 0x7f7b7415ef60 2024-05-08T21:23:18.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.158+0000 7f7b79d04700 1 --2- 172.21.15.3:0/1999995377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b740cf000 0x7f7b7409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:18.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.159+0000 7f7b7a505700 1 --2- 172.21.15.3:0/1999995377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b7415ef60 0x7f7b741f3590 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00cea0 tx=0x7f7b6c00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:18.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.159+0000 7f7b6a7fc700 1 -- 172.21.15.3:0/1999995377 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c00ae50 con 0x7f7b7415ef60 2024-05-08T21:23:18.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.159+0000 7f7b6a7fc700 1 -- 172.21.15.3:0/1999995377 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c003a20 con 0x7f7b7415ef60 2024-05-08T21:23:18.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.159+0000 7f7b7bf68700 1 -- 172.21.15.3:0/1999995377 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7409b260 con 0x7f7b7415ef60 2024-05-08T21:23:18.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.159+0000 7f7b6a7fc700 1 -- 172.21.15.3:0/1999995377 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c003dd0 con 0x7f7b7415ef60 2024-05-08T21:23:18.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.159+0000 7f7b7bf68700 1 -- 172.21.15.3:0/1999995377 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b74153c70 con 0x7f7b7415ef60 2024-05-08T21:23:18.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.160+0000 7f7b7bf68700 1 -- 172.21.15.3:0/1999995377 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7409b780 con 0x7f7b7415ef60 2024-05-08T21:23:18.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.160+0000 7f7b6a7fc700 1 -- 172.21.15.3:0/1999995377 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7b6c003f30 con 0x7f7b7415ef60 2024-05-08T21:23:18.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.161+0000 7f7b6a7fc700 1 --2- 172.21.15.3:0/1999995377 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7b5806ac90 0x7f7b5806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:18.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.161+0000 7f7b6a7fc700 1 -- 172.21.15.3:0/1999995377 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f7b6c089a10 con 0x7f7b7415ef60 2024-05-08T21:23:18.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.161+0000 7f7b79d04700 1 --2- 172.21.15.3:0/1999995377 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7b5806ac90 0x7f7b5806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:18.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.162+0000 7f7b79d04700 1 --2- 172.21.15.3:0/1999995377 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7b5806ac90 0x7f7b5806d150 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7b70009b60 tx=0x7f7b70008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:18.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.166+0000 7f7b6a7fc700 1 -- 172.21.15.3:0/1999995377 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f7b7409b780 con 0x7f7b7415ef60 2024-05-08T21:23:18.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.436+0000 7f7b7bf68700 1 -- 172.21.15.3:0/1999995377 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b7409b780 con 0x7f7b7415ef60 2024-05-08T21:23:18.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.436+0000 7f7b6a7fc700 1 -- 172.21.15.3:0/1999995377 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5936 (secure 0 0 0) 0x7f7b7409b780 con 0x7f7b7415ef60 2024-05-08T21:23:18.438 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:18.438 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6833/2104267343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2104267343},{"type":"v1","addr":"172.21.15.3:6833","nonce":2104267343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/208321708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":208321708},{"type":"v1","addr":"172.21.15.3:6835","nonce":208321708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/1591386131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1591386131},{"type":"v1","addr":"172.21.15.186:6836","nonce":1591386131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:22:50.184245+0000","modified":"2024-05-08T21:23:01.563810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:23:05.475355+0000","modified":"2024-05-08T21:23:15.662101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6837/2000072997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":2000072997},{"type":"v1","addr":"172.21.15.186:6837","nonce":2000072997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":6,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-08T21:23:18.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.438+0000 7f7b7bf68700 1 -- 172.21.15.3:0/1999995377 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7b5806ac90 msgr2=0x7f7b5806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.438+0000 7f7b7bf68700 1 --2- 172.21.15.3:0/1999995377 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7b5806ac90 0x7f7b5806d150 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7b70009b60 tx=0x7f7b70008030 comp rx=0 tx=0).stop 2024-05-08T21:23:18.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.438+0000 7f7b7bf68700 1 -- 172.21.15.3:0/1999995377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b7415ef60 msgr2=0x7f7b741f3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.438+0000 7f7b7bf68700 1 --2- 172.21.15.3:0/1999995377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b7415ef60 0x7f7b741f3590 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00cea0 tx=0x7f7b6c00cf80 comp rx=0 tx=0).stop 2024-05-08T21:23:18.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.438+0000 7f7b7bf68700 1 -- 172.21.15.3:0/1999995377 shutdown_connections 2024-05-08T21:23:18.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.438+0000 7f7b7bf68700 1 --2- 172.21.15.3:0/1999995377 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7b5806ac90 0x7f7b5806d150 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.438+0000 7f7b7bf68700 1 --2- 172.21.15.3:0/1999995377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b7415ef60 0x7f7b741f3590 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.438+0000 7f7b7bf68700 1 --2- 172.21.15.3:0/1999995377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b740cf000 0x7f7b7409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.438+0000 7f7b7bf68700 1 --2- 172.21.15.3:0/1999995377 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7b740cfb70 0x7f7b74153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.438+0000 7f7b7bf68700 1 -- 172.21.15.3:0/1999995377 >> 172.21.15.3:0/1999995377 conn(0x7f7b740b2f20 msgr2=0x7f7b740b5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:18.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.439+0000 7f7b7bf68700 1 -- 172.21.15.3:0/1999995377 shutdown_connections 2024-05-08T21:23:18.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.439+0000 7f7b7bf68700 1 -- 172.21.15.3:0/1999995377 wait complete. 2024-05-08T21:23:18.441 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 26 2024-05-08T21:23:18.454 INFO:tasks.cephfs.test_recovery_pool:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 26, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-08T21:22:50.184245+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [ 0 ], "flags": 19, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": false, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": {}, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-05-08T21:23:01.563810+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": {} } }, { "id": 3, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-08T21:23:05.475355+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 26, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "recovery_fs", "in": [ 0 ], "info": { "gid_5191": { "addr": "172.21.15.186:6837/2000072997", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 2000072997, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 2000072997, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5191, "incarnation": 23, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 6, "modified": "2024-05-08T21:23:15.662101+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5191 } } } ], "standbys": [ { "addr": "172.21.15.3:6833/2104267343", "addrs": { "addrvec": [ { "addr": "172.21.15.3:6832", "nonce": 2104267343, "type": "v2" }, { "addr": "172.21.15.3:6833", "nonce": 2104267343, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 18, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4937, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.3:6835/208321708", "addrs": { "addrvec": [ { "addr": "172.21.15.3:6834", "nonce": 208321708, "type": "v2" }, { "addr": "172.21.15.3:6835", "nonce": 208321708, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 18, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4940, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/1591386131", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 1591386131, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 1591386131, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 18, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5188, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-08T21:23:18.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-08T21:23:18.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.590+0000 7fc1ce66d700 1 Processor -- start 2024-05-08T21:23:18.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.590+0000 7fc1ce66d700 1 -- start start 2024-05-08T21:23:18.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.590+0000 7fc1ce66d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc1c806a960 0x7fc1c815d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:18.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.590+0000 7fc1ce66d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c80cd980 con 0x7fc1c80d0eb0 2024-05-08T21:23:18.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.590+0000 7fc1ce66d700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c80ccdb0 con 0x7fc1c80cf840 2024-05-08T21:23:18.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.590+0000 7fc1ce66d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c80cd050 con 0x7fc1c806a960 2024-05-08T21:23:18.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.591+0000 7fc1ccc0a700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc1c80d0eb0 0x7fc1c8161620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51160/0 (socket says 172.21.15.3:51160) 2024-05-08T21:23:18.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.591+0000 7fc1ccc0a700 1 -- 172.21.15.3:0/3866867814 learned_addr learned my addr 172.21.15.3:0/3866867814 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:18.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.591+0000 7fc1c77fe700 1 --2- 172.21.15.3:0/3866867814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc1c806a960 0x7fc1c815d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:18.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.591+0000 7fc1c6ffd700 1 -- 172.21.15.3:0/3866867814 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417463342 0 0) 0x7fc1c80cd980 con 0x7fc1c80d0eb0 2024-05-08T21:23:18.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.591+0000 7fc1c6ffd700 1 -- 172.21.15.3:0/3866867814 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b40035b0 con 0x7fc1c80d0eb0 2024-05-08T21:23:18.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.591+0000 7fc1c6ffd700 1 -- 172.21.15.3:0/3866867814 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994930768 0 0) 0x7fc1c80ccdb0 con 0x7fc1c80cf840 2024-05-08T21:23:18.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.591+0000 7fc1c6ffd700 1 -- 172.21.15.3:0/3866867814 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1c80cd980 con 0x7fc1c80cf840 2024-05-08T21:23:18.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.591+0000 7fc1c77fe700 1 -- 172.21.15.3:0/3866867814 >> v1:172.21.15.186:6789/0 conn(0x7fc1c80cf840 legacy=0x7fc1c80af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.591+0000 7fc1c77fe700 1 -- 172.21.15.3:0/3866867814 >> v1:172.21.15.3:6789/0 conn(0x7fc1c80d0eb0 legacy=0x7fc1c8161620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.591+0000 7fc1c77fe700 1 -- 172.21.15.3:0/3866867814 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c80cd330 con 0x7fc1c806a960 2024-05-08T21:23:18.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.591+0000 7fc1c77fe700 1 --2- 172.21.15.3:0/3866867814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc1c806a960 0x7fc1c815d4f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0009870 tx=0x7fc1b0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b3b75ad4c6a376d server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:18.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.592+0000 7fc1c6ffd700 1 -- 172.21.15.3:0/3866867814 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b0011830 con 0x7fc1c806a960 2024-05-08T21:23:18.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.592+0000 7fc1c6ffd700 1 -- 172.21.15.3:0/3866867814 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b0011990 con 0x7fc1c806a960 2024-05-08T21:23:18.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.592+0000 7fc1c6ffd700 1 -- 172.21.15.3:0/3866867814 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b0011d20 con 0x7fc1c806a960 2024-05-08T21:23:18.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.592+0000 7fc1ce66d700 1 -- 172.21.15.3:0/3866867814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc1c806a960 msgr2=0x7fc1c815d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.592+0000 7fc1ce66d700 1 --2- 172.21.15.3:0/3866867814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc1c806a960 0x7fc1c815d4f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0009870 tx=0x7fc1b0009b80 comp rx=0 tx=0).stop 2024-05-08T21:23:18.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.592+0000 7fc1ce66d700 1 -- 172.21.15.3:0/3866867814 shutdown_connections 2024-05-08T21:23:18.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.592+0000 7fc1ce66d700 1 --2- 172.21.15.3:0/3866867814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc1c806a960 0x7fc1c815d4f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.592+0000 7fc1ce66d700 1 -- 172.21.15.3:0/3866867814 >> 172.21.15.3:0/3866867814 conn(0x7fc1c80d0320 msgr2=0x7fc1c80d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:18.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.592+0000 7fc1ce66d700 1 -- 172.21.15.3:0/3866867814 shutdown_connections 2024-05-08T21:23:18.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.592+0000 7fc1ce66d700 1 -- 172.21.15.3:0/3866867814 wait complete. 2024-05-08T21:23:18.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.593+0000 7fc1ce66d700 1 Processor -- start 2024-05-08T21:23:18.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.594+0000 7fc1ce66d700 1 -- start start 2024-05-08T21:23:18.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.594+0000 7fc1ce66d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1c806a960 0x7fc1c814d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:18.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.594+0000 7fc1ce66d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1c80cf840 0x7fc1c809a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:18.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.594+0000 7fc1ce66d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc1c80d0eb0 0x7fc1c81f7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:18.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.594+0000 7fc1ce66d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c814dd20 con 0x7fc1c806a960 2024-05-08T21:23:18.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.594+0000 7fc1ce66d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c809ad00 con 0x7fc1c80cf840 2024-05-08T21:23:18.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.594+0000 7fc1ce66d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c81f7a70 con 0x7fc1c80d0eb0 2024-05-08T21:23:18.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1c7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1c806a960 0x7fc1c814d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:18.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1c7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1c806a960 0x7fc1c814d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57714/0 (socket says 172.21.15.3:57714) 2024-05-08T21:23:18.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1c7fff700 1 -- 172.21.15.3:0/2992279375 learned_addr learned my addr 172.21.15.3:0/2992279375 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:18.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1c77fe700 1 --2- 172.21.15.3:0/2992279375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1c80cf840 0x7fc1c809a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:18.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1ccc0a700 1 --2- 172.21.15.3:0/2992279375 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc1c80d0eb0 0x7fc1c81f7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:18.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1c7fff700 1 -- 172.21.15.3:0/2992279375 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc1c80d0eb0 msgr2=0x7fc1c81f7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1c7fff700 1 --2- 172.21.15.3:0/2992279375 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc1c80d0eb0 0x7fc1c81f7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1c7fff700 1 -- 172.21.15.3:0/2992279375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1c80cf840 msgr2=0x7fc1c809a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1c7fff700 1 --2- 172.21.15.3:0/2992279375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1c80cf840 0x7fc1c809a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1c7fff700 1 -- 172.21.15.3:0/2992279375 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b00094a0 con 0x7fc1c806a960 2024-05-08T21:23:18.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1ccc0a700 1 --2- 172.21.15.3:0/2992279375 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc1c80d0eb0 0x7fc1c81f7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:18.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1c7fff700 1 --2- 172.21.15.3:0/2992279375 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1c806a960 0x7fc1c814d7e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800c520 tx=0x7fc1b800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:18.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1c4ff9700 1 -- 172.21.15.3:0/2992279375 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80052d0 con 0x7fc1c806a960 2024-05-08T21:23:18.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1c4ff9700 1 -- 172.21.15.3:0/2992279375 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b8005430 con 0x7fc1c806a960 2024-05-08T21:23:18.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1ce66d700 1 -- 172.21.15.3:0/2992279375 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c80b53c0 con 0x7fc1c806a960 2024-05-08T21:23:18.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1c4ff9700 1 -- 172.21.15.3:0/2992279375 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80038a0 con 0x7fc1c806a960 2024-05-08T21:23:18.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.595+0000 7fc1ce66d700 1 -- 172.21.15.3:0/2992279375 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c80b3200 con 0x7fc1c806a960 2024-05-08T21:23:18.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.596+0000 7fc1ce66d700 1 -- 172.21.15.3:0/2992279375 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c8072df0 con 0x7fc1c806a960 2024-05-08T21:23:18.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.597+0000 7fc1c4ff9700 1 -- 172.21.15.3:0/2992279375 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc1b8003a00 con 0x7fc1c806a960 2024-05-08T21:23:18.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.597+0000 7fc1c4ff9700 1 --2- 172.21.15.3:0/2992279375 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc1a806ac90 0x7fc1a806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:18.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.597+0000 7fc1c4ff9700 1 -- 172.21.15.3:0/2992279375 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7fc1b8088a50 con 0x7fc1c806a960 2024-05-08T21:23:18.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.598+0000 7fc1c77fe700 1 --2- 172.21.15.3:0/2992279375 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc1a806ac90 0x7fc1a806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:18.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.598+0000 7fc1c77fe700 1 --2- 172.21.15.3:0/2992279375 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc1a806ac90 0x7fc1a806d150 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0009870 tx=0x7fc1b001c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:18.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.602+0000 7fc1c4ff9700 1 -- 172.21.15.3:0/2992279375 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc1b8053a10 con 0x7fc1c806a960 2024-05-08T21:23:18.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.867+0000 7fc1ce66d700 1 -- 172.21.15.3:0/2992279375 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc1c8068000 con 0x7fc1c806a960 2024-05-08T21:23:18.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.868+0000 7fc1c4ff9700 1 -- 172.21.15.3:0/2992279375 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+156 (secure 0 0 0) 0x7fc1b8017070 con 0x7fc1c806a960 2024-05-08T21:23:18.869 INFO:teuthology.orchestra.run.smithi003.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-08T21:23:18.870 INFO:teuthology.orchestra.run.smithi003.stdout:name: recovery_fs, metadata pool: recovery_fs_metadata, data pools: [cephfs_data ] 2024-05-08T21:23:18.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.870+0000 7fc1ce66d700 1 -- 172.21.15.3:0/2992279375 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc1a806ac90 msgr2=0x7fc1a806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.870+0000 7fc1ce66d700 1 --2- 172.21.15.3:0/2992279375 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc1a806ac90 0x7fc1a806d150 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0009870 tx=0x7fc1b001c040 comp rx=0 tx=0).stop 2024-05-08T21:23:18.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.870+0000 7fc1ce66d700 1 -- 172.21.15.3:0/2992279375 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1c806a960 msgr2=0x7fc1c814d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:18.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.870+0000 7fc1ce66d700 1 --2- 172.21.15.3:0/2992279375 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1c806a960 0x7fc1c814d7e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800c520 tx=0x7fc1b800c830 comp rx=0 tx=0).stop 2024-05-08T21:23:18.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.870+0000 7fc1ce66d700 1 -- 172.21.15.3:0/2992279375 shutdown_connections 2024-05-08T21:23:18.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.870+0000 7fc1ce66d700 1 --2- 172.21.15.3:0/2992279375 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc1c80d0eb0 0x7fc1c81f7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.870+0000 7fc1ce66d700 1 --2- 172.21.15.3:0/2992279375 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc1a806ac90 0x7fc1a806d150 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.870+0000 7fc1ce66d700 1 --2- 172.21.15.3:0/2992279375 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1c806a960 0x7fc1c814d7e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.870+0000 7fc1ce66d700 1 --2- 172.21.15.3:0/2992279375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1c80cf840 0x7fc1c809a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:18.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.870+0000 7fc1ce66d700 1 -- 172.21.15.3:0/2992279375 >> 172.21.15.3:0/2992279375 conn(0x7fc1c80d0320 msgr2=0x7fc1c815ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:18.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.870+0000 7fc1ce66d700 1 -- 172.21.15.3:0/2992279375 shutdown_connections 2024-05-08T21:23:18.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:18.870+0000 7fc1ce66d700 1 -- 172.21.15.3:0/2992279375 wait complete. 2024-05-08T21:23:18.885 INFO:tasks.cephfs.mount:Mounting Ceph FS recovery_fs; just confirmed its presence on cluster 2024-05-08T21:23:18.885 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-08T21:23:18.886 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi003.front.sepia.ceph.com 2024-05-08T21:23:18.886 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-08T21:23:18.886 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-08T21:23:18.886 INFO:tasks.cephfs.mount:self.cephfs_name = recovery_fs 2024-05-08T21:23:18.886 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-05-08T21:23:18.886 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-08T21:23:18.886 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-08T21:23:18.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:18.886 DEBUG:teuthology.orchestra.run.smithi003:> ip addr 2024-05-08T21:23:18.906 INFO:teuthology.orchestra.run.smithi003.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-08T21:23:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-08T21:23:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: inet 127.0.0.1/8 scope host lo 2024-05-08T21:23:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:23:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 ::1/128 scope host 2024-05-08T21:23:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:23:18.906 INFO:teuthology.orchestra.run.smithi003.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:23:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ac brd ff:ff:ff:ff:ff:ff 2024-05-08T21:23:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f0 2024-05-08T21:23:18.907 INFO:teuthology.orchestra.run.smithi003.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:23:18.907 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ad brd ff:ff:ff:ff:ff:ff 2024-05-08T21:23:18.907 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f1 2024-05-08T21:23:18.907 INFO:teuthology.orchestra.run.smithi003.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-08T21:23:18.907 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dc brd ff:ff:ff:ff:ff:ff 2024-05-08T21:23:18.907 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f0 2024-05-08T21:23:18.907 INFO:teuthology.orchestra.run.smithi003.stdout: inet 172.21.15.3/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-08T21:23:18.907 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft 42133sec preferred_lft 42133sec 2024-05-08T21:23:18.907 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::ec4:7aff:febd:15dc/64 scope link 2024-05-08T21:23:18.907 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:23:18.907 INFO:teuthology.orchestra.run.smithi003.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T21:23:18.907 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dd brd ff:ff:ff:ff:ff:ff 2024-05-08T21:23:18.907 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f1 2024-05-08T21:23:18.907 INFO:teuthology.orchestra.run.smithi003.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-08T21:23:18.907 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether a6:6a:ef:07:5e:cb brd ff:ff:ff:ff:ff:ff 2024-05-08T21:23:18.908 INFO:teuthology.orchestra.run.smithi003.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-08T21:23:18.908 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:23:18.908 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::a89e:98ff:fec6:3929/64 scope link 2024-05-08T21:23:18.908 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:23:18.908 INFO:teuthology.orchestra.run.smithi003.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-08T21:23:18.908 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether ee:e2:a2:93:f0:94 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-08T21:23:18.908 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::ece2:a2ff:fe93:f094/64 scope link 2024-05-08T21:23:18.908 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:23:18.908 INFO:teuthology.orchestra.run.smithi003.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-08T21:23:18.908 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether a6:6a:ef:07:5e:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-08T21:23:18.908 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::a46a:efff:fe07:5ecb/64 scope link 2024-05-08T21:23:18.908 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-05-08T21:23:18.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:18.908 DEBUG:teuthology.orchestra.run.smithi003:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-08T21:23:18.977 INFO:teuthology.orchestra.run.smithi003.stdout:1 2024-05-08T21:23:18.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:18.979 DEBUG:teuthology.orchestra.run.smithi003:> route 2024-05-08T21:23:19.043 INFO:teuthology.orchestra.run.smithi003.stdout:Kernel IP routing table 2024-05-08T21:23:19.043 INFO:teuthology.orchestra.run.smithi003.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-08T21:23:19.043 INFO:teuthology.orchestra.run.smithi003.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-08T21:23:19.043 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-08T21:23:19.044 INFO:teuthology.orchestra.run.smithi003.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-08T21:23:19.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:19.044 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2024-05-08T21:23:19.044 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-05-08T21:23:19.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-08T21:23:19.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-08T21:23:19.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-08T21:23:19.044 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-05-08T21:23:19.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:19.169 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2024-05-08T21:23:19.189 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-08T21:23:19.190 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-08T21:23:19.190 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-08T21:23:19.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:19.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set brx.0 up 2024-05-08T21:23:19.264 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-08T21:23:19.265 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-08T21:23:19.327 INFO:teuthology.orchestra.run.smithi003.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-08T21:23:19.328 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-08T21:23:19.328 DEBUG:teuthology.orchestra.run.smithi003:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-08T21:23:19.389 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-05-08T21:23:19.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:19.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=recovery_fs 2024-05-08T21:23:19.488 INFO:teuthology.orchestra.run.smithi003.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=recovery_fs 2024-05-08T21:23:19.488 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=recovery_fs". 2024-05-08T21:23:19.488 INFO:teuthology.orchestra.run.smithi003.stdout:invalid new device string format 2024-05-08T21:23:19.498 INFO:teuthology.orchestra.run.smithi003.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=recovery_fs 2024-05-08T21:23:19.498 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=recovery_fs". 2024-05-08T21:23:19.498 INFO:teuthology.orchestra.run.smithi003.stdout:invalid new device string format 2024-05-08T21:23:19.499 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: resolved to: "172.21.15.3:3300,172.21.15.186:3300,172.21.15.186:3301" 2024-05-08T21:23:19.499 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-08T21:23:19.518 INFO:teuthology.orchestra.run.smithi003.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=recovery_fs 2024-05-08T21:23:19.518 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=recovery_fs". 2024-05-08T21:23:19.518 INFO:teuthology.orchestra.run.smithi003.stdout:invalid new device string format 2024-05-08T21:23:19.518 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: resolved to: "172.21.15.3:3300,172.21.15.186:3300,172.21.15.186:3301" 2024-05-08T21:23:19.518 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-08T21:23:19.518 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: resolved to: "172.21.15.3:6789,172.21.15.186:6789,172.21.15.186:6790" 2024-05-08T21:23:19.518 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.3:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2024-05-08T21:23:19.518 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=recovery_fs,key=0,fsid=d0f58275-758f-4470-b003-84d6abe25489" will pass to kernel 2024-05-08T21:23:19.521 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-08T21:23:19.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:19.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-08T21:23:19.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:19.553 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-08T21:23:19.648 INFO:teuthology.orchestra.run.smithi003.stdout:client5177 2024-05-08T21:23:19.649 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:23:19.649 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:23:19.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:19.671 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls subdir) 2024-05-08T21:23:19.751 INFO:teuthology.orchestra.run.smithi003.stdout:sixmegs 2024-05-08T21:23:19.752 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits daemon-helper kill python3 -c ' 2024-05-08T21:23:19.752 DEBUG:teuthology.orchestra.run.smithi003:> import os 2024-05-08T21:23:19.752 DEBUG:teuthology.orchestra.run.smithi003:> import stat 2024-05-08T21:23:19.752 DEBUG:teuthology.orchestra.run.smithi003:> import json 2024-05-08T21:23:19.752 DEBUG:teuthology.orchestra.run.smithi003:> import sys 2024-05-08T21:23:19.753 DEBUG:teuthology.orchestra.run.smithi003:> 2024-05-08T21:23:19.753 DEBUG:teuthology.orchestra.run.smithi003:> try: 2024-05-08T21:23:19.753 DEBUG:teuthology.orchestra.run.smithi003:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/sixmegs'"'"') 2024-05-08T21:23:19.753 DEBUG:teuthology.orchestra.run.smithi003:> except OSError as e: 2024-05-08T21:23:19.753 DEBUG:teuthology.orchestra.run.smithi003:> sys.exit(e.errno) 2024-05-08T21:23:19.753 DEBUG:teuthology.orchestra.run.smithi003:> 2024-05-08T21:23:19.753 DEBUG:teuthology.orchestra.run.smithi003:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-05-08T21:23:19.753 DEBUG:teuthology.orchestra.run.smithi003:> print(json.dumps( 2024-05-08T21:23:19.753 DEBUG:teuthology.orchestra.run.smithi003:> dict([(a, getattr(s, a)) for a in attrs]), 2024-05-08T21:23:19.753 DEBUG:teuthology.orchestra.run.smithi003:> indent=2)) 2024-05-08T21:23:19.753 DEBUG:teuthology.orchestra.run.smithi003:> ' 2024-05-08T21:23:19.896 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:23:19.896 INFO:teuthology.orchestra.run.smithi003.stdout: "st_mode": 33088, 2024-05-08T21:23:19.896 INFO:teuthology.orchestra.run.smithi003.stdout: "st_ino": 1099511627777, 2024-05-08T21:23:19.896 INFO:teuthology.orchestra.run.smithi003.stdout: "st_dev": 47, 2024-05-08T21:23:19.896 INFO:teuthology.orchestra.run.smithi003.stdout: "st_nlink": 1, 2024-05-08T21:23:19.897 INFO:teuthology.orchestra.run.smithi003.stdout: "st_uid": 0, 2024-05-08T21:23:19.897 INFO:teuthology.orchestra.run.smithi003.stdout: "st_gid": 0, 2024-05-08T21:23:19.897 INFO:teuthology.orchestra.run.smithi003.stdout: "st_size": 6291456, 2024-05-08T21:23:19.897 INFO:teuthology.orchestra.run.smithi003.stdout: "st_atime": 1715203381.0, 2024-05-08T21:23:19.897 INFO:teuthology.orchestra.run.smithi003.stdout: "st_mtime": 1715203381.0, 2024-05-08T21:23:19.897 INFO:teuthology.orchestra.run.smithi003.stdout: "st_ctime": 1715203381.0 2024-05-08T21:23:19.897 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:23:20.062 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-08T21:23:20.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-08T21:23:20.157 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-08T21:23:20.206 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-08T21:23:20.206 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:23:20.206 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:23:20.228 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:23:20.229 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:23:20.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:23:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.473+0000 7fbedbee7700 1 Processor -- start 2024-05-08T21:23:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.473+0000 7fbedbee7700 1 -- start start 2024-05-08T21:23:20.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.474+0000 7fbedbee7700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbed415ef60 0x7fbed40b2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:20.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.474+0000 7fbedbee7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbed40cf000 0x7fbed40cf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.474+0000 7fbedbee7700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbed40cf9a0 0x7fbed4099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.474+0000 7fbedbee7700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbed40cd400 con 0x7fbed415ef60 2024-05-08T21:23:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.474+0000 7fbedbee7700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbed40cfe30 con 0x7fbed40cf9a0 2024-05-08T21:23:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.474+0000 7fbedbee7700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed409ad40 con 0x7fbed40cf000 2024-05-08T21:23:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.474+0000 7fbed9c83700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbed415ef60 0x7fbed40b2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.474+0000 7fbed9c83700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbed415ef60 0x7fbed40b2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57726/0 (socket says 172.21.15.3:57726) 2024-05-08T21:23:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.474+0000 7fbed9c83700 1 -- 172.21.15.3:0/473760949 learned_addr learned my addr 172.21.15.3:0/473760949 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.475+0000 7fbeda484700 1 --2- 172.21.15.3:0/473760949 >> v2:172.21.15.186:3300/0 conn(0x7fbed40cf9a0 0x7fbed4099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.475+0000 7fbed9482700 1 --2- 172.21.15.3:0/473760949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbed40cf000 0x7fbed40cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.475+0000 7fbed9c83700 1 -- 172.21.15.3:0/473760949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbed40cf000 msgr2=0x7fbed40cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.475+0000 7fbed9c83700 1 --2- 172.21.15.3:0/473760949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbed40cf000 0x7fbed40cf460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.475+0000 7fbed9c83700 1 -- 172.21.15.3:0/473760949 >> v2:172.21.15.186:3300/0 conn(0x7fbed40cf9a0 msgr2=0x7fbed4099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.475+0000 7fbed9c83700 1 --2- 172.21.15.3:0/473760949 >> v2:172.21.15.186:3300/0 conn(0x7fbed40cf9a0 0x7fbed4099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.475+0000 7fbed9c83700 1 -- 172.21.15.3:0/473760949 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed40ccb60 con 0x7fbed415ef60 2024-05-08T21:23:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.475+0000 7fbed9482700 1 --2- 172.21.15.3:0/473760949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbed40cf000 0x7fbed40cf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.475+0000 7fbeda484700 1 --2- 172.21.15.3:0/473760949 >> v2:172.21.15.186:3300/0 conn(0x7fbed40cf9a0 0x7fbed4099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.475+0000 7fbed9c83700 1 --2- 172.21.15.3:0/473760949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbed415ef60 0x7fbed40b2340 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fbed4072200 tx=0x7fbed0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4562126cbdc826a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.475+0000 7fbed8c81700 1 -- 172.21.15.3:0/473760949 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed000b870 con 0x7fbed415ef60 2024-05-08T21:23:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.475+0000 7fbed8c81700 1 -- 172.21.15.3:0/473760949 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed000b9d0 con 0x7fbed415ef60 2024-05-08T21:23:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.475+0000 7fbed8c81700 1 -- 172.21.15.3:0/473760949 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed000bd60 con 0x7fbed415ef60 2024-05-08T21:23:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.476+0000 7fbedbee7700 1 -- 172.21.15.3:0/473760949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbed415ef60 msgr2=0x7fbed40b2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.476+0000 7fbedbee7700 1 --2- 172.21.15.3:0/473760949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbed415ef60 0x7fbed40b2340 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fbed4072200 tx=0x7fbed0009bd0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.476+0000 7fbedbee7700 1 -- 172.21.15.3:0/473760949 shutdown_connections 2024-05-08T21:23:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.476+0000 7fbedbee7700 1 --2- 172.21.15.3:0/473760949 >> v2:172.21.15.186:3300/0 conn(0x7fbed40cf9a0 0x7fbed4099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.476+0000 7fbedbee7700 1 --2- 172.21.15.3:0/473760949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbed415ef60 0x7fbed40b2340 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.476+0000 7fbedbee7700 1 --2- 172.21.15.3:0/473760949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbed40cf000 0x7fbed40cf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.476+0000 7fbedbee7700 1 -- 172.21.15.3:0/473760949 >> 172.21.15.3:0/473760949 conn(0x7fbed40b2f20 msgr2=0x7fbed40b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.476+0000 7fbedbee7700 1 -- 172.21.15.3:0/473760949 shutdown_connections 2024-05-08T21:23:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.476+0000 7fbedbee7700 1 -- 172.21.15.3:0/473760949 wait complete. 2024-05-08T21:23:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.477+0000 7fbedbee7700 1 Processor -- start 2024-05-08T21:23:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.477+0000 7fbedbee7700 1 -- start start 2024-05-08T21:23:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbedbee7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbed40cf000 0x7fbed415ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbedbee7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbed40cf9a0 0x7fbed40c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbedbee7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbed415ef60 0x7fbed406fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbedbee7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed415b220 con 0x7fbed40cf000 2024-05-08T21:23:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbedbee7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed40c9760 con 0x7fbed40cf9a0 2024-05-08T21:23:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbedbee7700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed40c9a40 con 0x7fbed415ef60 2024-05-08T21:23:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbed9c83700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbed40cf000 0x7fbed415ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbed9c83700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbed40cf000 0x7fbed415ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57738/0 (socket says 172.21.15.3:57738) 2024-05-08T21:23:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbed9c83700 1 -- 172.21.15.3:0/2290341798 learned_addr learned my addr 172.21.15.3:0/2290341798 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbed9482700 1 --2- 172.21.15.3:0/2290341798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbed40cf9a0 0x7fbed40c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbeda484700 1 --2- 172.21.15.3:0/2290341798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbed415ef60 0x7fbed406fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbed9c83700 1 -- 172.21.15.3:0/2290341798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbed415ef60 msgr2=0x7fbed406fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbed9c83700 1 --2- 172.21.15.3:0/2290341798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbed415ef60 0x7fbed406fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbed9c83700 1 -- 172.21.15.3:0/2290341798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbed40cf9a0 msgr2=0x7fbed40c8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbed9c83700 1 --2- 172.21.15.3:0/2290341798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbed40cf9a0 0x7fbed40c8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbed9c83700 1 -- 172.21.15.3:0/2290341798 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed0009550 con 0x7fbed40cf000 2024-05-08T21:23:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.478+0000 7fbed9482700 1 --2- 172.21.15.3:0/2290341798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbed40cf9a0 0x7fbed40c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.479+0000 7fbed9c83700 1 --2- 172.21.15.3:0/2290341798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbed40cf000 0x7fbed415ace0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fbed0000c00 tx=0x7fbed0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.479+0000 7fbeca7fc700 1 -- 172.21.15.3:0/2290341798 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed0004520 con 0x7fbed40cf000 2024-05-08T21:23:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.479+0000 7fbeca7fc700 1 -- 172.21.15.3:0/2290341798 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed001e070 con 0x7fbed40cf000 2024-05-08T21:23:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.479+0000 7fbedbee7700 1 -- 172.21.15.3:0/2290341798 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed40c9d20 con 0x7fbed40cf000 2024-05-08T21:23:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.479+0000 7fbeca7fc700 1 -- 172.21.15.3:0/2290341798 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed00038c0 con 0x7fbed40cf000 2024-05-08T21:23:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.479+0000 7fbedbee7700 1 -- 172.21.15.3:0/2290341798 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed40c9450 con 0x7fbed40cf000 2024-05-08T21:23:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.480+0000 7fbedbee7700 1 -- 172.21.15.3:0/2290341798 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed4068000 con 0x7fbed40cf000 2024-05-08T21:23:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.480+0000 7fbeca7fc700 1 -- 172.21.15.3:0/2290341798 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbed0003a20 con 0x7fbed40cf000 2024-05-08T21:23:20.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.481+0000 7fbeca7fc700 1 --2- 172.21.15.3:0/2290341798 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fbeb806ac90 0x7fbeb806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:20.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.481+0000 7fbeca7fc700 1 -- 172.21.15.3:0/2290341798 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7fbed0020070 con 0x7fbed40cf000 2024-05-08T21:23:20.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.481+0000 7fbed9482700 1 --2- 172.21.15.3:0/2290341798 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fbeb806ac90 0x7fbeb806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:20.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.482+0000 7fbed9482700 1 --2- 172.21.15.3:0/2290341798 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fbeb806ac90 0x7fbeb806d150 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbed415e1c0 tx=0x7fbec4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:20.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.486+0000 7fbeca7fc700 1 -- 172.21.15.3:0/2290341798 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fbed0055ae0 con 0x7fbed40cf000 2024-05-08T21:23:20.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.704+0000 7fbedbee7700 1 -- 172.21.15.3:0/2290341798 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbed41f2ee0 con 0x7fbed40cf000 2024-05-08T21:23:20.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.705+0000 7fbeca7fc700 1 -- 172.21.15.3:0/2290341798 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+18029 (secure 0 0 0) 0x7fbed002a090 con 0x7fbed40cf000 2024-05-08T21:23:20.707 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:20.707 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":42,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:23:06.476868+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:22:47.853786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-08T21:22:49.153759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"recovery_fs_metadata","create_time":"2024-05-08T21:23:03.921695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6834/3599481370":"2024-05-09T21:23:01.563392+0000","172.21.15.3:6832/3599481370":"2024-05-09T21:23:01.563392+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:23:20.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.708+0000 7fbedbee7700 1 -- 172.21.15.3:0/2290341798 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fbeb806ac90 msgr2=0x7fbeb806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:20.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.708+0000 7fbedbee7700 1 --2- 172.21.15.3:0/2290341798 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fbeb806ac90 0x7fbeb806d150 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbed415e1c0 tx=0x7fbec4008040 comp rx=0 tx=0).stop 2024-05-08T21:23:20.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.708+0000 7fbedbee7700 1 -- 172.21.15.3:0/2290341798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbed40cf000 msgr2=0x7fbed415ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:20.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.708+0000 7fbedbee7700 1 --2- 172.21.15.3:0/2290341798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbed40cf000 0x7fbed415ace0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fbed0000c00 tx=0x7fbed0004750 comp rx=0 tx=0).stop 2024-05-08T21:23:20.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.708+0000 7fbedbee7700 1 -- 172.21.15.3:0/2290341798 shutdown_connections 2024-05-08T21:23:20.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.708+0000 7fbedbee7700 1 --2- 172.21.15.3:0/2290341798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbed415ef60 0x7fbed406fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.708+0000 7fbedbee7700 1 --2- 172.21.15.3:0/2290341798 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fbeb806ac90 0x7fbeb806d150 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.708+0000 7fbedbee7700 1 --2- 172.21.15.3:0/2290341798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbed40cf000 0x7fbed415ace0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.708+0000 7fbedbee7700 1 --2- 172.21.15.3:0/2290341798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbed40cf9a0 0x7fbed40c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.708+0000 7fbedbee7700 1 -- 172.21.15.3:0/2290341798 >> 172.21.15.3:0/2290341798 conn(0x7fbed40b2f20 msgr2=0x7fbed4073910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:20.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.708+0000 7fbedbee7700 1 -- 172.21.15.3:0/2290341798 shutdown_connections 2024-05-08T21:23:20.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.708+0000 7fbedbee7700 1 -- 172.21.15.3:0/2290341798 wait complete. 2024-05-08T21:23:20.723 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-08T21:22:47.853786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:22:49.153759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'recovery_fs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'recovery_fs_metadata', 'create_time': '2024-05-08T21:23:03.921695+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'recovery_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:23:20.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:23:20.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.861+0000 7fe5df278700 1 Processor -- start 2024-05-08T21:23:20.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.861+0000 7fe5df278700 1 -- start start 2024-05-08T21:23:20.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.862+0000 7fe5df278700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe5d8145040 0x7fe5d80b5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:20.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.862+0000 7fe5df278700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5d815fbb0 0x7fe5d80d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:20.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.862+0000 7fe5df278700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe5d80d0890 0x7fe5d806a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:20.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.862+0000 7fe5df278700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5d80d0d20 con 0x7fe5d80d0890 2024-05-08T21:23:20.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.862+0000 7fe5df278700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5d80b6460 con 0x7fe5d8145040 2024-05-08T21:23:20.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.862+0000 7fe5df278700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80af6e0 con 0x7fe5d815fbb0 2024-05-08T21:23:20.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.862+0000 7fe5dd815700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe5d80d0890 0x7fe5d806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:20.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.863+0000 7fe5dd815700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe5d80d0890 0x7fe5d806a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57740/0 (socket says 172.21.15.3:57740) 2024-05-08T21:23:20.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.863+0000 7fe5dd815700 1 -- 172.21.15.3:0/1020581669 learned_addr learned my addr 172.21.15.3:0/1020581669 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:20.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.863+0000 7fe5dc813700 1 --2- 172.21.15.3:0/1020581669 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5d815fbb0 0x7fe5d80d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:20.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.863+0000 7fe5dd014700 1 --2- 172.21.15.3:0/1020581669 >> v2:172.21.15.186:3300/0 conn(0x7fe5d8145040 0x7fe5d80b5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:20.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.863+0000 7fe5dd815700 1 -- 172.21.15.3:0/1020581669 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5d815fbb0 msgr2=0x7fe5d80d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:20.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.863+0000 7fe5dd815700 1 --2- 172.21.15.3:0/1020581669 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5d815fbb0 0x7fe5d80d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.863+0000 7fe5dd815700 1 -- 172.21.15.3:0/1020581669 >> v2:172.21.15.186:3300/0 conn(0x7fe5d8145040 msgr2=0x7fe5d80b5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.863+0000 7fe5dd815700 1 --2- 172.21.15.3:0/1020581669 >> v2:172.21.15.186:3300/0 conn(0x7fe5d8145040 0x7fe5d80b5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.863+0000 7fe5dd815700 1 -- 172.21.15.3:0/1020581669 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d80aeb50 con 0x7fe5d80d0890 2024-05-08T21:23:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.863+0000 7fe5dd014700 1 --2- 172.21.15.3:0/1020581669 >> v2:172.21.15.186:3300/0 conn(0x7fe5d8145040 0x7fe5d80b5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.863+0000 7fe5dd815700 1 --2- 172.21.15.3:0/1020581669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d80d0890 0x7fe5d806a960 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80d1460 tx=0x7fe5d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b56dc0701b35b7aa server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.863+0000 7fe5cffff700 1 -- 172.21.15.3:0/1020581669 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d000e070 con 0x7fe5d80d0890 2024-05-08T21:23:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.863+0000 7fe5cffff700 1 -- 172.21.15.3:0/1020581669 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d000b980 con 0x7fe5d80d0890 2024-05-08T21:23:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.863+0000 7fe5cffff700 1 -- 172.21.15.3:0/1020581669 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d000bd30 con 0x7fe5d80d0890 2024-05-08T21:23:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.864+0000 7fe5df278700 1 -- 172.21.15.3:0/1020581669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d80d0890 msgr2=0x7fe5d806a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.864+0000 7fe5df278700 1 --2- 172.21.15.3:0/1020581669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d80d0890 0x7fe5d806a960 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80d1460 tx=0x7fe5d000b330 comp rx=0 tx=0).stop 2024-05-08T21:23:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.864+0000 7fe5df278700 1 -- 172.21.15.3:0/1020581669 shutdown_connections 2024-05-08T21:23:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.864+0000 7fe5df278700 1 --2- 172.21.15.3:0/1020581669 >> v2:172.21.15.186:3300/0 conn(0x7fe5d8145040 0x7fe5d80b5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.864+0000 7fe5df278700 1 --2- 172.21.15.3:0/1020581669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d80d0890 0x7fe5d806a960 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.864+0000 7fe5df278700 1 --2- 172.21.15.3:0/1020581669 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5d815fbb0 0x7fe5d80d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.864+0000 7fe5df278700 1 -- 172.21.15.3:0/1020581669 >> 172.21.15.3:0/1020581669 conn(0x7fe5d8099a50 msgr2=0x7fe5d80b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:20.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.864+0000 7fe5df278700 1 -- 172.21.15.3:0/1020581669 shutdown_connections 2024-05-08T21:23:20.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.864+0000 7fe5df278700 1 -- 172.21.15.3:0/1020581669 wait complete. 2024-05-08T21:23:20.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.865+0000 7fe5df278700 1 Processor -- start 2024-05-08T21:23:20.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.865+0000 7fe5df278700 1 -- start start 2024-05-08T21:23:20.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.866+0000 7fe5df278700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5d80d0890 0x7fe5d80cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:20.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.866+0000 7fe5df278700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d8145040 0x7fe5d815cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:20.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.866+0000 7fe5df278700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5d815fbb0 0x7fe5d814fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:20.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.866+0000 7fe5df278700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d815d9d0 con 0x7fe5d8145040 2024-05-08T21:23:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.866+0000 7fe5df278700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d815dcb0 con 0x7fe5d80d0890 2024-05-08T21:23:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.866+0000 7fe5df278700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d815df90 con 0x7fe5d815fbb0 2024-05-08T21:23:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.866+0000 7fe5dc813700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d8145040 0x7fe5d815cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.866+0000 7fe5dc813700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d8145040 0x7fe5d815cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57742/0 (socket says 172.21.15.3:57742) 2024-05-08T21:23:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.866+0000 7fe5dc813700 1 -- 172.21.15.3:0/1115470036 learned_addr learned my addr 172.21.15.3:0/1115470036 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.867+0000 7fe5dd014700 1 --2- 172.21.15.3:0/1115470036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5d80d0890 0x7fe5d80cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.867+0000 7fe5dd815700 1 --2- 172.21.15.3:0/1115470036 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5d815fbb0 0x7fe5d814fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.867+0000 7fe5dc813700 1 -- 172.21.15.3:0/1115470036 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5d815fbb0 msgr2=0x7fe5d814fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.867+0000 7fe5dc813700 1 --2- 172.21.15.3:0/1115470036 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5d815fbb0 0x7fe5d814fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.867+0000 7fe5dc813700 1 -- 172.21.15.3:0/1115470036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5d80d0890 msgr2=0x7fe5d80cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.867+0000 7fe5dc813700 1 --2- 172.21.15.3:0/1115470036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5d80d0890 0x7fe5d80cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.867+0000 7fe5dc813700 1 -- 172.21.15.3:0/1115470036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d0009b30 con 0x7fe5d8145040 2024-05-08T21:23:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.867+0000 7fe5dc813700 1 --2- 172.21.15.3:0/1115470036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d8145040 0x7fe5d815cfd0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe5c800d800 tx=0x7fe5c800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:20.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.867+0000 7fe5cdffb700 1 -- 172.21.15.3:0/1115470036 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c80125f0 con 0x7fe5d8145040 2024-05-08T21:23:20.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.867+0000 7fe5cdffb700 1 -- 172.21.15.3:0/1115470036 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c8012750 con 0x7fe5d8145040 2024-05-08T21:23:20.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.867+0000 7fe5df278700 1 -- 172.21.15.3:0/1115470036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d815e270 con 0x7fe5d8145040 2024-05-08T21:23:20.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.867+0000 7fe5cdffb700 1 -- 172.21.15.3:0/1115470036 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8012b00 con 0x7fe5d8145040 2024-05-08T21:23:20.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.867+0000 7fe5df278700 1 -- 172.21.15.3:0/1115470036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d815d510 con 0x7fe5d8145040 2024-05-08T21:23:20.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.868+0000 7fe5df278700 1 -- 172.21.15.3:0/1115470036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d815e400 con 0x7fe5d8145040 2024-05-08T21:23:20.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.869+0000 7fe5cdffb700 1 -- 172.21.15.3:0/1115470036 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe5c8012c60 con 0x7fe5d8145040 2024-05-08T21:23:20.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.869+0000 7fe5cdffb700 1 --2- 172.21.15.3:0/1115470036 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe5bc06ac90 0x7fe5bc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:20.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.869+0000 7fe5cdffb700 1 -- 172.21.15.3:0/1115470036 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7fe5c80898e0 con 0x7fe5d8145040 2024-05-08T21:23:20.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.870+0000 7fe5dd014700 1 --2- 172.21.15.3:0/1115470036 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe5bc06ac90 0x7fe5bc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:20.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.870+0000 7fe5dd014700 1 --2- 172.21.15.3:0/1115470036 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe5bc06ac90 0x7fe5bc06d150 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4006fd0 tx=0x7fe5d4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:20.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:20.874+0000 7fe5cdffb700 1 -- 172.21.15.3:0/1115470036 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe5c8054890 con 0x7fe5d8145040 2024-05-08T21:23:21.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.094+0000 7fe5df278700 1 -- 172.21.15.3:0/1115470036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe5d8074bc0 con 0x7fe5d8145040 2024-05-08T21:23:21.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.094+0000 7fe5cdffb700 1 -- 172.21.15.3:0/1115470036 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe5c8057eb0 con 0x7fe5d8145040 2024-05-08T21:23:21.096 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:23:21.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.097+0000 7fe5df278700 1 -- 172.21.15.3:0/1115470036 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe5bc06ac90 msgr2=0x7fe5bc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:21.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.097+0000 7fe5df278700 1 --2- 172.21.15.3:0/1115470036 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe5bc06ac90 0x7fe5bc06d150 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4006fd0 tx=0x7fe5d4008040 comp rx=0 tx=0).stop 2024-05-08T21:23:21.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.097+0000 7fe5df278700 1 -- 172.21.15.3:0/1115470036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d8145040 msgr2=0x7fe5d815cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:21.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.097+0000 7fe5df278700 1 --2- 172.21.15.3:0/1115470036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d8145040 0x7fe5d815cfd0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe5c800d800 tx=0x7fe5c800dbc0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.097+0000 7fe5df278700 1 -- 172.21.15.3:0/1115470036 shutdown_connections 2024-05-08T21:23:21.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.097+0000 7fe5df278700 1 --2- 172.21.15.3:0/1115470036 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5d815fbb0 0x7fe5d814fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.097+0000 7fe5df278700 1 --2- 172.21.15.3:0/1115470036 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe5bc06ac90 0x7fe5bc06d150 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.097+0000 7fe5df278700 1 --2- 172.21.15.3:0/1115470036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5d80d0890 0x7fe5d80cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.097+0000 7fe5df278700 1 --2- 172.21.15.3:0/1115470036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d8145040 0x7fe5d815cfd0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.097+0000 7fe5df278700 1 -- 172.21.15.3:0/1115470036 >> 172.21.15.3:0/1115470036 conn(0x7fe5d8099a50 msgr2=0x7fe5d8142e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:21.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.097+0000 7fe5df278700 1 -- 172.21.15.3:0/1115470036 shutdown_connections 2024-05-08T21:23:21.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.097+0000 7fe5df278700 1 -- 172.21.15.3:0/1115470036 wait complete. 2024-05-08T21:23:21.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-08T21:23:21.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.249+0000 7f4a38706700 1 Processor -- start 2024-05-08T21:23:21.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.249+0000 7f4a38706700 1 -- start start 2024-05-08T21:23:21.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.250+0000 7f4a38706700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4a300cbf50 0x7f4a30161d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:21.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.250+0000 7f4a38706700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a30162240 0x7f4a3015e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:21.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.250+0000 7f4a38706700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a300cf970 con 0x7f4a300d0f10 2024-05-08T21:23:21.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.250+0000 7f4a38706700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a300cfc50 con 0x7f4a300cbf50 2024-05-08T21:23:21.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.250+0000 7f4a38706700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a300cff30 con 0x7f4a30162240 2024-05-08T21:23:21.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.251+0000 7f4a364a2700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4a300cbf50 0x7f4a30161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:21.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.251+0000 7f4a35ca1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a30162240 0x7f4a3015e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:21.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.251+0000 7f4a35ca1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a30162240 0x7f4a3015e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.3:56534/0 (socket says 172.21.15.3:56534) 2024-05-08T21:23:21.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.251+0000 7f4a364a2700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4a300cbf50 0x7f4a30161d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:59396/0 (socket says 172.21.15.3:59396) 2024-05-08T21:23:21.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.251+0000 7f4a35ca1700 1 -- 172.21.15.3:0/4054308063 learned_addr learned my addr 172.21.15.3:0/4054308063 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:21.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.251+0000 7f4a35ca1700 1 -- 172.21.15.3:0/4054308063 >> v2:172.21.15.186:3300/0 conn(0x7f4a300cbf50 msgr2=0x7f4a30161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:21.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.251+0000 7f4a35ca1700 1 --2- 172.21.15.3:0/4054308063 >> v2:172.21.15.186:3300/0 conn(0x7f4a300cbf50 0x7f4a30161d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.251+0000 7f4a35ca1700 1 -- 172.21.15.3:0/4054308063 >> v1:172.21.15.186:6789/0 conn(0x7f4a300d0f10 legacy=0x7f4a3006a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:21.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.251+0000 7f4a35ca1700 1 -- 172.21.15.3:0/4054308063 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a301617e0 con 0x7f4a30162240 2024-05-08T21:23:21.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.251+0000 7f4a364a2700 1 --2- 172.21.15.3:0/4054308063 >> v2:172.21.15.186:3300/0 conn(0x7f4a300cbf50 0x7f4a30161d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:21.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.251+0000 7f4a35ca1700 1 --2- 172.21.15.3:0/4054308063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a30162240 0x7f4a3015e050 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4a200098f0 tx=0x7f4a20009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c8b0ad146bb6886 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:21.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.251+0000 7f4a354a0700 1 -- 172.21.15.3:0/4054308063 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a20011830 con 0x7f4a30162240 2024-05-08T21:23:21.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.251+0000 7f4a354a0700 1 -- 172.21.15.3:0/4054308063 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a20011990 con 0x7f4a30162240 2024-05-08T21:23:21.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.251+0000 7f4a354a0700 1 -- 172.21.15.3:0/4054308063 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a20011d20 con 0x7f4a30162240 2024-05-08T21:23:21.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.252+0000 7f4a38706700 1 -- 172.21.15.3:0/4054308063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a30162240 msgr2=0x7f4a3015e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:21.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.252+0000 7f4a38706700 1 --2- 172.21.15.3:0/4054308063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a30162240 0x7f4a3015e050 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4a200098f0 tx=0x7f4a20009c00 comp rx=0 tx=0).stop 2024-05-08T21:23:21.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.252+0000 7f4a38706700 1 -- 172.21.15.3:0/4054308063 shutdown_connections 2024-05-08T21:23:21.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.252+0000 7f4a38706700 1 --2- 172.21.15.3:0/4054308063 >> v2:172.21.15.186:3300/0 conn(0x7f4a300cbf50 0x7f4a30161d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.252+0000 7f4a38706700 1 --2- 172.21.15.3:0/4054308063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a30162240 0x7f4a3015e050 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.252+0000 7f4a38706700 1 -- 172.21.15.3:0/4054308063 >> 172.21.15.3:0/4054308063 conn(0x7f4a30099a50 msgr2=0x7f4a300d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:21.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.252+0000 7f4a38706700 1 -- 172.21.15.3:0/4054308063 shutdown_connections 2024-05-08T21:23:21.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.252+0000 7f4a38706700 1 -- 172.21.15.3:0/4054308063 wait complete. 2024-05-08T21:23:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.253+0000 7f4a38706700 1 Processor -- start 2024-05-08T21:23:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.254+0000 7f4a38706700 1 -- start start 2024-05-08T21:23:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.254+0000 7f4a38706700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a300cbf50 0x7f4a300b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.254+0000 7f4a38706700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a300d0f10 0x7f4a300b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.254+0000 7f4a38706700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a30162240 0x7f4a300b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.254+0000 7f4a38706700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a300b4660 con 0x7f4a300d0f10 2024-05-08T21:23:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.254+0000 7f4a38706700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a300b5630 con 0x7f4a30162240 2024-05-08T21:23:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.254+0000 7f4a38706700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a300b5910 con 0x7f4a300cbf50 2024-05-08T21:23:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.255+0000 7f4a35ca1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a300d0f10 0x7f4a300b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.255+0000 7f4a35ca1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a300d0f10 0x7f4a300b4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57746/0 (socket says 172.21.15.3:57746) 2024-05-08T21:23:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.255+0000 7f4a35ca1700 1 -- 172.21.15.3:0/330223298 learned_addr learned my addr 172.21.15.3:0/330223298 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.255+0000 7f4a364a2700 1 --2- 172.21.15.3:0/330223298 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a300cbf50 0x7f4a300b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.255+0000 7f4a36ca3700 1 --2- 172.21.15.3:0/330223298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a30162240 0x7f4a300b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:21.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.255+0000 7f4a35ca1700 1 -- 172.21.15.3:0/330223298 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a300cbf50 msgr2=0x7f4a300b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:21.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.255+0000 7f4a35ca1700 1 --2- 172.21.15.3:0/330223298 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a300cbf50 0x7f4a300b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.255+0000 7f4a35ca1700 1 -- 172.21.15.3:0/330223298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a30162240 msgr2=0x7f4a300b4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:21.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.255+0000 7f4a35ca1700 1 --2- 172.21.15.3:0/330223298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a30162240 0x7f4a300b4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.255+0000 7f4a35ca1700 1 -- 172.21.15.3:0/330223298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c0094a0 con 0x7f4a300d0f10 2024-05-08T21:23:21.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.255+0000 7f4a35ca1700 1 --2- 172.21.15.3:0/330223298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a300d0f10 0x7f4a300b4a50 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4a20009c40 tx=0x7f4a20004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:21.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.255+0000 7f4a26ffd700 1 -- 172.21.15.3:0/330223298 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a200036a0 con 0x7f4a300d0f10 2024-05-08T21:23:21.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.255+0000 7f4a26ffd700 1 -- 172.21.15.3:0/330223298 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2001e070 con 0x7f4a300d0f10 2024-05-08T21:23:21.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.255+0000 7f4a38706700 1 -- 172.21.15.3:0/330223298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a20009550 con 0x7f4a300d0f10 2024-05-08T21:23:21.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.255+0000 7f4a26ffd700 1 -- 172.21.15.3:0/330223298 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a20003a40 con 0x7f4a300d0f10 2024-05-08T21:23:21.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.255+0000 7f4a38706700 1 -- 172.21.15.3:0/330223298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3015dac0 con 0x7f4a300d0f10 2024-05-08T21:23:21.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.256+0000 7f4a38706700 1 -- 172.21.15.3:0/330223298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a30074bc0 con 0x7f4a300d0f10 2024-05-08T21:23:21.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.257+0000 7f4a26ffd700 1 -- 172.21.15.3:0/330223298 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4a20003ba0 con 0x7f4a300d0f10 2024-05-08T21:23:21.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.257+0000 7f4a26ffd700 1 --2- 172.21.15.3:0/330223298 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a1406ac90 0x7f4a1406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:21.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.257+0000 7f4a26ffd700 1 -- 172.21.15.3:0/330223298 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f4a2008a4d0 con 0x7f4a300d0f10 2024-05-08T21:23:21.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.258+0000 7f4a364a2700 1 --2- 172.21.15.3:0/330223298 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a1406ac90 0x7f4a1406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:21.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.258+0000 7f4a364a2700 1 --2- 172.21.15.3:0/330223298 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a1406ac90 0x7f4a1406d150 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c009d00 tx=0x7f4a2c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:21.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.262+0000 7f4a26ffd700 1 -- 172.21.15.3:0/330223298 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f4a20055520 con 0x7f4a300d0f10 2024-05-08T21:23:21.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.478+0000 7f4a38706700 1 -- 172.21.15.3:0/330223298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4a300b62b0 con 0x7f4a300d0f10 2024-05-08T21:23:21.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.479+0000 7f4a26ffd700 1 -- 172.21.15.3:0/330223298 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4a20058b40 con 0x7f4a300d0f10 2024-05-08T21:23:21.480 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:23:21.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.481+0000 7f4a38706700 1 -- 172.21.15.3:0/330223298 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a1406ac90 msgr2=0x7f4a1406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:21.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.481+0000 7f4a38706700 1 --2- 172.21.15.3:0/330223298 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a1406ac90 0x7f4a1406d150 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c009d00 tx=0x7f4a2c008040 comp rx=0 tx=0).stop 2024-05-08T21:23:21.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.481+0000 7f4a38706700 1 -- 172.21.15.3:0/330223298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a300d0f10 msgr2=0x7f4a300b4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:21.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.481+0000 7f4a38706700 1 --2- 172.21.15.3:0/330223298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a300d0f10 0x7f4a300b4a50 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4a20009c40 tx=0x7f4a20004750 comp rx=0 tx=0).stop 2024-05-08T21:23:21.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.481+0000 7f4a38706700 1 -- 172.21.15.3:0/330223298 shutdown_connections 2024-05-08T21:23:21.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.481+0000 7f4a38706700 1 --2- 172.21.15.3:0/330223298 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a300cbf50 0x7f4a300b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.481+0000 7f4a38706700 1 --2- 172.21.15.3:0/330223298 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a1406ac90 0x7f4a1406d150 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.481+0000 7f4a38706700 1 --2- 172.21.15.3:0/330223298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a30162240 0x7f4a300b4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.481+0000 7f4a38706700 1 --2- 172.21.15.3:0/330223298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a300d0f10 0x7f4a300b4a50 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.481+0000 7f4a38706700 1 -- 172.21.15.3:0/330223298 >> 172.21.15.3:0/330223298 conn(0x7f4a30099a50 msgr2=0x7f4a3015c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:21.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.481+0000 7f4a38706700 1 -- 172.21.15.3:0/330223298 shutdown_connections 2024-05-08T21:23:21.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.481+0000 7f4a38706700 1 -- 172.21.15.3:0/330223298 wait complete. 2024-05-08T21:23:21.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:23:21.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.635+0000 7effab9fb700 1 Processor -- start 2024-05-08T21:23:21.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.635+0000 7effab9fb700 1 -- start start 2024-05-08T21:23:21.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.636+0000 7effab9fb700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7effa406bd60 0x7effa40cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:21.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.636+0000 7effab9fb700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7effa40cdc90 0x7effa40b19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:21.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.636+0000 7effab9fb700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa40b1200 con 0x7effa40d13a0 2024-05-08T21:23:21.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.636+0000 7effab9fb700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa40cfcc0 con 0x7effa40cdc90 2024-05-08T21:23:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.636+0000 7effab9fb700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa40ce5c0 con 0x7effa406bd60 2024-05-08T21:23:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.636+0000 7effa9f98700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7effa40cdc90 0x7effa40b19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.636+0000 7effa9797700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7effa40d13a0 0x7effa40d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50436/0 (socket says 172.21.15.3:50436) 2024-05-08T21:23:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.636+0000 7effa9f98700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7effa40cdc90 0x7effa40b19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41912/0 (socket says 172.21.15.3:41912) 2024-05-08T21:23:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.636+0000 7effa9797700 1 -- 172.21.15.3:0/2357270588 learned_addr learned my addr 172.21.15.3:0/2357270588 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.636+0000 7effa8f96700 1 --2- 172.21.15.3:0/2357270588 >> v2:172.21.15.186:3300/0 conn(0x7effa406bd60 0x7effa40cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.637+0000 7effa9f98700 1 -- 172.21.15.3:0/2357270588 >> v2:172.21.15.186:3300/0 conn(0x7effa406bd60 msgr2=0x7effa40cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.637+0000 7effa9f98700 1 --2- 172.21.15.3:0/2357270588 >> v2:172.21.15.186:3300/0 conn(0x7effa406bd60 0x7effa40cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.637+0000 7effa9f98700 1 -- 172.21.15.3:0/2357270588 >> v1:172.21.15.3:6789/0 conn(0x7effa40d13a0 legacy=0x7effa40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.637+0000 7effa9f98700 1 -- 172.21.15.3:0/2357270588 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa40ce8a0 con 0x7effa40cdc90 2024-05-08T21:23:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.637+0000 7eff9bfff700 1 -- 172.21.15.3:0/2357270588 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728749906 0 0) 0x7effa40b1200 con 0x7effa40d13a0 2024-05-08T21:23:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.637+0000 7effa9f98700 1 --2- 172.21.15.3:0/2357270588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effa40cdc90 0x7effa40b19c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7effa40b0f90 tx=0x7eff9c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2794f24a290a934a server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.637+0000 7eff9bfff700 1 -- 172.21.15.3:0/2357270588 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c00e070 con 0x7effa40cdc90 2024-05-08T21:23:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.637+0000 7eff9bfff700 1 -- 172.21.15.3:0/2357270588 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9c00bb20 con 0x7effa40cdc90 2024-05-08T21:23:21.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.637+0000 7eff9bfff700 1 -- 172.21.15.3:0/2357270588 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c012430 con 0x7effa40cdc90 2024-05-08T21:23:21.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.638+0000 7effab9fb700 1 -- 172.21.15.3:0/2357270588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effa40cdc90 msgr2=0x7effa40b19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:21.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.638+0000 7effab9fb700 1 --2- 172.21.15.3:0/2357270588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effa40cdc90 0x7effa40b19c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7effa40b0f90 tx=0x7eff9c00b330 comp rx=0 tx=0).stop 2024-05-08T21:23:21.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.638+0000 7effab9fb700 1 -- 172.21.15.3:0/2357270588 shutdown_connections 2024-05-08T21:23:21.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.638+0000 7effab9fb700 1 --2- 172.21.15.3:0/2357270588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effa40cdc90 0x7effa40b19c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.638+0000 7effab9fb700 1 --2- 172.21.15.3:0/2357270588 >> v2:172.21.15.186:3300/0 conn(0x7effa406bd60 0x7effa40cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.638+0000 7effab9fb700 1 -- 172.21.15.3:0/2357270588 >> 172.21.15.3:0/2357270588 conn(0x7effa408afe0 msgr2=0x7effa408b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:21.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.638+0000 7effab9fb700 1 -- 172.21.15.3:0/2357270588 shutdown_connections 2024-05-08T21:23:21.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.638+0000 7effab9fb700 1 -- 172.21.15.3:0/2357270588 wait complete. 2024-05-08T21:23:21.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.639+0000 7effab9fb700 1 Processor -- start 2024-05-08T21:23:21.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.639+0000 7effab9fb700 1 -- start start 2024-05-08T21:23:21.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.640+0000 7effab9fb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effa406bd60 0x7effa4151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:21.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.640+0000 7effab9fb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effa40cdc90 0x7effa409dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:21.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.640+0000 7effab9fb700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effa40d13a0 0x7effa40cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:21.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.640+0000 7effab9fb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa4151e60 con 0x7effa40cdc90 2024-05-08T21:23:21.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.640+0000 7effab9fb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa409e4c0 con 0x7effa406bd60 2024-05-08T21:23:21.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.640+0000 7effab9fb700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa409e7a0 con 0x7effa40d13a0 2024-05-08T21:23:21.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.640+0000 7effa8f96700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effa40cdc90 0x7effa409dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:21.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.640+0000 7effa8f96700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effa40cdc90 0x7effa409dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41918/0 (socket says 172.21.15.3:41918) 2024-05-08T21:23:21.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.640+0000 7effa8f96700 1 -- 172.21.15.3:0/1889354965 learned_addr learned my addr 172.21.15.3:0/1889354965 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:21.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.640+0000 7effa9f98700 1 --2- 172.21.15.3:0/1889354965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effa40d13a0 0x7effa40cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:21.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.640+0000 7effa9797700 1 --2- 172.21.15.3:0/1889354965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effa406bd60 0x7effa4151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:21.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.640+0000 7effa8f96700 1 -- 172.21.15.3:0/1889354965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effa40d13a0 msgr2=0x7effa40cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:21.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.640+0000 7effa8f96700 1 --2- 172.21.15.3:0/1889354965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effa40d13a0 0x7effa40cc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.640+0000 7effa8f96700 1 -- 172.21.15.3:0/1889354965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effa406bd60 msgr2=0x7effa4151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:21.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.640+0000 7effa8f96700 1 --2- 172.21.15.3:0/1889354965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effa406bd60 0x7effa4151920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.640+0000 7effa8f96700 1 -- 172.21.15.3:0/1889354965 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9c009a70 con 0x7effa40cdc90 2024-05-08T21:23:21.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.641+0000 7effa8f96700 1 --2- 172.21.15.3:0/1889354965 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effa40cdc90 0x7effa409dac0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7eff9400e870 tx=0x7eff9400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:21.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.641+0000 7eff99ffb700 1 -- 172.21.15.3:0/1889354965 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff94009590 con 0x7effa40cdc90 2024-05-08T21:23:21.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.641+0000 7eff99ffb700 1 -- 172.21.15.3:0/1889354965 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff94013680 con 0x7effa40cdc90 2024-05-08T21:23:21.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.641+0000 7eff99ffb700 1 -- 172.21.15.3:0/1889354965 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff940139f0 con 0x7effa40cdc90 2024-05-08T21:23:21.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.641+0000 7effab9fb700 1 -- 172.21.15.3:0/1889354965 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa409ea80 con 0x7effa40cdc90 2024-05-08T21:23:21.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.641+0000 7effab9fb700 1 -- 172.21.15.3:0/1889354965 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa409e000 con 0x7effa40cdc90 2024-05-08T21:23:21.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.642+0000 7effab9fb700 1 -- 172.21.15.3:0/1889354965 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa409e280 con 0x7effa40cdc90 2024-05-08T21:23:21.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.643+0000 7eff99ffb700 1 -- 172.21.15.3:0/1889354965 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7eff94013b50 con 0x7effa40cdc90 2024-05-08T21:23:21.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.643+0000 7eff99ffb700 1 --2- 172.21.15.3:0/1889354965 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7eff8806ac90 0x7eff8806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:21.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.643+0000 7eff99ffb700 1 -- 172.21.15.3:0/1889354965 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7eff940894c0 con 0x7effa40cdc90 2024-05-08T21:23:21.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.643+0000 7effa9797700 1 --2- 172.21.15.3:0/1889354965 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7eff8806ac90 0x7eff8806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:21.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.644+0000 7effa9797700 1 --2- 172.21.15.3:0/1889354965 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7eff8806ac90 0x7eff8806d150 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7effa0007fd0 tx=0x7effa0007de0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:21.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.648+0000 7eff99ffb700 1 -- 172.21.15.3:0/1889354965 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7effa409e280 con 0x7effa40cdc90 2024-05-08T21:23:21.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.868+0000 7effab9fb700 1 -- 172.21.15.3:0/1889354965 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7effa409e280 con 0x7effa40cdc90 2024-05-08T21:23:21.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.869+0000 7eff99ffb700 1 -- 172.21.15.3:0/1889354965 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+11 (secure 0 0 0) 0x7effa409e280 con 0x7effa40cdc90 2024-05-08T21:23:21.870 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:23:21.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.871+0000 7effab9fb700 1 -- 172.21.15.3:0/1889354965 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7eff8806ac90 msgr2=0x7eff8806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:21.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.871+0000 7effab9fb700 1 --2- 172.21.15.3:0/1889354965 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7eff8806ac90 0x7eff8806d150 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7effa0007fd0 tx=0x7effa0007de0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.871+0000 7effab9fb700 1 -- 172.21.15.3:0/1889354965 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effa40cdc90 msgr2=0x7effa409dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:21.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.871+0000 7effab9fb700 1 --2- 172.21.15.3:0/1889354965 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effa40cdc90 0x7effa409dac0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7eff9400e870 tx=0x7eff9400eb80 comp rx=0 tx=0).stop 2024-05-08T21:23:21.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.871+0000 7effab9fb700 1 -- 172.21.15.3:0/1889354965 shutdown_connections 2024-05-08T21:23:21.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.871+0000 7effab9fb700 1 --2- 172.21.15.3:0/1889354965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effa40d13a0 0x7effa40cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.871+0000 7effab9fb700 1 --2- 172.21.15.3:0/1889354965 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7eff8806ac90 0x7eff8806d150 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.871+0000 7effab9fb700 1 --2- 172.21.15.3:0/1889354965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effa406bd60 0x7effa4151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.871+0000 7effab9fb700 1 --2- 172.21.15.3:0/1889354965 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effa40cdc90 0x7effa409dac0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:21.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.871+0000 7effab9fb700 1 -- 172.21.15.3:0/1889354965 >> 172.21.15.3:0/1889354965 conn(0x7effa408afe0 msgr2=0x7effa40b0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:21.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.871+0000 7effab9fb700 1 -- 172.21.15.3:0/1889354965 shutdown_connections 2024-05-08T21:23:21.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:21.871+0000 7effab9fb700 1 -- 172.21.15.3:0/1889354965 wait complete. 2024-05-08T21:23:21.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get recovery_fs_metadata pg_num 2024-05-08T21:23:22.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.026+0000 7f2a5e26f700 1 Processor -- start 2024-05-08T21:23:22.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.026+0000 7f2a5e26f700 1 -- start start 2024-05-08T21:23:22.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.026+0000 7f2a5e26f700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2a580b0d80 0x7f2a580b3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.026+0000 7f2a5e26f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a580b4d50 0x7f2a5806a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.026+0000 7f2a5e26f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2a5806aea0 0x7f2a580d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.026+0000 7f2a5e26f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a5806b330 con 0x7f2a5806aea0 2024-05-08T21:23:22.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a5e26f700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a58145040 con 0x7f2a580b0d80 2024-05-08T21:23:22.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a5e26f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a580b38c0 con 0x7f2a580b4d50 2024-05-08T21:23:22.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a5c80c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2a5806aea0 0x7f2a580d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a5c80c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2a5806aea0 0x7f2a580d0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41920/0 (socket says 172.21.15.3:41920) 2024-05-08T21:23:22.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a5c80c700 1 -- 172.21.15.3:0/1046688039 learned_addr learned my addr 172.21.15.3:0/1046688039 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:22.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a57fff700 1 --2- 172.21.15.3:0/1046688039 >> v2:172.21.15.186:3300/0 conn(0x7f2a580b0d80 0x7f2a580b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a577fe700 1 --2- 172.21.15.3:0/1046688039 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a580b4d50 0x7f2a5806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a5c80c700 1 -- 172.21.15.3:0/1046688039 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a580b4d50 msgr2=0x7f2a5806a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:22.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a5c80c700 1 --2- 172.21.15.3:0/1046688039 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a580b4d50 0x7f2a5806a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a5c80c700 1 -- 172.21.15.3:0/1046688039 >> v2:172.21.15.186:3300/0 conn(0x7f2a580b0d80 msgr2=0x7f2a580b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:22.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a5c80c700 1 --2- 172.21.15.3:0/1046688039 >> v2:172.21.15.186:3300/0 conn(0x7f2a580b0d80 0x7f2a580b3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a5c80c700 1 -- 172.21.15.3:0/1046688039 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a580b16e0 con 0x7f2a5806aea0 2024-05-08T21:23:22.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a577fe700 1 --2- 172.21.15.3:0/1046688039 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a580b4d50 0x7f2a5806a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:22.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a5c80c700 1 --2- 172.21.15.3:0/1046688039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a5806aea0 0x7f2a580d0350 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2a5809a0a0 tx=0x7f2a5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de0f0ed3e50d5c server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:22.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a56ffd700 1 -- 172.21.15.3:0/1046688039 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5000e070 con 0x7f2a5806aea0 2024-05-08T21:23:22.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a56ffd700 1 -- 172.21.15.3:0/1046688039 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a5000b910 con 0x7f2a5806aea0 2024-05-08T21:23:22.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.027+0000 7f2a56ffd700 1 -- 172.21.15.3:0/1046688039 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5000bcc0 con 0x7f2a5806aea0 2024-05-08T21:23:22.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.028+0000 7f2a5e26f700 1 -- 172.21.15.3:0/1046688039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a5806aea0 msgr2=0x7f2a580d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:22.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.028+0000 7f2a5e26f700 1 --2- 172.21.15.3:0/1046688039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a5806aea0 0x7f2a580d0350 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2a5809a0a0 tx=0x7f2a5000b330 comp rx=0 tx=0).stop 2024-05-08T21:23:22.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.028+0000 7f2a5e26f700 1 -- 172.21.15.3:0/1046688039 shutdown_connections 2024-05-08T21:23:22.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.028+0000 7f2a5e26f700 1 --2- 172.21.15.3:0/1046688039 >> v2:172.21.15.186:3300/0 conn(0x7f2a580b0d80 0x7f2a580b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.028+0000 7f2a5e26f700 1 --2- 172.21.15.3:0/1046688039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a5806aea0 0x7f2a580d0350 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.028+0000 7f2a5e26f700 1 --2- 172.21.15.3:0/1046688039 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a580b4d50 0x7f2a5806a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.028+0000 7f2a5e26f700 1 -- 172.21.15.3:0/1046688039 >> 172.21.15.3:0/1046688039 conn(0x7f2a58099a50 msgr2=0x7f2a580b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:22.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.028+0000 7f2a5e26f700 1 -- 172.21.15.3:0/1046688039 shutdown_connections 2024-05-08T21:23:22.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.028+0000 7f2a5e26f700 1 -- 172.21.15.3:0/1046688039 wait complete. 2024-05-08T21:23:22.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.029+0000 7f2a5e26f700 1 Processor -- start 2024-05-08T21:23:22.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.030+0000 7f2a5e26f700 1 -- start start 2024-05-08T21:23:22.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.030+0000 7f2a5e26f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a5806aea0 0x7f2a5813b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.030+0000 7f2a5e26f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a580b0d80 0x7f2a5815fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.030+0000 7f2a5e26f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a580b4d50 0x7f2a5814fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.030+0000 7f2a5e26f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5813a8e0 con 0x7f2a5806aea0 2024-05-08T21:23:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.030+0000 7f2a5e26f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5813bee0 con 0x7f2a580b0d80 2024-05-08T21:23:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.030+0000 7f2a5e26f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5813b510 con 0x7f2a580b4d50 2024-05-08T21:23:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a57fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a5806aea0 0x7f2a5813b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a57fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a5806aea0 0x7f2a5813b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41932/0 (socket says 172.21.15.3:41932) 2024-05-08T21:23:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a57fff700 1 -- 172.21.15.3:0/886160366 learned_addr learned my addr 172.21.15.3:0/886160366 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a577fe700 1 --2- 172.21.15.3:0/886160366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a580b0d80 0x7f2a5815fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a5c80c700 1 --2- 172.21.15.3:0/886160366 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a580b4d50 0x7f2a5814fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a57fff700 1 -- 172.21.15.3:0/886160366 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a580b4d50 msgr2=0x7f2a5814fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a57fff700 1 --2- 172.21.15.3:0/886160366 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a580b4d50 0x7f2a5814fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a57fff700 1 -- 172.21.15.3:0/886160366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a580b0d80 msgr2=0x7f2a5815fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a57fff700 1 --2- 172.21.15.3:0/886160366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a580b0d80 0x7f2a5815fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a57fff700 1 -- 172.21.15.3:0/886160366 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a50009b30 con 0x7f2a5806aea0 2024-05-08T21:23:22.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a577fe700 1 --2- 172.21.15.3:0/886160366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a580b0d80 0x7f2a5815fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:22.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a5c80c700 1 --2- 172.21.15.3:0/886160366 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a580b4d50 0x7f2a5814fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:22.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a57fff700 1 --2- 172.21.15.3:0/886160366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a5806aea0 0x7f2a5813b9a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2a4800e960 tx=0x7f2a4800ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:22.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a54ff9700 1 -- 172.21.15.3:0/886160366 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a48009690 con 0x7f2a5806aea0 2024-05-08T21:23:22.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a54ff9700 1 -- 172.21.15.3:0/886160366 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4800c850 con 0x7f2a5806aea0 2024-05-08T21:23:22.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a5e26f700 1 -- 172.21.15.3:0/886160366 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a5815d1b0 con 0x7f2a5806aea0 2024-05-08T21:23:22.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a54ff9700 1 -- 172.21.15.3:0/886160366 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4800cbc0 con 0x7f2a5806aea0 2024-05-08T21:23:22.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.031+0000 7f2a5e26f700 1 -- 172.21.15.3:0/886160366 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a58150d20 con 0x7f2a5806aea0 2024-05-08T21:23:22.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.032+0000 7f2a5e26f700 1 -- 172.21.15.3:0/886160366 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a5813b7f0 con 0x7f2a5806aea0 2024-05-08T21:23:22.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.033+0000 7f2a54ff9700 1 -- 172.21.15.3:0/886160366 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2a4800cd20 con 0x7f2a5806aea0 2024-05-08T21:23:22.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.033+0000 7f2a54ff9700 1 --2- 172.21.15.3:0/886160366 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2a3806ac90 0x7f2a3806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.033+0000 7f2a54ff9700 1 -- 172.21.15.3:0/886160366 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f2a480895e0 con 0x7f2a5806aea0 2024-05-08T21:23:22.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.034+0000 7f2a577fe700 1 --2- 172.21.15.3:0/886160366 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2a3806ac90 0x7f2a3806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.034+0000 7f2a577fe700 1 --2- 172.21.15.3:0/886160366 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2a3806ac90 0x7f2a3806d150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2a58154550 tx=0x7f2a40008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:22.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.038+0000 7f2a54ff9700 1 -- 172.21.15.3:0/886160366 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2a480545a0 con 0x7f2a5806aea0 2024-05-08T21:23:22.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.256+0000 7f2a5e26f700 1 -- 172.21.15.3:0/886160366 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2a58074bc0 con 0x7f2a5806aea0 2024-05-08T21:23:22.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.257+0000 7f2a54ff9700 1 -- 172.21.15.3:0/886160366 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 109+0+11 (secure 0 0 0) 0x7f2a48057bc0 con 0x7f2a5806aea0 2024-05-08T21:23:22.258 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:23:22.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.259+0000 7f2a5e26f700 1 -- 172.21.15.3:0/886160366 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2a3806ac90 msgr2=0x7f2a3806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:22.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.259+0000 7f2a5e26f700 1 --2- 172.21.15.3:0/886160366 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2a3806ac90 0x7f2a3806d150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2a58154550 tx=0x7f2a40008040 comp rx=0 tx=0).stop 2024-05-08T21:23:22.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.259+0000 7f2a5e26f700 1 -- 172.21.15.3:0/886160366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a5806aea0 msgr2=0x7f2a5813b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:22.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.259+0000 7f2a5e26f700 1 --2- 172.21.15.3:0/886160366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a5806aea0 0x7f2a5813b9a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2a4800e960 tx=0x7f2a4800ed20 comp rx=0 tx=0).stop 2024-05-08T21:23:22.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.259+0000 7f2a5e26f700 1 -- 172.21.15.3:0/886160366 shutdown_connections 2024-05-08T21:23:22.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.259+0000 7f2a5e26f700 1 --2- 172.21.15.3:0/886160366 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a580b4d50 0x7f2a5814fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.259+0000 7f2a5e26f700 1 --2- 172.21.15.3:0/886160366 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2a3806ac90 0x7f2a3806d150 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.259+0000 7f2a5e26f700 1 --2- 172.21.15.3:0/886160366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a5806aea0 0x7f2a5813b9a0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.259+0000 7f2a5e26f700 1 --2- 172.21.15.3:0/886160366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a580b0d80 0x7f2a5815fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.259+0000 7f2a5e26f700 1 -- 172.21.15.3:0/886160366 >> 172.21.15.3:0/886160366 conn(0x7f2a58099a50 msgr2=0x7f2a581460b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:22.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.259+0000 7f2a5e26f700 1 -- 172.21.15.3:0/886160366 shutdown_connections 2024-05-08T21:23:22.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.259+0000 7f2a5e26f700 1 -- 172.21.15.3:0/886160366 wait complete. 2024-05-08T21:23:22.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-08T21:23:22.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.409+0000 7f08a7c86700 1 Processor -- start 2024-05-08T21:23:22.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.410+0000 7f08a7c86700 1 -- start start 2024-05-08T21:23:22.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.410+0000 7f08a7c86700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f08a00cc060 0x7f08a00cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.411+0000 7f08a7c86700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a00cd7c0 con 0x7f08a00cfde0 2024-05-08T21:23:22.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.411+0000 7f08a7c86700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a00cce90 con 0x7f08a006b070 2024-05-08T21:23:22.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.411+0000 7f08a7c86700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a00ccc80 con 0x7f08a00cc060 2024-05-08T21:23:22.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.411+0000 7f08a5221700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f08a00cfde0 0x7f08a006a960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50438/0 (socket says 172.21.15.3:50438) 2024-05-08T21:23:22.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.411+0000 7f08a5221700 1 -- 172.21.15.3:0/2624975045 learned_addr learned my addr 172.21.15.3:0/2624975045 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:22.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.411+0000 7f08a5a22700 1 --2- 172.21.15.3:0/2624975045 >> v2:172.21.15.186:3300/0 conn(0x7f08a00cc060 0x7f08a00cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.411+0000 7f08a4a20700 1 -- 172.21.15.3:0/2624975045 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713809798 0 0) 0x7f08a00cd7c0 con 0x7f08a00cfde0 2024-05-08T21:23:22.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.411+0000 7f08a4a20700 1 -- 172.21.15.3:0/2624975045 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f088c0035b0 con 0x7f08a00cfde0 2024-05-08T21:23:22.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.411+0000 7f08a5a22700 1 -- 172.21.15.3:0/2624975045 >> v1:172.21.15.186:6789/0 conn(0x7f08a006b070 legacy=0x7f08a00ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:22.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.411+0000 7f08a5a22700 1 -- 172.21.15.3:0/2624975045 >> v1:172.21.15.3:6789/0 conn(0x7f08a00cfde0 legacy=0x7f08a006a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:22.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.411+0000 7f08a5a22700 1 -- 172.21.15.3:0/2624975045 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a00cd170 con 0x7f08a00cc060 2024-05-08T21:23:22.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.411+0000 7f08a5a22700 1 --2- 172.21.15.3:0/2624975045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08a00cc060 0x7f08a00cf8a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f089c009840 tx=0x7f089c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e393ea6579b76d6a server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:22.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.412+0000 7f08a4a20700 1 -- 172.21.15.3:0/2624975045 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c011830 con 0x7f08a00cc060 2024-05-08T21:23:22.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.412+0000 7f08a4a20700 1 -- 172.21.15.3:0/2624975045 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089c011990 con 0x7f08a00cc060 2024-05-08T21:23:22.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.412+0000 7f08a4a20700 1 -- 172.21.15.3:0/2624975045 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c011d20 con 0x7f08a00cc060 2024-05-08T21:23:22.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.412+0000 7f08a7c86700 1 -- 172.21.15.3:0/2624975045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08a00cc060 msgr2=0x7f08a00cf8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:22.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.412+0000 7f08a7c86700 1 --2- 172.21.15.3:0/2624975045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08a00cc060 0x7f08a00cf8a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f089c009840 tx=0x7f089c009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:22.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.412+0000 7f08a7c86700 1 -- 172.21.15.3:0/2624975045 shutdown_connections 2024-05-08T21:23:22.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.412+0000 7f08a7c86700 1 --2- 172.21.15.3:0/2624975045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08a00cc060 0x7f08a00cf8a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.412+0000 7f08a7c86700 1 -- 172.21.15.3:0/2624975045 >> 172.21.15.3:0/2624975045 conn(0x7f08a0099a50 msgr2=0x7f08a00d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:22.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.412+0000 7f08a7c86700 1 -- 172.21.15.3:0/2624975045 shutdown_connections 2024-05-08T21:23:22.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.412+0000 7f08a7c86700 1 -- 172.21.15.3:0/2624975045 wait complete. 2024-05-08T21:23:22.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.414+0000 7f08a7c86700 1 Processor -- start 2024-05-08T21:23:22.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.414+0000 7f08a7c86700 1 -- start start 2024-05-08T21:23:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.414+0000 7f08a7c86700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08a006b070 0x7f08a009a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.414+0000 7f08a7c86700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08a00cc060 0x7f08a009bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.414+0000 7f08a7c86700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08a00cfde0 0x7f08a014c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.414+0000 7f08a7c86700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a009a710 con 0x7f08a00cc060 2024-05-08T21:23:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.414+0000 7f08a7c86700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a014e270 con 0x7f08a006b070 2024-05-08T21:23:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.414+0000 7f08a7c86700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a009c120 con 0x7f08a00cfde0 2024-05-08T21:23:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.415+0000 7f08a5221700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08a00cc060 0x7f08a009bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.415+0000 7f08a5221700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08a00cc060 0x7f08a009bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41948/0 (socket says 172.21.15.3:41948) 2024-05-08T21:23:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.415+0000 7f08a5221700 1 -- 172.21.15.3:0/113112977 learned_addr learned my addr 172.21.15.3:0/113112977 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.415+0000 7f08a5a22700 1 --2- 172.21.15.3:0/113112977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08a006b070 0x7f08a009a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.415+0000 7f08a6223700 1 --2- 172.21.15.3:0/113112977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08a00cfde0 0x7f08a014c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.415+0000 7f08a5221700 1 -- 172.21.15.3:0/113112977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08a00cfde0 msgr2=0x7f08a014c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:22.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.415+0000 7f08a5221700 1 --2- 172.21.15.3:0/113112977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08a00cfde0 0x7f08a014c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.415+0000 7f08a5221700 1 -- 172.21.15.3:0/113112977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08a006b070 msgr2=0x7f08a009a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:22.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.415+0000 7f08a5221700 1 --2- 172.21.15.3:0/113112977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08a006b070 0x7f08a009a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.415+0000 7f08a5221700 1 -- 172.21.15.3:0/113112977 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c0094a0 con 0x7f08a00cc060 2024-05-08T21:23:22.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.415+0000 7f08a5221700 1 --2- 172.21.15.3:0/113112977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08a00cc060 0x7f08a009bbe0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f089000c4d0 tx=0x7f089000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:22.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.415+0000 7f08967fc700 1 -- 172.21.15.3:0/113112977 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089000cea0 con 0x7f08a00cc060 2024-05-08T21:23:22.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.415+0000 7f08967fc700 1 -- 172.21.15.3:0/113112977 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0890005d70 con 0x7f08a00cc060 2024-05-08T21:23:22.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.415+0000 7f08a7c86700 1 -- 172.21.15.3:0/113112977 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a014d660 con 0x7f08a00cc060 2024-05-08T21:23:22.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.415+0000 7f08967fc700 1 -- 172.21.15.3:0/113112977 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08900072f0 con 0x7f08a00cc060 2024-05-08T21:23:22.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.415+0000 7f08a7c86700 1 -- 172.21.15.3:0/113112977 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a014dc20 con 0x7f08a00cc060 2024-05-08T21:23:22.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.416+0000 7f08a7c86700 1 -- 172.21.15.3:0/113112977 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a014e420 con 0x7f08a00cc060 2024-05-08T21:23:22.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.417+0000 7f08967fc700 1 -- 172.21.15.3:0/113112977 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0890007450 con 0x7f08a00cc060 2024-05-08T21:23:22.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.417+0000 7f08967fc700 1 --2- 172.21.15.3:0/113112977 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f088406ac90 0x7f088406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.418+0000 7f08967fc700 1 -- 172.21.15.3:0/113112977 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f08900888e0 con 0x7f08a00cc060 2024-05-08T21:23:22.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.418+0000 7f08a5a22700 1 --2- 172.21.15.3:0/113112977 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f088406ac90 0x7f088406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.418+0000 7f08a5a22700 1 --2- 172.21.15.3:0/113112977 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f088406ac90 0x7f088406d150 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f089c009fd0 tx=0x7f089c008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:22.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.422+0000 7f08967fc700 1 -- 172.21.15.3:0/113112977 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f08a014e420 con 0x7f08a00cc060 2024-05-08T21:23:22.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.660+0000 7f08a7c86700 1 -- 172.21.15.3:0/113112977 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f08a014e420 con 0x7f08a00cc060 2024-05-08T21:23:22.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.660+0000 7f08967fc700 1 -- 172.21.15.3:0/113112977 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) v1 ==== 80+0+120 (secure 0 0 0) 0x7f08a014e420 con 0x7f08a00cc060 2024-05-08T21:23:22.662 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6834/3599481370 2024-05-09T21:23:01.563392+0000 2024-05-08T21:23:22.662 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6832/3599481370 2024-05-09T21:23:01.563392+0000 2024-05-08T21:23:22.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.663+0000 7f08a7c86700 1 -- 172.21.15.3:0/113112977 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f088406ac90 msgr2=0x7f088406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:22.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.663+0000 7f08a7c86700 1 --2- 172.21.15.3:0/113112977 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f088406ac90 0x7f088406d150 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f089c009fd0 tx=0x7f089c008030 comp rx=0 tx=0).stop 2024-05-08T21:23:22.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.663+0000 7f08a7c86700 1 -- 172.21.15.3:0/113112977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08a00cc060 msgr2=0x7f08a009bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:22.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.663+0000 7f08a7c86700 1 --2- 172.21.15.3:0/113112977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08a00cc060 0x7f08a009bbe0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f089000c4d0 tx=0x7f089000c7e0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.663+0000 7f08a7c86700 1 -- 172.21.15.3:0/113112977 shutdown_connections 2024-05-08T21:23:22.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.663+0000 7f08a7c86700 1 --2- 172.21.15.3:0/113112977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08a00cfde0 0x7f08a014c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.663+0000 7f08a7c86700 1 --2- 172.21.15.3:0/113112977 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f088406ac90 0x7f088406d150 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.663+0000 7f08a7c86700 1 --2- 172.21.15.3:0/113112977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08a006b070 0x7f08a009a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.663+0000 7f08a7c86700 1 --2- 172.21.15.3:0/113112977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08a00cc060 0x7f08a009bbe0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.663+0000 7f08a7c86700 1 -- 172.21.15.3:0/113112977 >> 172.21.15.3:0/113112977 conn(0x7f08a0099a50 msgr2=0x7f08a00b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:22.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.663+0000 7f08a7c86700 1 -- 172.21.15.3:0/113112977 shutdown_connections 2024-05-08T21:23:22.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.663+0000 7f08a7c86700 1 -- 172.21.15.3:0/113112977 wait complete. 2024-05-08T21:23:22.666 INFO:teuthology.orchestra.run.smithi003.stderr:listed 2 entries 2024-05-08T21:23:22.678 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-08T21:23:22.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:22.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-08T21:23:22.724 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi003.front.sepia.ceph.com 2024-05-08T21:23:22.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:22.724 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-08T21:23:22.744 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:23:22.745 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:23:22.766 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244180k,nr_inodes=4061045,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15987 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-05-08T21:23:22.766 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:23:22.766 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:23:22.805 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16247272k,nr_inodes=4061818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=4629 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253140k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-05-08T21:23:22.805 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:23:22.806 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:23:22.827 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244180k,nr_inodes=4061045,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15987 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-05-08T21:23:22.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:23:22.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.989+0000 7ff299054700 1 Processor -- start 2024-05-08T21:23:22.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.989+0000 7ff299054700 1 -- start start 2024-05-08T21:23:22.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.990+0000 7ff299054700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff29406b140 0x7ff2940ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.990+0000 7ff299054700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29406a960 0x7ff29415bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.990+0000 7ff299054700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2940cdbb0 con 0x7ff29415c1c0 2024-05-08T21:23:22.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.990+0000 7ff299054700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2940cd310 con 0x7ff29406b140 2024-05-08T21:23:22.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.990+0000 7ff299054700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2940cd5f0 con 0x7ff29406a960 2024-05-08T21:23:22.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.990+0000 7ff292d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff29406b140 0x7ff2940ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.990+0000 7ff292d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff29406b140 0x7ff2940ccd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41954/0 (socket says 172.21.15.3:41954) 2024-05-08T21:23:22.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.990+0000 7ff292d9d700 1 -- 172.21.15.3:0/2600622148 learned_addr learned my addr 172.21.15.3:0/2600622148 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:22.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.990+0000 7ff29259c700 1 --2- 172.21.15.3:0/2600622148 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29406a960 0x7ff29415bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.990+0000 7ff292d9d700 1 -- 172.21.15.3:0/2600622148 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29406a960 msgr2=0x7ff29415bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:22.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.990+0000 7ff292d9d700 1 --2- 172.21.15.3:0/2600622148 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29406a960 0x7ff29415bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.990+0000 7ff292d9d700 1 -- 172.21.15.3:0/2600622148 >> v1:172.21.15.186:6789/0 conn(0x7ff29415c1c0 legacy=0x7ff2940cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.990+0000 7ff292d9d700 1 -- 172.21.15.3:0/2600622148 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2940cd8d0 con 0x7ff29406b140 2024-05-08T21:23:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.991+0000 7ff292d9d700 1 --2- 172.21.15.3:0/2600622148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff29406b140 0x7ff2940ccd30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff2840098f0 tx=0x7ff284009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b9b3a21699f56bf server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.991+0000 7ff291d9b700 1 -- 172.21.15.3:0/2600622148 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28400b8a0 con 0x7ff29406b140 2024-05-08T21:23:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.991+0000 7ff291d9b700 1 -- 172.21.15.3:0/2600622148 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28400ba00 con 0x7ff29406b140 2024-05-08T21:23:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.991+0000 7ff291d9b700 1 -- 172.21.15.3:0/2600622148 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28400bd90 con 0x7ff29406b140 2024-05-08T21:23:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.991+0000 7ff299054700 1 -- 172.21.15.3:0/2600622148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff29406b140 msgr2=0x7ff2940ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.991+0000 7ff299054700 1 --2- 172.21.15.3:0/2600622148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff29406b140 0x7ff2940ccd30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff2840098f0 tx=0x7ff284009c00 comp rx=0 tx=0).stop 2024-05-08T21:23:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.991+0000 7ff299054700 1 -- 172.21.15.3:0/2600622148 shutdown_connections 2024-05-08T21:23:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.991+0000 7ff299054700 1 --2- 172.21.15.3:0/2600622148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff29406b140 0x7ff2940ccd30 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.991+0000 7ff299054700 1 --2- 172.21.15.3:0/2600622148 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29406a960 0x7ff29415bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.991+0000 7ff299054700 1 -- 172.21.15.3:0/2600622148 >> 172.21.15.3:0/2600622148 conn(0x7ff2940d0320 msgr2=0x7ff2940d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.991+0000 7ff299054700 1 -- 172.21.15.3:0/2600622148 shutdown_connections 2024-05-08T21:23:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.991+0000 7ff299054700 1 -- 172.21.15.3:0/2600622148 wait complete. 2024-05-08T21:23:22.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.993+0000 7ff299054700 1 Processor -- start 2024-05-08T21:23:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.993+0000 7ff299054700 1 -- start start 2024-05-08T21:23:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff299054700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29406a960 0x7ff2940c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff299054700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff29406b140 0x7ff2940c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff299054700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29415c1c0 0x7ff2940c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff299054700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff294142ae0 con 0x7ff29406b140 2024-05-08T21:23:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff299054700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2940ca8a0 con 0x7ff29406a960 2024-05-08T21:23:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff299054700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29409b9a0 con 0x7ff29415c1c0 2024-05-08T21:23:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff29259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff29406b140 0x7ff2940c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff29259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff29406b140 0x7ff2940c64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41966/0 (socket says 172.21.15.3:41966) 2024-05-08T21:23:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff29259c700 1 -- 172.21.15.3:0/1627453590 learned_addr learned my addr 172.21.15.3:0/1627453590 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff292d9d700 1 --2- 172.21.15.3:0/1627453590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29406a960 0x7ff2940c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff29359e700 1 --2- 172.21.15.3:0/1627453590 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29415c1c0 0x7ff2940c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff29259c700 1 -- 172.21.15.3:0/1627453590 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29415c1c0 msgr2=0x7ff2940c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff29259c700 1 --2- 172.21.15.3:0/1627453590 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29415c1c0 0x7ff2940c4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff29259c700 1 -- 172.21.15.3:0/1627453590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29406a960 msgr2=0x7ff2940c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:22.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff29259c700 1 --2- 172.21.15.3:0/1627453590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29406a960 0x7ff2940c5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:22.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff29259c700 1 -- 172.21.15.3:0/1627453590 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff284009550 con 0x7ff29406b140 2024-05-08T21:23:22.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.994+0000 7ff29259c700 1 --2- 172.21.15.3:0/1627453590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff29406b140 0x7ff2940c64f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff28800e840 tx=0x7ff28800eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:22.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.995+0000 7ff27b7fe700 1 -- 172.21.15.3:0/1627453590 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288013610 con 0x7ff29406b140 2024-05-08T21:23:22.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.995+0000 7ff27b7fe700 1 -- 172.21.15.3:0/1627453590 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff288013770 con 0x7ff29406b140 2024-05-08T21:23:22.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.995+0000 7ff299054700 1 -- 172.21.15.3:0/1627453590 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2940c3cc0 con 0x7ff29406b140 2024-05-08T21:23:22.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.995+0000 7ff27b7fe700 1 -- 172.21.15.3:0/1627453590 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288013b20 con 0x7ff29406b140 2024-05-08T21:23:22.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.995+0000 7ff299054700 1 -- 172.21.15.3:0/1627453590 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff294142490 con 0x7ff29406b140 2024-05-08T21:23:22.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.996+0000 7ff299054700 1 -- 172.21.15.3:0/1627453590 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff294142c50 con 0x7ff29406b140 2024-05-08T21:23:22.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.996+0000 7ff27b7fe700 1 -- 172.21.15.3:0/1627453590 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff288013c80 con 0x7ff29406b140 2024-05-08T21:23:22.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.997+0000 7ff27b7fe700 1 --2- 172.21.15.3:0/1627453590 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff28006ac90 0x7ff28006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:22.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.997+0000 7ff27b7fe700 1 -- 172.21.15.3:0/1627453590 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7ff288089100 con 0x7ff29406b140 2024-05-08T21:23:22.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.997+0000 7ff292d9d700 1 --2- 172.21.15.3:0/1627453590 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff28006ac90 0x7ff28006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:22.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:22.997+0000 7ff292d9d700 1 --2- 172.21.15.3:0/1627453590 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff28006ac90 0x7ff28006d150 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff284009c40 tx=0x7ff284008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:23.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.001+0000 7ff27b7fe700 1 -- 172.21.15.3:0/1627453590 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff2880540d0 con 0x7ff29406b140 2024-05-08T21:23:23.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.270+0000 7ff299054700 1 -- 172.21.15.3:0/1627453590 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff294068000 con 0x7ff29406b140 2024-05-08T21:23:23.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.271+0000 7ff27b7fe700 1 -- 172.21.15.3:0/1627453590 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5936 (secure 0 0 0) 0x7ff2880576f0 con 0x7ff29406b140 2024-05-08T21:23:23.272 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:23.272 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6833/2104267343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2104267343},{"type":"v1","addr":"172.21.15.3:6833","nonce":2104267343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/208321708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":208321708},{"type":"v1","addr":"172.21.15.3:6835","nonce":208321708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/1591386131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1591386131},{"type":"v1","addr":"172.21.15.186:6836","nonce":1591386131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:22:50.184245+0000","modified":"2024-05-08T21:23:01.563810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:23:05.475355+0000","modified":"2024-05-08T21:23:15.662101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6837/2000072997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":2000072997},{"type":"v1","addr":"172.21.15.186:6837","nonce":2000072997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":6,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-08T21:23:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.273+0000 7ff299054700 1 -- 172.21.15.3:0/1627453590 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff28006ac90 msgr2=0x7ff28006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.273+0000 7ff299054700 1 --2- 172.21.15.3:0/1627453590 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff28006ac90 0x7ff28006d150 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff284009c40 tx=0x7ff284008030 comp rx=0 tx=0).stop 2024-05-08T21:23:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.273+0000 7ff299054700 1 -- 172.21.15.3:0/1627453590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff29406b140 msgr2=0x7ff2940c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.273+0000 7ff299054700 1 --2- 172.21.15.3:0/1627453590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff29406b140 0x7ff2940c64f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff28800e840 tx=0x7ff28800eb50 comp rx=0 tx=0).stop 2024-05-08T21:23:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.273+0000 7ff299054700 1 -- 172.21.15.3:0/1627453590 shutdown_connections 2024-05-08T21:23:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.273+0000 7ff299054700 1 --2- 172.21.15.3:0/1627453590 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29415c1c0 0x7ff2940c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.273+0000 7ff299054700 1 --2- 172.21.15.3:0/1627453590 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff28006ac90 0x7ff28006d150 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.273+0000 7ff299054700 1 --2- 172.21.15.3:0/1627453590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29406a960 0x7ff2940c5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.273+0000 7ff299054700 1 --2- 172.21.15.3:0/1627453590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff29406b140 0x7ff2940c64f0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.273+0000 7ff299054700 1 -- 172.21.15.3:0/1627453590 >> 172.21.15.3:0/1627453590 conn(0x7ff2940d0320 msgr2=0x7ff2940d0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.273+0000 7ff299054700 1 -- 172.21.15.3:0/1627453590 shutdown_connections 2024-05-08T21:23:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.273+0000 7ff299054700 1 -- 172.21.15.3:0/1627453590 wait complete. 2024-05-08T21:23:23.276 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 26 2024-05-08T21:23:23.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:23:23.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.424+0000 7f6a7b579700 1 Processor -- start 2024-05-08T21:23:23.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.424+0000 7f6a7b579700 1 -- start start 2024-05-08T21:23:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.425+0000 7f6a7b579700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6a740b03e0 0x7f6a7409a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.425+0000 7f6a7b579700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a740cc6b0 con 0x7f6a7415ed40 2024-05-08T21:23:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.425+0000 7f6a7b579700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a740d0030 con 0x7f6a74099a50 2024-05-08T21:23:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.425+0000 7f6a7b579700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a740b14b0 con 0x7f6a740b03e0 2024-05-08T21:23:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.425+0000 7f6a79315700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6a7415ed40 0x7f6a740b0da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50452/0 (socket says 172.21.15.3:50452) 2024-05-08T21:23:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.425+0000 7f6a79315700 1 -- 172.21.15.3:0/4218480099 learned_addr learned my addr 172.21.15.3:0/4218480099 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.425+0000 7f6a78b14700 1 --2- 172.21.15.3:0/4218480099 >> v2:172.21.15.186:3300/0 conn(0x7f6a740b03e0 0x7f6a7409a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.425+0000 7f6a6bfff700 1 -- 172.21.15.3:0/4218480099 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741687586 0 0) 0x7f6a740cc6b0 con 0x7f6a7415ed40 2024-05-08T21:23:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.425+0000 7f6a6bfff700 1 -- 172.21.15.3:0/4218480099 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a600035b0 con 0x7f6a7415ed40 2024-05-08T21:23:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.425+0000 7f6a6bfff700 1 -- 172.21.15.3:0/4218480099 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 941513033 0 0) 0x7f6a600035b0 con 0x7f6a7415ed40 2024-05-08T21:23:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.425+0000 7f6a6bfff700 1 -- 172.21.15.3:0/4218480099 >> v2:172.21.15.186:3300/0 conn(0x7f6a740b03e0 msgr2=0x7f6a7409a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.425+0000 7f6a6bfff700 1 --2- 172.21.15.3:0/4218480099 >> v2:172.21.15.186:3300/0 conn(0x7f6a740b03e0 0x7f6a7409a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.425+0000 7f6a6bfff700 1 -- 172.21.15.3:0/4218480099 >> v1:172.21.15.186:6789/0 conn(0x7f6a74099a50 legacy=0x7f6a740ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.425+0000 7f6a6bfff700 1 -- 172.21.15.3:0/4218480099 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a740d0310 con 0x7f6a7415ed40 2024-05-08T21:23:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.425+0000 7f6a6bfff700 1 -- 172.21.15.3:0/4218480099 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f6a70001670 con 0x7f6a7415ed40 2024-05-08T21:23:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.425+0000 7f6a6bfff700 1 -- 172.21.15.3:0/4218480099 >> v1:172.21.15.3:6789/0 conn(0x7f6a7415ed40 legacy=0x7f6a740b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a6bfff700 1 --2- 172.21.15.3:0/4218480099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a60003a00 0x7f6a60003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a6bfff700 1 --2- 172.21.15.3:0/4218480099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a60004850 0x7f6a60004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a6bfff700 1 --2- 172.21.15.3:0/4218480099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a60005210 0x7f6a6000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a6bfff700 1 -- 172.21.15.3:0/4218480099 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740b14b0 con 0x7f6a60003a00 2024-05-08T21:23:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a6bfff700 1 -- 172.21.15.3:0/4218480099 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6000c0b0 con 0x7f6a60005210 2024-05-08T21:23:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a6bfff700 1 -- 172.21.15.3:0/4218480099 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6000c270 con 0x7f6a60004850 2024-05-08T21:23:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a79315700 1 --2- 172.21.15.3:0/4218480099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a60003a00 0x7f6a60003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a79b16700 1 --2- 172.21.15.3:0/4218480099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a60005210 0x7f6a6000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a78b14700 1 --2- 172.21.15.3:0/4218480099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a60004850 0x7f6a60004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a79315700 1 -- 172.21.15.3:0/4218480099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a60004850 msgr2=0x7f6a60004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a79315700 1 --2- 172.21.15.3:0/4218480099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a60004850 0x7f6a60004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a79315700 1 -- 172.21.15.3:0/4218480099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a60005210 msgr2=0x7f6a6000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a79315700 1 --2- 172.21.15.3:0/4218480099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a60005210 0x7f6a6000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a79315700 1 -- 172.21.15.3:0/4218480099 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a600035b0 con 0x7f6a60003a00 2024-05-08T21:23:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a79b16700 1 --2- 172.21.15.3:0/4218480099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a60005210 0x7f6a6000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a79b16700 1 -- 172.21.15.3:0/4218480099 reap_dead start 2024-05-08T21:23:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a79315700 1 --2- 172.21.15.3:0/4218480099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a60003a00 0x7f6a60003e20 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000c240 tx=0x7f6a60005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7627d1e307eb960f server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a6bfff700 1 -- 172.21.15.3:0/4218480099 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a70007c10 con 0x7f6a60003a00 2024-05-08T21:23:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a6bfff700 1 -- 172.21.15.3:0/4218480099 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7000ce00 con 0x7f6a60003a00 2024-05-08T21:23:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.426+0000 7f6a6bfff700 1 -- 172.21.15.3:0/4218480099 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a70005ba0 con 0x7f6a60003a00 2024-05-08T21:23:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.427+0000 7f6a7b579700 1 -- 172.21.15.3:0/4218480099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a60003a00 msgr2=0x7f6a60003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.427+0000 7f6a7b579700 1 --2- 172.21.15.3:0/4218480099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a60003a00 0x7f6a60003e20 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000c240 tx=0x7f6a60005880 comp rx=0 tx=0).stop 2024-05-08T21:23:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.427+0000 7f6a7b579700 1 -- 172.21.15.3:0/4218480099 shutdown_connections 2024-05-08T21:23:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.427+0000 7f6a7b579700 1 --2- 172.21.15.3:0/4218480099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a60003a00 0x7f6a60003e20 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.427+0000 7f6a7b579700 1 -- 172.21.15.3:0/4218480099 >> 172.21.15.3:0/4218480099 conn(0x7f6a740b2f30 msgr2=0x7f6a740b3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:23.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.427+0000 7f6a7b579700 1 -- 172.21.15.3:0/4218480099 shutdown_connections 2024-05-08T21:23:23.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.427+0000 7f6a7b579700 1 -- 172.21.15.3:0/4218480099 wait complete. 2024-05-08T21:23:23.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.428+0000 7f6a7b579700 1 Processor -- start 2024-05-08T21:23:23.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.428+0000 7f6a7b579700 1 -- start start 2024-05-08T21:23:23.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.429+0000 7f6a7b579700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a60003a00 0x7f6a740c7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.429+0000 7f6a7b579700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a740b6a40 0x7f6a74157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.429+0000 7f6a7b579700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a7406cb90 0x7f6a740b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.429+0000 7f6a7b579700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740c7000 con 0x7f6a7406cb90 2024-05-08T21:23:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.429+0000 7f6a7b579700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7406c700 con 0x7f6a740b6a40 2024-05-08T21:23:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.429+0000 7f6a7b579700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740b6ec0 con 0x7f6a60003a00 2024-05-08T21:23:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.429+0000 7f6a79b16700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a7406cb90 0x7f6a740b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.429+0000 7f6a79b16700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a7406cb90 0x7f6a740b49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41988/0 (socket says 172.21.15.3:41988) 2024-05-08T21:23:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.429+0000 7f6a79b16700 1 -- 172.21.15.3:0/142962551 learned_addr learned my addr 172.21.15.3:0/142962551 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.430+0000 7f6a79315700 1 --2- 172.21.15.3:0/142962551 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a60003a00 0x7f6a740c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.430+0000 7f6a78b14700 1 --2- 172.21.15.3:0/142962551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a740b6a40 0x7f6a74157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.430+0000 7f6a79b16700 1 -- 172.21.15.3:0/142962551 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a60003a00 msgr2=0x7f6a740c7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.430+0000 7f6a79b16700 1 --2- 172.21.15.3:0/142962551 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a60003a00 0x7f6a740c7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.430+0000 7f6a79b16700 1 -- 172.21.15.3:0/142962551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a740b6a40 msgr2=0x7f6a74157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.430+0000 7f6a79b16700 1 --2- 172.21.15.3:0/142962551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a740b6a40 0x7f6a74157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.430+0000 7f6a79b16700 1 -- 172.21.15.3:0/142962551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a740b7120 con 0x7f6a7406cb90 2024-05-08T21:23:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.430+0000 7f6a79b16700 1 --2- 172.21.15.3:0/142962551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a7406cb90 0x7f6a740b49f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f6a740b2e10 tx=0x7f6a6c00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.430+0000 7f6a69ffb700 1 -- 172.21.15.3:0/142962551 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c00ee80 con 0x7f6a7406cb90 2024-05-08T21:23:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.430+0000 7f6a69ffb700 1 -- 172.21.15.3:0/142962551 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6c003c10 con 0x7f6a7406cb90 2024-05-08T21:23:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.430+0000 7f6a7b579700 1 -- 172.21.15.3:0/142962551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a740b4060 con 0x7f6a7406cb90 2024-05-08T21:23:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.430+0000 7f6a69ffb700 1 -- 172.21.15.3:0/142962551 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c0050d0 con 0x7f6a7406cb90 2024-05-08T21:23:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.431+0000 7f6a7b579700 1 -- 172.21.15.3:0/142962551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a740b4620 con 0x7f6a7406cb90 2024-05-08T21:23:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.431+0000 7f6a7b579700 1 -- 172.21.15.3:0/142962551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a740b47e0 con 0x7f6a7406cb90 2024-05-08T21:23:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.432+0000 7f6a69ffb700 1 -- 172.21.15.3:0/142962551 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6a6c005230 con 0x7f6a7406cb90 2024-05-08T21:23:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.432+0000 7f6a69ffb700 1 --2- 172.21.15.3:0/142962551 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6a5806ac90 0x7f6a5806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.432+0000 7f6a69ffb700 1 -- 172.21.15.3:0/142962551 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f6a6c089320 con 0x7f6a7406cb90 2024-05-08T21:23:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.432+0000 7f6a79315700 1 --2- 172.21.15.3:0/142962551 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6a5806ac90 0x7f6a5806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.433+0000 7f6a79315700 1 --2- 172.21.15.3:0/142962551 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6a5806ac90 0x7f6a5806d150 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6a70000c00 tx=0x7f6a70007000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.437+0000 7f6a69ffb700 1 -- 172.21.15.3:0/142962551 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6a740b47e0 con 0x7f6a7406cb90 2024-05-08T21:23:23.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.655+0000 7f6a7b579700 1 -- 172.21.15.3:0/142962551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a740b47e0 con 0x7f6a7406cb90 2024-05-08T21:23:23.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.657+0000 7f6a69ffb700 1 -- 172.21.15.3:0/142962551 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+18029 (secure 0 0 0) 0x7f6a740b47e0 con 0x7f6a7406cb90 2024-05-08T21:23:23.659 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:23.659 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":42,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:23:06.476868+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:22:47.853786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-08T21:22:49.153759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"recovery_fs_metadata","create_time":"2024-05-08T21:23:03.921695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6834/3599481370":"2024-05-09T21:23:01.563392+0000","172.21.15.3:6832/3599481370":"2024-05-09T21:23:01.563392+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:23:23.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.659+0000 7f6a7b579700 1 -- 172.21.15.3:0/142962551 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6a5806ac90 msgr2=0x7f6a5806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:23.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.659+0000 7f6a7b579700 1 --2- 172.21.15.3:0/142962551 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6a5806ac90 0x7f6a5806d150 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6a70000c00 tx=0x7f6a70007000 comp rx=0 tx=0).stop 2024-05-08T21:23:23.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.659+0000 7f6a7b579700 1 -- 172.21.15.3:0/142962551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a7406cb90 msgr2=0x7f6a740b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:23.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.659+0000 7f6a7b579700 1 --2- 172.21.15.3:0/142962551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a7406cb90 0x7f6a740b49f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f6a740b2e10 tx=0x7f6a6c00ed50 comp rx=0 tx=0).stop 2024-05-08T21:23:23.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.659+0000 7f6a7b579700 1 -- 172.21.15.3:0/142962551 shutdown_connections 2024-05-08T21:23:23.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.659+0000 7f6a7b579700 1 --2- 172.21.15.3:0/142962551 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a60003a00 0x7f6a740c7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.659+0000 7f6a7b579700 1 --2- 172.21.15.3:0/142962551 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6a5806ac90 0x7f6a5806d150 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.659+0000 7f6a7b579700 1 --2- 172.21.15.3:0/142962551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a7406cb90 0x7f6a740b49f0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.659+0000 7f6a7b579700 1 --2- 172.21.15.3:0/142962551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a740b6a40 0x7f6a74157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.659+0000 7f6a7b579700 1 -- 172.21.15.3:0/142962551 >> 172.21.15.3:0/142962551 conn(0x7f6a740b2f30 msgr2=0x7f6a74070140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:23.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.659+0000 7f6a7b579700 1 -- 172.21.15.3:0/142962551 shutdown_connections 2024-05-08T21:23:23.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.659+0000 7f6a7b579700 1 -- 172.21.15.3:0/142962551 wait complete. 2024-05-08T21:23:23.674 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-08T21:22:47.853786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:22:49.153759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'recovery_fs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'recovery_fs_metadata', 'create_time': '2024-05-08T21:23:03.921695+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'recovery_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:23:23.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:23:23.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.812+0000 7f7d14db0700 1 Processor -- start 2024-05-08T21:23:23.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.812+0000 7f7d14db0700 1 -- start start 2024-05-08T21:23:23.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.813+0000 7f7d14db0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d1015b4e0 0x7f7d100b1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:23.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.813+0000 7f7d14db0700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7d1015b8c0 0x7f7d100d1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:23.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.813+0000 7f7d14db0700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7d100d1740 0x7f7d100cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:23.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.813+0000 7f7d14db0700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d100cefe0 con 0x7f7d1015b8c0 2024-05-08T21:23:23.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.813+0000 7f7d14db0700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d100cf560 con 0x7f7d100d1740 2024-05-08T21:23:23.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.813+0000 7f7d14db0700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d100d1bb0 con 0x7f7d1015b4e0 2024-05-08T21:23:23.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.813+0000 7f7d0dd9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7d1015b8c0 0x7f7d100d1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:23.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.813+0000 7f7d0dd9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7d1015b8c0 0x7f7d100d1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42004/0 (socket says 172.21.15.3:42004) 2024-05-08T21:23:23.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.813+0000 7f7d0dd9b700 1 -- 172.21.15.3:0/3383397722 learned_addr learned my addr 172.21.15.3:0/3383397722 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:23.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.813+0000 7f7d0e59c700 1 --2- 172.21.15.3:0/3383397722 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d1015b4e0 0x7f7d100b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:23.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.813+0000 7f7d0ed9d700 1 --2- 172.21.15.3:0/3383397722 >> v2:172.21.15.186:3300/0 conn(0x7f7d100d1740 0x7f7d100cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:23.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.814+0000 7f7d0dd9b700 1 -- 172.21.15.3:0/3383397722 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d1015b4e0 msgr2=0x7f7d100b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:23.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.814+0000 7f7d0dd9b700 1 --2- 172.21.15.3:0/3383397722 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d1015b4e0 0x7f7d100b1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.814+0000 7f7d0dd9b700 1 -- 172.21.15.3:0/3383397722 >> v2:172.21.15.186:3300/0 conn(0x7f7d100d1740 msgr2=0x7f7d100cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:23.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.814+0000 7f7d0dd9b700 1 --2- 172.21.15.3:0/3383397722 >> v2:172.21.15.186:3300/0 conn(0x7f7d100d1740 0x7f7d100cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.814+0000 7f7d0dd9b700 1 -- 172.21.15.3:0/3383397722 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d100ccff0 con 0x7f7d1015b8c0 2024-05-08T21:23:23.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.814+0000 7f7d0dd9b700 1 --2- 172.21.15.3:0/3383397722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d1015b8c0 0x7f7d100d1200 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7d10073150 tx=0x7f7d04009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0e2d71a75bdf01 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:23.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.814+0000 7f7d0d59a700 1 -- 172.21.15.3:0/3383397722 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d04011830 con 0x7f7d1015b8c0 2024-05-08T21:23:23.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.814+0000 7f7d0d59a700 1 -- 172.21.15.3:0/3383397722 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d04011990 con 0x7f7d1015b8c0 2024-05-08T21:23:23.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.814+0000 7f7d0d59a700 1 -- 172.21.15.3:0/3383397722 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d04011d20 con 0x7f7d1015b8c0 2024-05-08T21:23:23.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.815+0000 7f7d14db0700 1 -- 172.21.15.3:0/3383397722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d1015b8c0 msgr2=0x7f7d100d1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:23.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.815+0000 7f7d14db0700 1 --2- 172.21.15.3:0/3383397722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d1015b8c0 0x7f7d100d1200 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7d10073150 tx=0x7f7d04009bd0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.815+0000 7f7d14db0700 1 -- 172.21.15.3:0/3383397722 shutdown_connections 2024-05-08T21:23:23.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.815+0000 7f7d14db0700 1 --2- 172.21.15.3:0/3383397722 >> v2:172.21.15.186:3300/0 conn(0x7f7d100d1740 0x7f7d100cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.815+0000 7f7d14db0700 1 --2- 172.21.15.3:0/3383397722 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d1015b4e0 0x7f7d100b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.815+0000 7f7d14db0700 1 --2- 172.21.15.3:0/3383397722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d1015b8c0 0x7f7d100d1200 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.815+0000 7f7d14db0700 1 -- 172.21.15.3:0/3383397722 >> 172.21.15.3:0/3383397722 conn(0x7f7d1008afe0 msgr2=0x7f7d1008b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:23.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.815+0000 7f7d14db0700 1 -- 172.21.15.3:0/3383397722 shutdown_connections 2024-05-08T21:23:23.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.815+0000 7f7d14db0700 1 -- 172.21.15.3:0/3383397722 wait complete. 2024-05-08T21:23:23.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.816+0000 7f7d14db0700 1 Processor -- start 2024-05-08T21:23:23.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.816+0000 7f7d14db0700 1 -- start start 2024-05-08T21:23:23.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.817+0000 7f7d14db0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d100d1740 0x7f7d1014fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:23.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.817+0000 7f7d14db0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d1015b4e0 0x7f7d1015bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:23.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.817+0000 7f7d14db0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d1015b8c0 0x7f7d1015d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:23.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.817+0000 7f7d14db0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1015c2c0 con 0x7f7d1015b4e0 2024-05-08T21:23:23.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.817+0000 7f7d14db0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1015c590 con 0x7f7d1015b8c0 2024-05-08T21:23:23.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.817+0000 7f7d14db0700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1015c870 con 0x7f7d100d1740 2024-05-08T21:23:23.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.817+0000 7f7d0dd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d1015b4e0 0x7f7d1015bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:23.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.818+0000 7f7d0dd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d1015b4e0 0x7f7d1015bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42008/0 (socket says 172.21.15.3:42008) 2024-05-08T21:23:23.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.818+0000 7f7d0dd9b700 1 -- 172.21.15.3:0/3819273041 learned_addr learned my addr 172.21.15.3:0/3819273041 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:23.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.818+0000 7f7d0e59c700 1 --2- 172.21.15.3:0/3819273041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d100d1740 0x7f7d1014fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:23.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.818+0000 7f7d0ed9d700 1 --2- 172.21.15.3:0/3819273041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d1015b8c0 0x7f7d1015d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:23.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.818+0000 7f7d0dd9b700 1 -- 172.21.15.3:0/3819273041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d100d1740 msgr2=0x7f7d1014fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:23.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.818+0000 7f7d0dd9b700 1 --2- 172.21.15.3:0/3819273041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d100d1740 0x7f7d1014fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.818+0000 7f7d0dd9b700 1 -- 172.21.15.3:0/3819273041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d1015b8c0 msgr2=0x7f7d1015d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:23.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.818+0000 7f7d0dd9b700 1 --2- 172.21.15.3:0/3819273041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d1015b8c0 0x7f7d1015d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:23.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.818+0000 7f7d0dd9b700 1 -- 172.21.15.3:0/3819273041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d04009550 con 0x7f7d1015b4e0 2024-05-08T21:23:23.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.818+0000 7f7d0dd9b700 1 --2- 172.21.15.3:0/3819273041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d1015b4e0 0x7f7d1015bd80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009c10 tx=0x7f7d04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:23.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.820+0000 7f7cfeffd700 1 -- 172.21.15.3:0/3819273041 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d04004450 con 0x7f7d1015b4e0 2024-05-08T21:23:23.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.820+0000 7f7d14db0700 1 -- 172.21.15.3:0/3819273041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d1015cb50 con 0x7f7d1015b4e0 2024-05-08T21:23:23.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.820+0000 7f7cfeffd700 1 -- 172.21.15.3:0/3819273041 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0401e070 con 0x7f7d1015b4e0 2024-05-08T21:23:23.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.820+0000 7f7cfeffd700 1 -- 172.21.15.3:0/3819273041 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d040038e0 con 0x7f7d1015b4e0 2024-05-08T21:23:23.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.820+0000 7f7d14db0700 1 -- 172.21.15.3:0/3819273041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d1006c8e0 con 0x7f7d1015b4e0 2024-05-08T21:23:23.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.821+0000 7f7d14db0700 1 -- 172.21.15.3:0/3819273041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d10074bc0 con 0x7f7d1015b4e0 2024-05-08T21:23:23.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.821+0000 7f7cfeffd700 1 -- 172.21.15.3:0/3819273041 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7d04003a40 con 0x7f7d1015b4e0 2024-05-08T21:23:23.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.821+0000 7f7cfeffd700 1 --2- 172.21.15.3:0/3819273041 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7cf006ac90 0x7f7cf006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:23.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.822+0000 7f7cfeffd700 1 -- 172.21.15.3:0/3819273041 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f7d0408a310 con 0x7f7d1015b4e0 2024-05-08T21:23:23.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.822+0000 7f7d0e59c700 1 --2- 172.21.15.3:0/3819273041 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7cf006ac90 0x7f7cf006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:23.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.822+0000 7f7d0e59c700 1 --2- 172.21.15.3:0/3819273041 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7cf006ac90 0x7f7cf006d150 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7d00009b60 tx=0x7f7d00009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:23.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:23.826+0000 7f7cfeffd700 1 -- 172.21.15.3:0/3819273041 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f7d04055240 con 0x7f7d1015b4e0 2024-05-08T21:23:24.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.042+0000 7f7d14db0700 1 -- 172.21.15.3:0/3819273041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d100b0fe0 con 0x7f7d1015b4e0 2024-05-08T21:23:24.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.042+0000 7f7cfeffd700 1 -- 172.21.15.3:0/3819273041 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7d04058860 con 0x7f7d1015b4e0 2024-05-08T21:23:24.044 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:23:24.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.045+0000 7f7d14db0700 1 -- 172.21.15.3:0/3819273041 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7cf006ac90 msgr2=0x7f7cf006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.045+0000 7f7d14db0700 1 --2- 172.21.15.3:0/3819273041 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7cf006ac90 0x7f7cf006d150 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7d00009b60 tx=0x7f7d00009210 comp rx=0 tx=0).stop 2024-05-08T21:23:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.045+0000 7f7d14db0700 1 -- 172.21.15.3:0/3819273041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d1015b4e0 msgr2=0x7f7d1015bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.045+0000 7f7d14db0700 1 --2- 172.21.15.3:0/3819273041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d1015b4e0 0x7f7d1015bd80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009c10 tx=0x7f7d04004750 comp rx=0 tx=0).stop 2024-05-08T21:23:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.045+0000 7f7d14db0700 1 -- 172.21.15.3:0/3819273041 shutdown_connections 2024-05-08T21:23:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.045+0000 7f7d14db0700 1 --2- 172.21.15.3:0/3819273041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d100d1740 0x7f7d1014fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.045+0000 7f7d14db0700 1 --2- 172.21.15.3:0/3819273041 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7cf006ac90 0x7f7cf006d150 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.045+0000 7f7d14db0700 1 --2- 172.21.15.3:0/3819273041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d1015b8c0 0x7f7d1015d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.045+0000 7f7d14db0700 1 --2- 172.21.15.3:0/3819273041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d1015b4e0 0x7f7d1015bd80 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.045+0000 7f7d14db0700 1 -- 172.21.15.3:0/3819273041 >> 172.21.15.3:0/3819273041 conn(0x7f7d1008afe0 msgr2=0x7f7d10155e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.045+0000 7f7d14db0700 1 -- 172.21.15.3:0/3819273041 shutdown_connections 2024-05-08T21:23:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.045+0000 7f7d14db0700 1 -- 172.21.15.3:0/3819273041 wait complete. 2024-05-08T21:23:24.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-08T21:23:24.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.198+0000 7f76f1384700 1 Processor -- start 2024-05-08T21:23:24.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.198+0000 7f76f1384700 1 -- start start 2024-05-08T21:23:24.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.198+0000 7f76f1384700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f76ec15a7c0 0x7f76ec0cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.198+0000 7f76f1384700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f76ec0ca130 0x7f76ec0b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.198+0000 7f76f1384700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec0cd7c0 con 0x7f76ec0727b0 2024-05-08T21:23:24.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.198+0000 7f76f1384700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76ec0cd1b0 con 0x7f76ec0ca130 2024-05-08T21:23:24.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.198+0000 7f76f1384700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76ec15aba0 con 0x7f76ec15a7c0 2024-05-08T21:23:24.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.199+0000 7f76eb7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f76ec0ca130 0x7f76ec0b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.199+0000 7f76ea7fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f76ec0727b0 0x7f76ec0c9a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50460/0 (socket says 172.21.15.3:50460) 2024-05-08T21:23:24.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.199+0000 7f76ea7fc700 1 -- 172.21.15.3:0/2388881109 learned_addr learned my addr 172.21.15.3:0/2388881109 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:24.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.199+0000 7f76eaffd700 1 --2- 172.21.15.3:0/2388881109 >> v2:172.21.15.186:3300/0 conn(0x7f76ec15a7c0 0x7f76ec0cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.199+0000 7f76eb7fe700 1 -- 172.21.15.3:0/2388881109 >> v2:172.21.15.186:3300/0 conn(0x7f76ec15a7c0 msgr2=0x7f76ec0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:24.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.199+0000 7f76eb7fe700 1 --2- 172.21.15.3:0/2388881109 >> v2:172.21.15.186:3300/0 conn(0x7f76ec15a7c0 0x7f76ec0cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.199+0000 7f76eb7fe700 1 -- 172.21.15.3:0/2388881109 >> v1:172.21.15.3:6789/0 conn(0x7f76ec0727b0 legacy=0x7f76ec0c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.199+0000 7f76eb7fe700 1 -- 172.21.15.3:0/2388881109 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec0b3e70 con 0x7f76ec0ca130 2024-05-08T21:23:24.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.199+0000 7f76e9ffb700 1 -- 172.21.15.3:0/2388881109 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288356789 0 0) 0x7f76ec0cd7c0 con 0x7f76ec0727b0 2024-05-08T21:23:24.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.199+0000 7f76eb7fe700 1 --2- 172.21.15.3:0/2388881109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76ec0ca130 0x7f76ec0b24f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f76ec06c590 tx=0x7f76e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c98b9afa32ac197d server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:24.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.199+0000 7f76e9ffb700 1 -- 172.21.15.3:0/2388881109 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e400e070 con 0x7f76ec0ca130 2024-05-08T21:23:24.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.199+0000 7f76e9ffb700 1 -- 172.21.15.3:0/2388881109 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e400bbd0 con 0x7f76ec0ca130 2024-05-08T21:23:24.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.199+0000 7f76e9ffb700 1 -- 172.21.15.3:0/2388881109 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e4012430 con 0x7f76ec0ca130 2024-05-08T21:23:24.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.200+0000 7f76f1384700 1 -- 172.21.15.3:0/2388881109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76ec0ca130 msgr2=0x7f76ec0b24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.200+0000 7f76f1384700 1 --2- 172.21.15.3:0/2388881109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76ec0ca130 0x7f76ec0b24f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f76ec06c590 tx=0x7f76e400b330 comp rx=0 tx=0).stop 2024-05-08T21:23:24.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.200+0000 7f76f1384700 1 -- 172.21.15.3:0/2388881109 shutdown_connections 2024-05-08T21:23:24.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.200+0000 7f76f1384700 1 --2- 172.21.15.3:0/2388881109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76ec0ca130 0x7f76ec0b24f0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.200+0000 7f76f1384700 1 --2- 172.21.15.3:0/2388881109 >> v2:172.21.15.186:3300/0 conn(0x7f76ec15a7c0 0x7f76ec0cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.200+0000 7f76f1384700 1 -- 172.21.15.3:0/2388881109 >> 172.21.15.3:0/2388881109 conn(0x7f76ec06a960 msgr2=0x7f76ec06ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:24.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.200+0000 7f76f1384700 1 -- 172.21.15.3:0/2388881109 shutdown_connections 2024-05-08T21:23:24.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.200+0000 7f76f1384700 1 -- 172.21.15.3:0/2388881109 wait complete. 2024-05-08T21:23:24.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.202+0000 7f76f1384700 1 Processor -- start 2024-05-08T21:23:24.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.202+0000 7f76f1384700 1 -- start start 2024-05-08T21:23:24.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.202+0000 7f76f1384700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76ec0727b0 0x7f76ec15d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.202+0000 7f76f1384700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76ec0ca130 0x7f76ec1f0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.202+0000 7f76f1384700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76ec15a7c0 0x7f76ec147f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.202+0000 7f76f1384700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec0ae660 con 0x7f76ec0ca130 2024-05-08T21:23:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.202+0000 7f76f1384700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec1f0db0 con 0x7f76ec15a7c0 2024-05-08T21:23:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.202+0000 7f76f1384700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec148480 con 0x7f76ec0727b0 2024-05-08T21:23:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76ea7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76ec0ca130 0x7f76ec1f0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76ea7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76ec0ca130 0x7f76ec1f0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42024/0 (socket says 172.21.15.3:42024) 2024-05-08T21:23:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76ea7fc700 1 -- 172.21.15.3:0/3259754247 learned_addr learned my addr 172.21.15.3:0/3259754247 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76eb7fe700 1 --2- 172.21.15.3:0/3259754247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76ec15a7c0 0x7f76ec147f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76eaffd700 1 --2- 172.21.15.3:0/3259754247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76ec0727b0 0x7f76ec15d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76ea7fc700 1 -- 172.21.15.3:0/3259754247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76ec0727b0 msgr2=0x7f76ec15d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76ea7fc700 1 --2- 172.21.15.3:0/3259754247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76ec0727b0 0x7f76ec15d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76ea7fc700 1 -- 172.21.15.3:0/3259754247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76ec15a7c0 msgr2=0x7f76ec147f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76ea7fc700 1 --2- 172.21.15.3:0/3259754247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76ec15a7c0 0x7f76ec147f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76ea7fc700 1 -- 172.21.15.3:0/3259754247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e4009a70 con 0x7f76ec0ca130 2024-05-08T21:23:24.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76eb7fe700 1 --2- 172.21.15.3:0/3259754247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76ec15a7c0 0x7f76ec147f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:24.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76eaffd700 1 --2- 172.21.15.3:0/3259754247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76ec0727b0 0x7f76ec15d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:24.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76ea7fc700 1 --2- 172.21.15.3:0/3259754247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76ec0ca130 0x7f76ec1f0870 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f76e000c540 tx=0x7f76e000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:24.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76d37fe700 1 -- 172.21.15.3:0/3259754247 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e0004c80 con 0x7f76ec0ca130 2024-05-08T21:23:24.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76d37fe700 1 -- 172.21.15.3:0/3259754247 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e0004de0 con 0x7f76ec0ca130 2024-05-08T21:23:24.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76f1384700 1 -- 172.21.15.3:0/3259754247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ec148720 con 0x7f76ec0ca130 2024-05-08T21:23:24.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76d37fe700 1 -- 172.21.15.3:0/3259754247 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e0007310 con 0x7f76ec0ca130 2024-05-08T21:23:24.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.203+0000 7f76f1384700 1 -- 172.21.15.3:0/3259754247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ec148ce0 con 0x7f76ec0ca130 2024-05-08T21:23:24.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.204+0000 7f76f1384700 1 -- 172.21.15.3:0/3259754247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76ec074bc0 con 0x7f76ec0ca130 2024-05-08T21:23:24.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.205+0000 7f76d37fe700 1 -- 172.21.15.3:0/3259754247 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f76e0007470 con 0x7f76ec0ca130 2024-05-08T21:23:24.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.205+0000 7f76d37fe700 1 --2- 172.21.15.3:0/3259754247 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f76d806ac90 0x7f76d806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.205+0000 7f76d37fe700 1 -- 172.21.15.3:0/3259754247 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f76e0089320 con 0x7f76ec0ca130 2024-05-08T21:23:24.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.206+0000 7f76eaffd700 1 --2- 172.21.15.3:0/3259754247 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f76d806ac90 0x7f76d806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.206+0000 7f76eaffd700 1 --2- 172.21.15.3:0/3259754247 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f76d806ac90 0x7f76d806d150 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f76dc007a50 tx=0x7f76dc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:24.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.210+0000 7f76d37fe700 1 -- 172.21.15.3:0/3259754247 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f76e0054260 con 0x7f76ec0ca130 2024-05-08T21:23:24.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.425+0000 7f76f1384700 1 -- 172.21.15.3:0/3259754247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f76ec0ced00 con 0x7f76ec0ca130 2024-05-08T21:23:24.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.425+0000 7f76d37fe700 1 -- 172.21.15.3:0/3259754247 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+11 (secure 0 0 0) 0x7f76e0057880 con 0x7f76ec0ca130 2024-05-08T21:23:24.427 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:23:24.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.428+0000 7f76f1384700 1 -- 172.21.15.3:0/3259754247 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f76d806ac90 msgr2=0x7f76d806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.428+0000 7f76f1384700 1 --2- 172.21.15.3:0/3259754247 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f76d806ac90 0x7f76d806d150 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f76dc007a50 tx=0x7f76dc008040 comp rx=0 tx=0).stop 2024-05-08T21:23:24.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.428+0000 7f76f1384700 1 -- 172.21.15.3:0/3259754247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76ec0ca130 msgr2=0x7f76ec1f0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.428+0000 7f76f1384700 1 --2- 172.21.15.3:0/3259754247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76ec0ca130 0x7f76ec1f0870 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f76e000c540 tx=0x7f76e000c900 comp rx=0 tx=0).stop 2024-05-08T21:23:24.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.428+0000 7f76f1384700 1 -- 172.21.15.3:0/3259754247 shutdown_connections 2024-05-08T21:23:24.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.428+0000 7f76f1384700 1 --2- 172.21.15.3:0/3259754247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76ec0727b0 0x7f76ec15d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.428+0000 7f76f1384700 1 --2- 172.21.15.3:0/3259754247 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f76d806ac90 0x7f76d806d150 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.428+0000 7f76f1384700 1 --2- 172.21.15.3:0/3259754247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76ec15a7c0 0x7f76ec147f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.428+0000 7f76f1384700 1 --2- 172.21.15.3:0/3259754247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76ec0ca130 0x7f76ec1f0870 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.428+0000 7f76f1384700 1 -- 172.21.15.3:0/3259754247 >> 172.21.15.3:0/3259754247 conn(0x7f76ec06a960 msgr2=0x7f76ec06db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:24.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.428+0000 7f76f1384700 1 -- 172.21.15.3:0/3259754247 shutdown_connections 2024-05-08T21:23:24.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.428+0000 7f76f1384700 1 -- 172.21.15.3:0/3259754247 wait complete. 2024-05-08T21:23:24.443 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:23:24.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.581+0000 7f6231685700 1 Processor -- start 2024-05-08T21:23:24.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.581+0000 7f6231685700 1 -- start start 2024-05-08T21:23:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.582+0000 7f6231685700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f622c15ed30 0x7f622c0b4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.582+0000 7f6231685700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f622c0cfe80 0x7f622c15f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.582+0000 7f6231685700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622c0cf000 con 0x7f622c0b22d0 2024-05-08T21:23:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.582+0000 7f6231685700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f622c09ac90 con 0x7f622c0cfe80 2024-05-08T21:23:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.582+0000 7f6231685700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f622c0ccc80 con 0x7f622c15ed30 2024-05-08T21:23:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.582+0000 7f622b7fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f622c0cfe80 0x7f622c15f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.582+0000 7f622affd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f622c15ed30 0x7f622c0b4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.583+0000 7f622b7fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f622c0cfe80 0x7f622c15f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:42402/0 (socket says 172.21.15.3:42402) 2024-05-08T21:23:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.583+0000 7f622b7fe700 1 -- 172.21.15.3:0/303704492 learned_addr learned my addr 172.21.15.3:0/303704492 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.583+0000 7f622affd700 1 -- 172.21.15.3:0/303704492 >> v2:172.21.15.186:3300/0 conn(0x7f622c0cfe80 msgr2=0x7f622c15f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.583+0000 7f622affd700 1 --2- 172.21.15.3:0/303704492 >> v2:172.21.15.186:3300/0 conn(0x7f622c0cfe80 0x7f622c15f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.583+0000 7f622affd700 1 -- 172.21.15.3:0/303704492 >> v1:172.21.15.186:6789/0 conn(0x7f622c0b22d0 legacy=0x7f622c0cf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.583+0000 7f622affd700 1 -- 172.21.15.3:0/303704492 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c0ccf60 con 0x7f622c15ed30 2024-05-08T21:23:24.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.583+0000 7f622affd700 1 --2- 172.21.15.3:0/303704492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f622c15ed30 0x7f622c0b4280 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f621c009840 tx=0x7f621c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ea80a5239dd4f93 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:24.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.583+0000 7f6229ffb700 1 -- 172.21.15.3:0/303704492 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c011830 con 0x7f622c15ed30 2024-05-08T21:23:24.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.583+0000 7f6229ffb700 1 -- 172.21.15.3:0/303704492 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621c011990 con 0x7f622c15ed30 2024-05-08T21:23:24.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.583+0000 7f6229ffb700 1 -- 172.21.15.3:0/303704492 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c011d20 con 0x7f622c15ed30 2024-05-08T21:23:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.584+0000 7f6231685700 1 -- 172.21.15.3:0/303704492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f622c15ed30 msgr2=0x7f622c0b4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.584+0000 7f6231685700 1 --2- 172.21.15.3:0/303704492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f622c15ed30 0x7f622c0b4280 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f621c009840 tx=0x7f621c009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.584+0000 7f6231685700 1 -- 172.21.15.3:0/303704492 shutdown_connections 2024-05-08T21:23:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.584+0000 7f6231685700 1 --2- 172.21.15.3:0/303704492 >> v2:172.21.15.186:3300/0 conn(0x7f622c0cfe80 0x7f622c15f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.584+0000 7f6231685700 1 --2- 172.21.15.3:0/303704492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f622c15ed30 0x7f622c0b4280 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.584+0000 7f6231685700 1 -- 172.21.15.3:0/303704492 >> 172.21.15.3:0/303704492 conn(0x7f622c0b2f20 msgr2=0x7f622c0b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.584+0000 7f6231685700 1 -- 172.21.15.3:0/303704492 shutdown_connections 2024-05-08T21:23:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.584+0000 7f6231685700 1 -- 172.21.15.3:0/303704492 wait complete. 2024-05-08T21:23:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.585+0000 7f6231685700 1 Processor -- start 2024-05-08T21:23:24.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.586+0000 7f6231685700 1 -- start start 2024-05-08T21:23:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.586+0000 7f6231685700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f622c0b22d0 0x7f622c0c5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.586+0000 7f6231685700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f622c0cfe80 0x7f622c0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.586+0000 7f6231685700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f622c15ed30 0x7f622c0c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.586+0000 7f6231685700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c154860 con 0x7f622c15ed30 2024-05-08T21:23:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.586+0000 7f6231685700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c151c10 con 0x7f622c0b22d0 2024-05-08T21:23:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.586+0000 7f6231685700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f622c0c5eb0 con 0x7f622c0cfe80 2024-05-08T21:23:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.586+0000 7f622b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f622c15ed30 0x7f622c0c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.586+0000 7f622b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f622c15ed30 0x7f622c0c66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42030/0 (socket says 172.21.15.3:42030) 2024-05-08T21:23:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.586+0000 7f622b7fe700 1 -- 172.21.15.3:0/4004700255 learned_addr learned my addr 172.21.15.3:0/4004700255 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.587+0000 7f622affd700 1 --2- 172.21.15.3:0/4004700255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f622c0b22d0 0x7f622c0c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.587+0000 7f622a7fc700 1 --2- 172.21.15.3:0/4004700255 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f622c0cfe80 0x7f622c0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.587+0000 7f622b7fe700 1 -- 172.21.15.3:0/4004700255 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f622c0cfe80 msgr2=0x7f622c0c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.587+0000 7f622b7fe700 1 --2- 172.21.15.3:0/4004700255 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f622c0cfe80 0x7f622c0c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.587+0000 7f622b7fe700 1 -- 172.21.15.3:0/4004700255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f622c0b22d0 msgr2=0x7f622c0c5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.587+0000 7f622b7fe700 1 --2- 172.21.15.3:0/4004700255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f622c0b22d0 0x7f622c0c5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.587+0000 7f622b7fe700 1 -- 172.21.15.3:0/4004700255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621c0094a0 con 0x7f622c15ed30 2024-05-08T21:23:24.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.587+0000 7f622b7fe700 1 --2- 172.21.15.3:0/4004700255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f622c15ed30 0x7f622c0c66f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f622400efc0 tx=0x7f622400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:24.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.587+0000 7f62137fe700 1 -- 172.21.15.3:0/4004700255 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6224003c20 con 0x7f622c15ed30 2024-05-08T21:23:24.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.587+0000 7f6231685700 1 -- 172.21.15.3:0/4004700255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f622c06caf0 con 0x7f622c15ed30 2024-05-08T21:23:24.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.587+0000 7f62137fe700 1 -- 172.21.15.3:0/4004700255 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622400f040 con 0x7f622c15ed30 2024-05-08T21:23:24.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.587+0000 7f62137fe700 1 -- 172.21.15.3:0/4004700255 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6224009e60 con 0x7f622c15ed30 2024-05-08T21:23:24.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.587+0000 7f6231685700 1 -- 172.21.15.3:0/4004700255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f622c0c9940 con 0x7f622c15ed30 2024-05-08T21:23:24.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.588+0000 7f6231685700 1 -- 172.21.15.3:0/4004700255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f622c074bc0 con 0x7f622c15ed30 2024-05-08T21:23:24.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.589+0000 7f62137fe700 1 -- 172.21.15.3:0/4004700255 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f622400c880 con 0x7f622c15ed30 2024-05-08T21:23:24.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.589+0000 7f62137fe700 1 --2- 172.21.15.3:0/4004700255 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f621806ac90 0x7f621806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.589+0000 7f62137fe700 1 -- 172.21.15.3:0/4004700255 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f62240897a0 con 0x7f622c15ed30 2024-05-08T21:23:24.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.589+0000 7f622affd700 1 --2- 172.21.15.3:0/4004700255 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f621806ac90 0x7f621806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.590+0000 7f622affd700 1 --2- 172.21.15.3:0/4004700255 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f621806ac90 0x7f621806d150 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f621c000c00 tx=0x7f621c008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:24.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.594+0000 7f62137fe700 1 -- 172.21.15.3:0/4004700255 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6224054770 con 0x7f622c15ed30 2024-05-08T21:23:24.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.809+0000 7f6231685700 1 -- 172.21.15.3:0/4004700255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f622c0b3a40 con 0x7f622c15ed30 2024-05-08T21:23:24.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.810+0000 7f62137fe700 1 -- 172.21.15.3:0/4004700255 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6224057d90 con 0x7f622c15ed30 2024-05-08T21:23:24.811 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:23:24.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.812+0000 7f6231685700 1 -- 172.21.15.3:0/4004700255 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f621806ac90 msgr2=0x7f621806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.812+0000 7f6231685700 1 --2- 172.21.15.3:0/4004700255 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f621806ac90 0x7f621806d150 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f621c000c00 tx=0x7f621c008030 comp rx=0 tx=0).stop 2024-05-08T21:23:24.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.812+0000 7f6231685700 1 -- 172.21.15.3:0/4004700255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f622c15ed30 msgr2=0x7f622c0c66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.812+0000 7f6231685700 1 --2- 172.21.15.3:0/4004700255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f622c15ed30 0x7f622c0c66f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f622400efc0 tx=0x7f622400c3a0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.812+0000 7f6231685700 1 -- 172.21.15.3:0/4004700255 shutdown_connections 2024-05-08T21:23:24.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.812+0000 7f6231685700 1 --2- 172.21.15.3:0/4004700255 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f621806ac90 0x7f621806d150 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.812+0000 7f6231685700 1 --2- 172.21.15.3:0/4004700255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f622c15ed30 0x7f622c0c66f0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.812+0000 7f6231685700 1 --2- 172.21.15.3:0/4004700255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f622c0b22d0 0x7f622c0c5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.812+0000 7f6231685700 1 --2- 172.21.15.3:0/4004700255 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f622c0cfe80 0x7f622c0c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.812+0000 7f6231685700 1 -- 172.21.15.3:0/4004700255 >> 172.21.15.3:0/4004700255 conn(0x7f622c0b2f20 msgr2=0x7f622c0cf670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:24.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.812+0000 7f6231685700 1 -- 172.21.15.3:0/4004700255 shutdown_connections 2024-05-08T21:23:24.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.812+0000 7f6231685700 1 -- 172.21.15.3:0/4004700255 wait complete. 2024-05-08T21:23:24.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get recovery_fs_metadata pg_num 2024-05-08T21:23:24.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.965+0000 7f2dafafe700 1 Processor -- start 2024-05-08T21:23:24.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.965+0000 7f2dafafe700 1 -- start start 2024-05-08T21:23:24.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.966+0000 7f2dafafe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2da806fa50 0x7f2da80cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.966+0000 7f2dafafe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2da8162290 0x7f2da806b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.966+0000 7f2dafafe700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da80cda40 con 0x7f2da806f670 2024-05-08T21:23:24.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.966+0000 7f2dafafe700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da8162790 con 0x7f2da8162290 2024-05-08T21:23:24.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.966+0000 7f2dafafe700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da8162a70 con 0x7f2da806fa50 2024-05-08T21:23:24.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.966+0000 7f2dae09b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2da8162290 0x7f2da806b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.966+0000 7f2dad099700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2da806fa50 0x7f2da80cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.966+0000 7f2dae09b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2da8162290 0x7f2da806b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:42432/0 (socket says 172.21.15.3:42432) 2024-05-08T21:23:24.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.966+0000 7f2dae09b700 1 -- 172.21.15.3:0/2404191696 learned_addr learned my addr 172.21.15.3:0/2404191696 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:24.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.967+0000 7f2dae09b700 1 -- 172.21.15.3:0/2404191696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2da806fa50 msgr2=0x7f2da80cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:24.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.967+0000 7f2dae09b700 1 --2- 172.21.15.3:0/2404191696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2da806fa50 0x7f2da80cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.967+0000 7f2dae09b700 1 -- 172.21.15.3:0/2404191696 >> v1:172.21.15.186:6789/0 conn(0x7f2da806f670 legacy=0x7f2da80b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.967+0000 7f2dae09b700 1 -- 172.21.15.3:0/2404191696 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da8162d50 con 0x7f2da8162290 2024-05-08T21:23:24.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.967+0000 7f2dad099700 1 --2- 172.21.15.3:0/2404191696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2da806fa50 0x7f2da80cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:24.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.967+0000 7f2dae09b700 1 --2- 172.21.15.3:0/2404191696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2da8162290 0x7f2da806b970 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009e50 tx=0x7f2da000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6c83a17eb9e6f54 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:24.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.967+0000 7f2dac898700 1 -- 172.21.15.3:0/2404191696 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da000e070 con 0x7f2da8162290 2024-05-08T21:23:24.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.967+0000 7f2dac898700 1 -- 172.21.15.3:0/2404191696 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da000b910 con 0x7f2da8162290 2024-05-08T21:23:24.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.967+0000 7f2dac898700 1 -- 172.21.15.3:0/2404191696 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da000bcc0 con 0x7f2da8162290 2024-05-08T21:23:24.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.968+0000 7f2dafafe700 1 -- 172.21.15.3:0/2404191696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2da8162290 msgr2=0x7f2da806b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.968+0000 7f2dafafe700 1 --2- 172.21.15.3:0/2404191696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2da8162290 0x7f2da806b970 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009e50 tx=0x7f2da000b330 comp rx=0 tx=0).stop 2024-05-08T21:23:24.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.968+0000 7f2dafafe700 1 -- 172.21.15.3:0/2404191696 shutdown_connections 2024-05-08T21:23:24.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.968+0000 7f2dafafe700 1 --2- 172.21.15.3:0/2404191696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2da8162290 0x7f2da806b970 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.968+0000 7f2dafafe700 1 --2- 172.21.15.3:0/2404191696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2da806fa50 0x7f2da80cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.968+0000 7f2dafafe700 1 -- 172.21.15.3:0/2404191696 >> 172.21.15.3:0/2404191696 conn(0x7f2da80713c0 msgr2=0x7f2da80717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:24.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.968+0000 7f2dafafe700 1 -- 172.21.15.3:0/2404191696 shutdown_connections 2024-05-08T21:23:24.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.968+0000 7f2dafafe700 1 -- 172.21.15.3:0/2404191696 wait complete. 2024-05-08T21:23:24.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.969+0000 7f2dafafe700 1 Processor -- start 2024-05-08T21:23:24.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.969+0000 7f2dafafe700 1 -- start start 2024-05-08T21:23:24.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.970+0000 7f2dafafe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2da806f670 0x7f2da815dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.970+0000 7f2dafafe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2da806fa50 0x7f2da8161010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.970+0000 7f2dafafe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2da8162290 0x7f2da814f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.970+0000 7f2dafafe700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da815e4d0 con 0x7f2da806fa50 2024-05-08T21:23:24.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.970+0000 7f2dafafe700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da815e7b0 con 0x7f2da8162290 2024-05-08T21:23:24.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.970+0000 7f2dafafe700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da815ea90 con 0x7f2da806f670 2024-05-08T21:23:24.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.970+0000 7f2dad099700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2da806fa50 0x7f2da8161010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.970+0000 7f2dad099700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2da806fa50 0x7f2da8161010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42044/0 (socket says 172.21.15.3:42044) 2024-05-08T21:23:24.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.970+0000 7f2dad099700 1 -- 172.21.15.3:0/1638626311 learned_addr learned my addr 172.21.15.3:0/1638626311 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:24.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.970+0000 7f2dae09b700 1 --2- 172.21.15.3:0/1638626311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2da8162290 0x7f2da814f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.970+0000 7f2dad89a700 1 --2- 172.21.15.3:0/1638626311 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2da806f670 0x7f2da815dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.971+0000 7f2dad099700 1 -- 172.21.15.3:0/1638626311 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2da806f670 msgr2=0x7f2da815dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.971+0000 7f2dad099700 1 --2- 172.21.15.3:0/1638626311 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2da806f670 0x7f2da815dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.971+0000 7f2dad099700 1 -- 172.21.15.3:0/1638626311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2da8162290 msgr2=0x7f2da814f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:24.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.971+0000 7f2dad099700 1 --2- 172.21.15.3:0/1638626311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2da8162290 0x7f2da814f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:24.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.971+0000 7f2dad099700 1 -- 172.21.15.3:0/1638626311 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da40094a0 con 0x7f2da806fa50 2024-05-08T21:23:24.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.971+0000 7f2dad099700 1 --2- 172.21.15.3:0/1638626311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2da806fa50 0x7f2da8161010 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2da400ebe0 tx=0x7f2da400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:24.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.971+0000 7f2d9a7fc700 1 -- 172.21.15.3:0/1638626311 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da40097a0 con 0x7f2da806fa50 2024-05-08T21:23:24.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.971+0000 7f2d9a7fc700 1 -- 172.21.15.3:0/1638626311 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da400c850 con 0x7f2da806fa50 2024-05-08T21:23:24.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.971+0000 7f2dafafe700 1 -- 172.21.15.3:0/1638626311 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da0009b30 con 0x7f2da806fa50 2024-05-08T21:23:24.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.971+0000 7f2d9a7fc700 1 -- 172.21.15.3:0/1638626311 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da400cbe0 con 0x7f2da806fa50 2024-05-08T21:23:24.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.971+0000 7f2dafafe700 1 -- 172.21.15.3:0/1638626311 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da815f050 con 0x7f2da806fa50 2024-05-08T21:23:24.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.972+0000 7f2dafafe700 1 -- 172.21.15.3:0/1638626311 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da8074bc0 con 0x7f2da806fa50 2024-05-08T21:23:24.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.973+0000 7f2d9a7fc700 1 -- 172.21.15.3:0/1638626311 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2da400cd40 con 0x7f2da806fa50 2024-05-08T21:23:24.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.973+0000 7f2d9a7fc700 1 --2- 172.21.15.3:0/1638626311 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2d8c06ac90 0x7f2d8c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:24.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.973+0000 7f2d9a7fc700 1 -- 172.21.15.3:0/1638626311 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f2da4089250 con 0x7f2da806fa50 2024-05-08T21:23:24.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.973+0000 7f2dad89a700 1 --2- 172.21.15.3:0/1638626311 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2d8c06ac90 0x7f2d8c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:24.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.974+0000 7f2dad89a700 1 --2- 172.21.15.3:0/1638626311 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2d8c06ac90 0x7f2d8c06d150 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c007fd0 tx=0x7f2d9c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:24.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:24.978+0000 7f2d9a7fc700 1 -- 172.21.15.3:0/1638626311 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2da4054220 con 0x7f2da806fa50 2024-05-08T21:23:25.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.190+0000 7f2dafafe700 1 -- 172.21.15.3:0/1638626311 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2da80cc420 con 0x7f2da806fa50 2024-05-08T21:23:25.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.191+0000 7f2d9a7fc700 1 -- 172.21.15.3:0/1638626311 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 109+0+11 (secure 0 0 0) 0x7f2da4057840 con 0x7f2da806fa50 2024-05-08T21:23:25.193 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:23:25.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.194+0000 7f2dafafe700 1 -- 172.21.15.3:0/1638626311 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2d8c06ac90 msgr2=0x7f2d8c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.194+0000 7f2dafafe700 1 --2- 172.21.15.3:0/1638626311 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2d8c06ac90 0x7f2d8c06d150 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c007fd0 tx=0x7f2d9c009040 comp rx=0 tx=0).stop 2024-05-08T21:23:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.194+0000 7f2dafafe700 1 -- 172.21.15.3:0/1638626311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2da806fa50 msgr2=0x7f2da8161010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.194+0000 7f2dafafe700 1 --2- 172.21.15.3:0/1638626311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2da806fa50 0x7f2da8161010 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2da400ebe0 tx=0x7f2da400eef0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.194+0000 7f2dafafe700 1 -- 172.21.15.3:0/1638626311 shutdown_connections 2024-05-08T21:23:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.194+0000 7f2dafafe700 1 --2- 172.21.15.3:0/1638626311 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2da806f670 0x7f2da815dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.194+0000 7f2dafafe700 1 --2- 172.21.15.3:0/1638626311 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f2d8c06ac90 0x7f2d8c06d150 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.194+0000 7f2dafafe700 1 --2- 172.21.15.3:0/1638626311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2da8162290 0x7f2da814f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.194+0000 7f2dafafe700 1 --2- 172.21.15.3:0/1638626311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2da806fa50 0x7f2da8161010 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.194+0000 7f2dafafe700 1 -- 172.21.15.3:0/1638626311 >> 172.21.15.3:0/1638626311 conn(0x7f2da80713c0 msgr2=0x7f2da80b29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.194+0000 7f2dafafe700 1 -- 172.21.15.3:0/1638626311 shutdown_connections 2024-05-08T21:23:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.194+0000 7f2dafafe700 1 -- 172.21.15.3:0/1638626311 wait complete. 2024-05-08T21:23:25.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:23:25.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.344+0000 7f325a1e1700 1 Processor -- start 2024-05-08T21:23:25.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.344+0000 7f325a1e1700 1 -- start start 2024-05-08T21:23:25.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.345+0000 7f325a1e1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32540cd4e0 0x7f325406f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.345+0000 7f325a1e1700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325415c280 con 0x7f325406f130 2024-05-08T21:23:25.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.345+0000 7f325a1e1700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325406e780 con 0x7f32540cecf0 2024-05-08T21:23:25.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.345+0000 7f325a1e1700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325406ea60 con 0x7f32540cd4e0 2024-05-08T21:23:25.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.345+0000 7f324affd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f325406f130 0x7f32540ce5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50470/0 (socket says 172.21.15.3:50470) 2024-05-08T21:23:25.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.345+0000 7f324affd700 1 -- 172.21.15.3:0/4143251906 learned_addr learned my addr 172.21.15.3:0/4143251906 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:25.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.345+0000 7f32537fe700 1 --2- 172.21.15.3:0/4143251906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32540cd4e0 0x7f325406f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.345+0000 7f3252ffd700 1 -- 172.21.15.3:0/4143251906 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324294095 0 0) 0x7f325415c280 con 0x7f325406f130 2024-05-08T21:23:25.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.345+0000 7f3252ffd700 1 -- 172.21.15.3:0/4143251906 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32400035b0 con 0x7f325406f130 2024-05-08T21:23:25.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f3252ffd700 1 -- 172.21.15.3:0/4143251906 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3452102114 0 0) 0x7f32400035b0 con 0x7f325406f130 2024-05-08T21:23:25.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f3252ffd700 1 -- 172.21.15.3:0/4143251906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32540cd4e0 msgr2=0x7f325406f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:25.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f3252ffd700 1 --2- 172.21.15.3:0/4143251906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32540cd4e0 0x7f325406f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f3252ffd700 1 -- 172.21.15.3:0/4143251906 >> v1:172.21.15.186:6789/0 conn(0x7f32540cecf0 legacy=0x7f325415c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:25.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f3252ffd700 1 -- 172.21.15.3:0/4143251906 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325406ed40 con 0x7f325406f130 2024-05-08T21:23:25.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f3252ffd700 1 -- 172.21.15.3:0/4143251906 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f323c001670 con 0x7f325406f130 2024-05-08T21:23:25.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f3252ffd700 1 -- 172.21.15.3:0/4143251906 >> v1:172.21.15.3:6789/0 conn(0x7f325406f130 legacy=0x7f32540ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:25.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f3252ffd700 1 --2- 172.21.15.3:0/4143251906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32540cd4e0 0x7f3240003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f3252ffd700 1 --2- 172.21.15.3:0/4143251906 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3240003f70 0x7f3240004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f3252ffd700 1 --2- 172.21.15.3:0/4143251906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3240004d90 0x7f3240005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f3252ffd700 1 -- 172.21.15.3:0/4143251906 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325406ea60 con 0x7f3240003f70 2024-05-08T21:23:25.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f3252ffd700 1 -- 172.21.15.3:0/4143251906 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324000ba70 con 0x7f3240004d90 2024-05-08T21:23:25.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f3252ffd700 1 -- 172.21.15.3:0/4143251906 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f324000de40 con 0x7f32540cd4e0 2024-05-08T21:23:25.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f324affd700 1 --2- 172.21.15.3:0/4143251906 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3240003f70 0x7f3240004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f32537fe700 1 --2- 172.21.15.3:0/4143251906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32540cd4e0 0x7f3240003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f3253fff700 1 --2- 172.21.15.3:0/4143251906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3240004d90 0x7f3240005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f324affd700 1 -- 172.21.15.3:0/4143251906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32540cd4e0 msgr2=0x7f3240003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:25.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f324affd700 1 --2- 172.21.15.3:0/4143251906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32540cd4e0 0x7f3240003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f324affd700 1 -- 172.21.15.3:0/4143251906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3240004d90 msgr2=0x7f3240005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:25.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f324affd700 1 --2- 172.21.15.3:0/4143251906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3240004d90 0x7f3240005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f324affd700 1 -- 172.21.15.3:0/4143251906 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32400035b0 con 0x7f3240003f70 2024-05-08T21:23:25.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.346+0000 7f324affd700 1 --2- 172.21.15.3:0/4143251906 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3240003f70 0x7f3240004850 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f323c00c240 tx=0x7f3240005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f90366b2b898213 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:25.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.347+0000 7f3252ffd700 1 -- 172.21.15.3:0/4143251906 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c00ccc0 con 0x7f3240003f70 2024-05-08T21:23:25.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.347+0000 7f3252ffd700 1 -- 172.21.15.3:0/4143251906 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323c00ce20 con 0x7f3240003f70 2024-05-08T21:23:25.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.347+0000 7f3252ffd700 1 -- 172.21.15.3:0/4143251906 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c005b80 con 0x7f3240003f70 2024-05-08T21:23:25.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.347+0000 7f325a1e1700 1 -- 172.21.15.3:0/4143251906 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3240003f70 msgr2=0x7f3240004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:25.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.347+0000 7f325a1e1700 1 --2- 172.21.15.3:0/4143251906 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3240003f70 0x7f3240004850 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f323c00c240 tx=0x7f3240005880 comp rx=0 tx=0).stop 2024-05-08T21:23:25.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.347+0000 7f3253fff700 1 -- 172.21.15.3:0/4143251906 reap_dead start 2024-05-08T21:23:25.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.347+0000 7f325a1e1700 1 -- 172.21.15.3:0/4143251906 shutdown_connections 2024-05-08T21:23:25.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.347+0000 7f325a1e1700 1 -- 172.21.15.3:0/4143251906 >> 172.21.15.3:0/4143251906 conn(0x7f32540d0320 msgr2=0x7f32540d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:25.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.347+0000 7f325a1e1700 1 -- 172.21.15.3:0/4143251906 shutdown_connections 2024-05-08T21:23:25.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.347+0000 7f325a1e1700 1 -- 172.21.15.3:0/4143251906 wait complete. 2024-05-08T21:23:25.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.348+0000 7f325a1e1700 1 Processor -- start 2024-05-08T21:23:25.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.349+0000 7f325a1e1700 1 -- start start 2024-05-08T21:23:25.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.349+0000 7f325a1e1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3240003f70 0x7f32540b2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.349+0000 7f325a1e1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32540b01d0 0x7f325415d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f325a1e1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325415db80 0x7f325415e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f325a1e1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32540afd40 con 0x7f3240003f70 2024-05-08T21:23:25.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f325a1e1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32540b2c40 con 0x7f325415db80 2024-05-08T21:23:25.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f325a1e1700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32540b07f0 con 0x7f32540b01d0 2024-05-08T21:23:25.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f32537fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3240003f70 0x7f32540b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f32537fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3240003f70 0x7f32540b2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42058/0 (socket says 172.21.15.3:42058) 2024-05-08T21:23:25.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f32537fe700 1 -- 172.21.15.3:0/2433000133 learned_addr learned my addr 172.21.15.3:0/2433000133 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:25.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f324affd700 1 --2- 172.21.15.3:0/2433000133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32540b01d0 0x7f325415d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f3253fff700 1 --2- 172.21.15.3:0/2433000133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325415db80 0x7f325415e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f32537fe700 1 -- 172.21.15.3:0/2433000133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32540b01d0 msgr2=0x7f325415d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:25.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f32537fe700 1 --2- 172.21.15.3:0/2433000133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32540b01d0 0x7f325415d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f32537fe700 1 -- 172.21.15.3:0/2433000133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325415db80 msgr2=0x7f325415e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:25.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f32537fe700 1 --2- 172.21.15.3:0/2433000133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325415db80 0x7f325415e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f32537fe700 1 -- 172.21.15.3:0/2433000133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32540b0ad0 con 0x7f3240003f70 2024-05-08T21:23:25.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f32537fe700 1 --2- 172.21.15.3:0/2433000133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3240003f70 0x7f32540b2700 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3244011aa0 tx=0x7f3244011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:25.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f3250ff9700 1 -- 172.21.15.3:0/2433000133 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244011ee0 con 0x7f3240003f70 2024-05-08T21:23:25.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f3250ff9700 1 -- 172.21.15.3:0/2433000133 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324400f940 con 0x7f3240003f70 2024-05-08T21:23:25.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f3250ff9700 1 -- 172.21.15.3:0/2433000133 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324400fcf0 con 0x7f3240003f70 2024-05-08T21:23:25.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.350+0000 7f325a1e1700 1 -- 172.21.15.3:0/2433000133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32540b0db0 con 0x7f3240003f70 2024-05-08T21:23:25.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.351+0000 7f325a1e1700 1 -- 172.21.15.3:0/2433000133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3254150370 con 0x7f3240003f70 2024-05-08T21:23:25.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.351+0000 7f325a1e1700 1 -- 172.21.15.3:0/2433000133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32540b0fd0 con 0x7f3240003f70 2024-05-08T21:23:25.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.352+0000 7f3250ff9700 1 -- 172.21.15.3:0/2433000133 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3244014440 con 0x7f3240003f70 2024-05-08T21:23:25.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.352+0000 7f3250ff9700 1 --2- 172.21.15.3:0/2433000133 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f323406ac90 0x7f323406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.353+0000 7f3250ff9700 1 -- 172.21.15.3:0/2433000133 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f3244089b30 con 0x7f3240003f70 2024-05-08T21:23:25.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.353+0000 7f324affd700 1 --2- 172.21.15.3:0/2433000133 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f323406ac90 0x7f323406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.353+0000 7f324affd700 1 --2- 172.21.15.3:0/2433000133 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f323406ac90 0x7f323406d150 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f32540c7fd0 tx=0x7f323c007000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:25.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.357+0000 7f3250ff9700 1 -- 172.21.15.3:0/2433000133 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f3244054c40 con 0x7f3240003f70 2024-05-08T21:23:25.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.625+0000 7f325a1e1700 1 -- 172.21.15.3:0/2433000133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f325406e780 con 0x7f3240003f70 2024-05-08T21:23:25.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.625+0000 7f3250ff9700 1 -- 172.21.15.3:0/2433000133 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5936 (secure 0 0 0) 0x7f3244022020 con 0x7f3240003f70 2024-05-08T21:23:25.627 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:25.627 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6833/2104267343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2104267343},{"type":"v1","addr":"172.21.15.3:6833","nonce":2104267343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/208321708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":208321708},{"type":"v1","addr":"172.21.15.3:6835","nonce":208321708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/1591386131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1591386131},{"type":"v1","addr":"172.21.15.186:6836","nonce":1591386131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:22:50.184245+0000","modified":"2024-05-08T21:23:01.563810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:23:05.475355+0000","modified":"2024-05-08T21:23:15.662101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6837/2000072997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":2000072997},{"type":"v1","addr":"172.21.15.186:6837","nonce":2000072997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":6,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-08T21:23:25.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.628+0000 7f325a1e1700 1 -- 172.21.15.3:0/2433000133 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f323406ac90 msgr2=0x7f323406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:25.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.628+0000 7f325a1e1700 1 --2- 172.21.15.3:0/2433000133 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f323406ac90 0x7f323406d150 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f32540c7fd0 tx=0x7f323c007000 comp rx=0 tx=0).stop 2024-05-08T21:23:25.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.628+0000 7f325a1e1700 1 -- 172.21.15.3:0/2433000133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3240003f70 msgr2=0x7f32540b2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:25.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.628+0000 7f325a1e1700 1 --2- 172.21.15.3:0/2433000133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3240003f70 0x7f32540b2700 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3244011aa0 tx=0x7f3244011db0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.628+0000 7f325a1e1700 1 -- 172.21.15.3:0/2433000133 shutdown_connections 2024-05-08T21:23:25.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.628+0000 7f325a1e1700 1 --2- 172.21.15.3:0/2433000133 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f323406ac90 0x7f323406d150 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.628+0000 7f325a1e1700 1 --2- 172.21.15.3:0/2433000133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325415db80 0x7f325415e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.628+0000 7f325a1e1700 1 --2- 172.21.15.3:0/2433000133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3240003f70 0x7f32540b2700 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.628+0000 7f325a1e1700 1 --2- 172.21.15.3:0/2433000133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32540b01d0 0x7f325415d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.628+0000 7f325a1e1700 1 -- 172.21.15.3:0/2433000133 >> 172.21.15.3:0/2433000133 conn(0x7f32540d0320 msgr2=0x7f32540aea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:25.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.628+0000 7f325a1e1700 1 -- 172.21.15.3:0/2433000133 shutdown_connections 2024-05-08T21:23:25.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.628+0000 7f325a1e1700 1 -- 172.21.15.3:0/2433000133 wait complete. 2024-05-08T21:23:25.631 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 26 2024-05-08T21:23:25.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:23:25.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.777+0000 7ff4d3356700 1 Processor -- start 2024-05-08T21:23:25.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.777+0000 7ff4d3356700 1 -- start start 2024-05-08T21:23:25.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.778+0000 7ff4d3356700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4cc15a750 0x7ff4cc0d0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.778+0000 7ff4d3356700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff4cc0c5720 0x7ff4cc0c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.778+0000 7ff4d3356700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4cc0cfbd0 con 0x7ff4cc0af860 2024-05-08T21:23:25.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.778+0000 7ff4d3356700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4cc0c60f0 con 0x7ff4cc0c5720 2024-05-08T21:23:25.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.778+0000 7ff4d3356700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc0cf440 con 0x7ff4cc15a750 2024-05-08T21:23:25.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.778+0000 7ff4d08f1700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff4cc0af860 0x7ff4cc0ad260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50476/0 (socket says 172.21.15.3:50476) 2024-05-08T21:23:25.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.778+0000 7ff4d08f1700 1 -- 172.21.15.3:0/3595315981 learned_addr learned my addr 172.21.15.3:0/3595315981 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:25.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4d10f2700 1 --2- 172.21.15.3:0/3595315981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4cc15a750 0x7ff4cc0d0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4d18f3700 1 --2- 172.21.15.3:0/3595315981 >> v2:172.21.15.186:3300/0 conn(0x7ff4cc0c5720 0x7ff4cc0c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4c3fff700 1 -- 172.21.15.3:0/3595315981 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223186221 0 0) 0x7ff4cc0cfbd0 con 0x7ff4cc0af860 2024-05-08T21:23:25.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4c3fff700 1 -- 172.21.15.3:0/3595315981 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b80035b0 con 0x7ff4cc0af860 2024-05-08T21:23:25.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4c3fff700 1 -- 172.21.15.3:0/3595315981 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4041689082 0 0) 0x7ff4b80035b0 con 0x7ff4cc0af860 2024-05-08T21:23:25.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4c3fff700 1 -- 172.21.15.3:0/3595315981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4cc15a750 msgr2=0x7ff4cc0d0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:25.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4c3fff700 1 --2- 172.21.15.3:0/3595315981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4cc15a750 0x7ff4cc0d0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4c3fff700 1 -- 172.21.15.3:0/3595315981 >> v2:172.21.15.186:3300/0 conn(0x7ff4cc0c5720 msgr2=0x7ff4cc0c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4c3fff700 1 --2- 172.21.15.3:0/3595315981 >> v2:172.21.15.186:3300/0 conn(0x7ff4cc0c5720 0x7ff4cc0c5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4c3fff700 1 -- 172.21.15.3:0/3595315981 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc0cb780 con 0x7ff4cc0af860 2024-05-08T21:23:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4c3fff700 1 -- 172.21.15.3:0/3595315981 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7ff4bc001670 con 0x7ff4cc0af860 2024-05-08T21:23:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4c3fff700 1 -- 172.21.15.3:0/3595315981 >> v1:172.21.15.3:6789/0 conn(0x7ff4cc0af860 legacy=0x7ff4cc0ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4c3fff700 1 --2- 172.21.15.3:0/3595315981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4cc15a750 0x7ff4b8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4c3fff700 1 --2- 172.21.15.3:0/3595315981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b8003f80 0x7ff4b8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4c3fff700 1 --2- 172.21.15.3:0/3595315981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4b8004d90 0x7ff4b8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4c3fff700 1 -- 172.21.15.3:0/3595315981 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc0c60f0 con 0x7ff4b8004d90 2024-05-08T21:23:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4c3fff700 1 -- 172.21.15.3:0/3595315981 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc0cf440 con 0x7ff4b8003f80 2024-05-08T21:23:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4c3fff700 1 -- 172.21.15.3:0/3595315981 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b800ba10 con 0x7ff4cc15a750 2024-05-08T21:23:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4d18f3700 1 --2- 172.21.15.3:0/3595315981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4b8004d90 0x7ff4b8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4d10f2700 1 --2- 172.21.15.3:0/3595315981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4cc15a750 0x7ff4b8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4d08f1700 1 --2- 172.21.15.3:0/3595315981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b8003f80 0x7ff4b8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4d18f3700 1 -- 172.21.15.3:0/3595315981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4cc15a750 msgr2=0x7ff4b8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4d18f3700 1 --2- 172.21.15.3:0/3595315981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4cc15a750 0x7ff4b8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4d18f3700 1 -- 172.21.15.3:0/3595315981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b8003f80 msgr2=0x7ff4b8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.779+0000 7ff4d18f3700 1 --2- 172.21.15.3:0/3595315981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b8003f80 0x7ff4b8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.780+0000 7ff4d18f3700 1 -- 172.21.15.3:0/3595315981 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b80035b0 con 0x7ff4b8004d90 2024-05-08T21:23:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.780+0000 7ff4d18f3700 1 --2- 172.21.15.3:0/3595315981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4b8004d90 0x7ff4b8005210 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009fc0 tx=0x7ff4b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ae3155059b8c32 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.780+0000 7ff4c3fff700 1 -- 172.21.15.3:0/3595315981 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c400ebd0 con 0x7ff4b8004d90 2024-05-08T21:23:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.780+0000 7ff4c3fff700 1 -- 172.21.15.3:0/3595315981 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c400f040 con 0x7ff4b8004d90 2024-05-08T21:23:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.780+0000 7ff4c3fff700 1 -- 172.21.15.3:0/3595315981 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4014480 con 0x7ff4b8004d90 2024-05-08T21:23:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.780+0000 7ff4d3356700 1 -- 172.21.15.3:0/3595315981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4b8004d90 msgr2=0x7ff4b8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.780+0000 7ff4d3356700 1 --2- 172.21.15.3:0/3595315981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4b8004d90 0x7ff4b8005210 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009fc0 tx=0x7ff4b8005880 comp rx=0 tx=0).stop 2024-05-08T21:23:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.780+0000 7ff4d18f3700 1 -- 172.21.15.3:0/3595315981 reap_dead start 2024-05-08T21:23:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.781+0000 7ff4d3356700 1 -- 172.21.15.3:0/3595315981 shutdown_connections 2024-05-08T21:23:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.781+0000 7ff4d3356700 1 -- 172.21.15.3:0/3595315981 >> 172.21.15.3:0/3595315981 conn(0x7ff4cc0d03a0 msgr2=0x7ff4cc0d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.781+0000 7ff4d3356700 1 -- 172.21.15.3:0/3595315981 shutdown_connections 2024-05-08T21:23:25.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.781+0000 7ff4d3356700 1 -- 172.21.15.3:0/3595315981 wait complete. 2024-05-08T21:23:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.782+0000 7ff4d3356700 1 Processor -- start 2024-05-08T21:23:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.782+0000 7ff4d3356700 1 -- start start 2024-05-08T21:23:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d3356700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b8004d90 0x7ff4cc1f3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d3356700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4cc15d240 0x7ff4cc15d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d3356700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4cc1e7a60 0x7ff4cc1e7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d3356700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc1e8430 con 0x7ff4cc15d240 2024-05-08T21:23:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d3356700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc0c7570 con 0x7ff4b8004d90 2024-05-08T21:23:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d3356700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc0c77c0 con 0x7ff4cc1e7a60 2024-05-08T21:23:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d08f1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4cc15d240 0x7ff4cc15d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d08f1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4cc15d240 0x7ff4cc15d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42072/0 (socket says 172.21.15.3:42072) 2024-05-08T21:23:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d08f1700 1 -- 172.21.15.3:0/1059827656 learned_addr learned my addr 172.21.15.3:0/1059827656 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d10f2700 1 --2- 172.21.15.3:0/1059827656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b8004d90 0x7ff4cc1f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d18f3700 1 --2- 172.21.15.3:0/1059827656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4cc1e7a60 0x7ff4cc1e7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d08f1700 1 -- 172.21.15.3:0/1059827656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4cc1e7a60 msgr2=0x7ff4cc1e7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d08f1700 1 --2- 172.21.15.3:0/1059827656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4cc1e7a60 0x7ff4cc1e7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d08f1700 1 -- 172.21.15.3:0/1059827656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b8004d90 msgr2=0x7ff4cc1f3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d08f1700 1 --2- 172.21.15.3:0/1059827656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b8004d90 0x7ff4cc1f3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d08f1700 1 -- 172.21.15.3:0/1059827656 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc0c7aa0 con 0x7ff4cc15d240 2024-05-08T21:23:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.783+0000 7ff4d10f2700 1 --2- 172.21.15.3:0/1059827656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b8004d90 0x7ff4cc1f3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.784+0000 7ff4d08f1700 1 --2- 172.21.15.3:0/1059827656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4cc15d240 0x7ff4cc15d6c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0017a0 tx=0x7ff4bc0076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.784+0000 7ff4c1ffb700 1 -- 172.21.15.3:0/1059827656 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc007ca0 con 0x7ff4cc15d240 2024-05-08T21:23:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.784+0000 7ff4c1ffb700 1 -- 172.21.15.3:0/1059827656 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4bc00f8d0 con 0x7ff4cc15d240 2024-05-08T21:23:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.784+0000 7ff4d3356700 1 -- 172.21.15.3:0/1059827656 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4cc0c7d80 con 0x7ff4cc15d240 2024-05-08T21:23:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.784+0000 7ff4c1ffb700 1 -- 172.21.15.3:0/1059827656 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc00fc80 con 0x7ff4cc15d240 2024-05-08T21:23:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.784+0000 7ff4d3356700 1 -- 172.21.15.3:0/1059827656 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4cc15ad70 con 0x7ff4cc15d240 2024-05-08T21:23:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.785+0000 7ff4d3356700 1 -- 172.21.15.3:0/1059827656 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4cc068000 con 0x7ff4cc15d240 2024-05-08T21:23:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.786+0000 7ff4c1ffb700 1 -- 172.21.15.3:0/1059827656 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff4bc01d460 con 0x7ff4cc15d240 2024-05-08T21:23:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.786+0000 7ff4c1ffb700 1 --2- 172.21.15.3:0/1059827656 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff4b006ac90 0x7ff4b006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.786+0000 7ff4c1ffb700 1 -- 172.21.15.3:0/1059827656 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7ff4bc0895f0 con 0x7ff4cc15d240 2024-05-08T21:23:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.786+0000 7ff4d10f2700 1 --2- 172.21.15.3:0/1059827656 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff4b006ac90 0x7ff4b006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.787+0000 7ff4d10f2700 1 --2- 172.21.15.3:0/1059827656 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff4b006ac90 0x7ff4b006d150 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009b50 tx=0x7ff4c8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:25.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:25.791+0000 7ff4c1ffb700 1 -- 172.21.15.3:0/1059827656 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff4bc0546f0 con 0x7ff4cc15d240 2024-05-08T21:23:26.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.012+0000 7ff4d3356700 1 -- 172.21.15.3:0/1059827656 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4cc0d0040 con 0x7ff4cc15d240 2024-05-08T21:23:26.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.013+0000 7ff4c1ffb700 1 -- 172.21.15.3:0/1059827656 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+18029 (secure 0 0 0) 0x7ff4cc0d0040 con 0x7ff4cc15d240 2024-05-08T21:23:26.015 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:26.015 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":42,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:23:06.476868+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:22:47.853786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-08T21:22:49.153759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"recovery_fs_metadata","create_time":"2024-05-08T21:23:03.921695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6834/3599481370":"2024-05-09T21:23:01.563392+0000","172.21.15.3:6832/3599481370":"2024-05-09T21:23:01.563392+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:23:26.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.015+0000 7ff4d3356700 1 -- 172.21.15.3:0/1059827656 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff4b006ac90 msgr2=0x7ff4b006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:26.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.016+0000 7ff4d3356700 1 --2- 172.21.15.3:0/1059827656 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff4b006ac90 0x7ff4b006d150 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009b50 tx=0x7ff4c8008040 comp rx=0 tx=0).stop 2024-05-08T21:23:26.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.016+0000 7ff4d3356700 1 -- 172.21.15.3:0/1059827656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4cc15d240 msgr2=0x7ff4cc15d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:26.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.016+0000 7ff4d3356700 1 --2- 172.21.15.3:0/1059827656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4cc15d240 0x7ff4cc15d6c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0017a0 tx=0x7ff4bc0076a0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.016+0000 7ff4d3356700 1 -- 172.21.15.3:0/1059827656 shutdown_connections 2024-05-08T21:23:26.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.016+0000 7ff4d3356700 1 --2- 172.21.15.3:0/1059827656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4cc1e7a60 0x7ff4cc1e7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.016+0000 7ff4d3356700 1 --2- 172.21.15.3:0/1059827656 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff4b006ac90 0x7ff4b006d150 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.016+0000 7ff4d3356700 1 --2- 172.21.15.3:0/1059827656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4b8004d90 0x7ff4cc1f3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.016+0000 7ff4d3356700 1 --2- 172.21.15.3:0/1059827656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4cc15d240 0x7ff4cc15d6c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.016+0000 7ff4d3356700 1 -- 172.21.15.3:0/1059827656 >> 172.21.15.3:0/1059827656 conn(0x7ff4cc0d03a0 msgr2=0x7ff4cc159ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:26.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.016+0000 7ff4d3356700 1 -- 172.21.15.3:0/1059827656 shutdown_connections 2024-05-08T21:23:26.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.016+0000 7ff4d3356700 1 -- 172.21.15.3:0/1059827656 wait complete. 2024-05-08T21:23:26.031 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-08T21:23:26.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:23:26.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.167+0000 7f075184e700 1 Processor -- start 2024-05-08T21:23:26.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.167+0000 7f075184e700 1 -- start start 2024-05-08T21:23:26.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.167+0000 7f075184e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f074c06b810 0x7f074c0cf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:26.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.167+0000 7f075184e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f074c06bc70 0x7f074c0cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:26.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.167+0000 7f075184e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f074c0d0880 con 0x7f074c15d9c0 2024-05-08T21:23:26.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.167+0000 7f075184e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f074c0d0b60 con 0x7f074c06b810 2024-05-08T21:23:26.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.167+0000 7f075184e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0ce5b0 con 0x7f074c06bc70 2024-05-08T21:23:26.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.168+0000 7f07427fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f074c06b810 0x7f074c0cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:26.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.168+0000 7f074affd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f074c15d9c0 0x7f074c0cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50488/0 (socket says 172.21.15.3:50488) 2024-05-08T21:23:26.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.168+0000 7f074affd700 1 -- 172.21.15.3:0/1570943693 learned_addr learned my addr 172.21.15.3:0/1570943693 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:26.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.168+0000 7f074b7fe700 1 --2- 172.21.15.3:0/1570943693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f074c06bc70 0x7f074c0cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:26.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.168+0000 7f07427fc700 1 -- 172.21.15.3:0/1570943693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f074c06bc70 msgr2=0x7f074c0cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:26.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.168+0000 7f07427fc700 1 --2- 172.21.15.3:0/1570943693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f074c06bc70 0x7f074c0cf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.168+0000 7f07427fc700 1 -- 172.21.15.3:0/1570943693 >> v1:172.21.15.3:6789/0 conn(0x7f074c15d9c0 legacy=0x7f074c0cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:26.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.168+0000 7f07427fc700 1 -- 172.21.15.3:0/1570943693 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c0ce890 con 0x7f074c06b810 2024-05-08T21:23:26.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.168+0000 7f074a7fc700 1 -- 172.21.15.3:0/1570943693 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053943006 0 0) 0x7f074c0d0880 con 0x7f074c15d9c0 2024-05-08T21:23:26.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.168+0000 7f07427fc700 1 --2- 172.21.15.3:0/1570943693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f074c06b810 0x7f074c0cf090 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0734009820 tx=0x7f0734009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531657fff37ecbd3 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:26.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.168+0000 7f074a7fc700 1 -- 172.21.15.3:0/1570943693 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0734005e30 con 0x7f074c06b810 2024-05-08T21:23:26.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.168+0000 7f074a7fc700 1 -- 172.21.15.3:0/1570943693 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0734006030 con 0x7f074c06b810 2024-05-08T21:23:26.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.168+0000 7f074a7fc700 1 -- 172.21.15.3:0/1570943693 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07340063e0 con 0x7f074c06b810 2024-05-08T21:23:26.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.169+0000 7f075184e700 1 -- 172.21.15.3:0/1570943693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f074c06b810 msgr2=0x7f074c0cf090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:26.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.169+0000 7f075184e700 1 --2- 172.21.15.3:0/1570943693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f074c06b810 0x7f074c0cf090 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0734009820 tx=0x7f0734009b30 comp rx=0 tx=0).stop 2024-05-08T21:23:26.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.169+0000 7f075184e700 1 -- 172.21.15.3:0/1570943693 shutdown_connections 2024-05-08T21:23:26.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.169+0000 7f075184e700 1 --2- 172.21.15.3:0/1570943693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f074c06bc70 0x7f074c0cf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.169+0000 7f075184e700 1 --2- 172.21.15.3:0/1570943693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f074c06b810 0x7f074c0cf090 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.169+0000 7f075184e700 1 -- 172.21.15.3:0/1570943693 >> 172.21.15.3:0/1570943693 conn(0x7f074c08afe0 msgr2=0x7f074c0d3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:26.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.169+0000 7f075184e700 1 -- 172.21.15.3:0/1570943693 shutdown_connections 2024-05-08T21:23:26.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.169+0000 7f075184e700 1 -- 172.21.15.3:0/1570943693 wait complete. 2024-05-08T21:23:26.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.170+0000 7f075184e700 1 Processor -- start 2024-05-08T21:23:26.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.171+0000 7f075184e700 1 -- start start 2024-05-08T21:23:26.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.171+0000 7f075184e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f074c06b810 0x7f074c15be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:26.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.171+0000 7f075184e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f074c06bc70 0x7f074c0b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:26.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.171+0000 7f075184e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f074c15d9c0 0x7f074c0b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:26.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.171+0000 7f075184e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0b6b60 con 0x7f074c15d9c0 2024-05-08T21:23:26.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.171+0000 7f075184e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0b6e40 con 0x7f074c06b810 2024-05-08T21:23:26.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.171+0000 7f075184e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0b7120 con 0x7f074c06bc70 2024-05-08T21:23:26.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.172+0000 7f074b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f074c15d9c0 0x7f074c0b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:26.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.172+0000 7f074b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f074c15d9c0 0x7f074c0b4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42102/0 (socket says 172.21.15.3:42102) 2024-05-08T21:23:26.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.172+0000 7f074b7fe700 1 -- 172.21.15.3:0/3267382969 learned_addr learned my addr 172.21.15.3:0/3267382969 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:26.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.172+0000 7f07427fc700 1 --2- 172.21.15.3:0/3267382969 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f074c06bc70 0x7f074c0b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:26.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.172+0000 7f074affd700 1 --2- 172.21.15.3:0/3267382969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f074c06b810 0x7f074c15be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:26.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.172+0000 7f074b7fe700 1 -- 172.21.15.3:0/3267382969 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f074c06bc70 msgr2=0x7f074c0b61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:26.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.172+0000 7f074b7fe700 1 --2- 172.21.15.3:0/3267382969 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f074c06bc70 0x7f074c0b61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.172+0000 7f074b7fe700 1 -- 172.21.15.3:0/3267382969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f074c06b810 msgr2=0x7f074c15be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:26.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.172+0000 7f074b7fe700 1 --2- 172.21.15.3:0/3267382969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f074c06b810 0x7f074c15be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.172+0000 7f074b7fe700 1 -- 172.21.15.3:0/3267382969 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0734009430 con 0x7f074c15d9c0 2024-05-08T21:23:26.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.172+0000 7f074b7fe700 1 --2- 172.21.15.3:0/3267382969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f074c15d9c0 0x7f074c0b4f40 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0744007e40 tx=0x7f074400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:26.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.172+0000 7f0748ff9700 1 -- 172.21.15.3:0/3267382969 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074400eea0 con 0x7f074c15d9c0 2024-05-08T21:23:26.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.172+0000 7f0748ff9700 1 -- 172.21.15.3:0/3267382969 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0744014420 con 0x7f074c15d9c0 2024-05-08T21:23:26.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.172+0000 7f0748ff9700 1 -- 172.21.15.3:0/3267382969 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07440147b0 con 0x7f074c15d9c0 2024-05-08T21:23:26.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.172+0000 7f075184e700 1 -- 172.21.15.3:0/3267382969 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f074c0b7400 con 0x7f074c15d9c0 2024-05-08T21:23:26.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.172+0000 7f075184e700 1 -- 172.21.15.3:0/3267382969 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f074c0b7c00 con 0x7f074c15d9c0 2024-05-08T21:23:26.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.174+0000 7f075184e700 1 -- 172.21.15.3:0/3267382969 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f074c0b7920 con 0x7f074c15d9c0 2024-05-08T21:23:26.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.174+0000 7f0748ff9700 1 -- 172.21.15.3:0/3267382969 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0744014910 con 0x7f074c15d9c0 2024-05-08T21:23:26.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.174+0000 7f0748ff9700 1 --2- 172.21.15.3:0/3267382969 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f072c06ac90 0x7f072c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:26.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.175+0000 7f0748ff9700 1 -- 172.21.15.3:0/3267382969 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f0744089e10 con 0x7f074c15d9c0 2024-05-08T21:23:26.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.175+0000 7f074affd700 1 --2- 172.21.15.3:0/3267382969 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f072c06ac90 0x7f072c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:26.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.175+0000 7f074affd700 1 --2- 172.21.15.3:0/3267382969 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f072c06ac90 0x7f072c06d150 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f073c000f60 tx=0x7f073c007c90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:26.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.179+0000 7f0748ff9700 1 -- 172.21.15.3:0/3267382969 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f074c0b7920 con 0x7f074c15d9c0 2024-05-08T21:23:26.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.449+0000 7f075184e700 1 -- 172.21.15.3:0/3267382969 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f074c0b7920 con 0x7f074c15d9c0 2024-05-08T21:23:26.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.450+0000 7f0748ff9700 1 -- 172.21.15.3:0/3267382969 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5936 (secure 0 0 0) 0x7f074c0b7920 con 0x7f074c15d9c0 2024-05-08T21:23:26.451 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:26.452 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6833/2104267343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2104267343},{"type":"v1","addr":"172.21.15.3:6833","nonce":2104267343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/208321708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":208321708},{"type":"v1","addr":"172.21.15.3:6835","nonce":208321708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/1591386131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1591386131},{"type":"v1","addr":"172.21.15.186:6836","nonce":1591386131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:22:50.184245+0000","modified":"2024-05-08T21:23:01.563810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:23:05.475355+0000","modified":"2024-05-08T21:23:15.662101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6837/2000072997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":2000072997},{"type":"v1","addr":"172.21.15.186:6837","nonce":2000072997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":6,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-08T21:23:26.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.452+0000 7f075184e700 1 -- 172.21.15.3:0/3267382969 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f072c06ac90 msgr2=0x7f072c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:26.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.452+0000 7f075184e700 1 --2- 172.21.15.3:0/3267382969 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f072c06ac90 0x7f072c06d150 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f073c000f60 tx=0x7f073c007c90 comp rx=0 tx=0).stop 2024-05-08T21:23:26.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.452+0000 7f075184e700 1 -- 172.21.15.3:0/3267382969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f074c15d9c0 msgr2=0x7f074c0b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:26.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.452+0000 7f075184e700 1 --2- 172.21.15.3:0/3267382969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f074c15d9c0 0x7f074c0b4f40 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0744007e40 tx=0x7f074400e3c0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.452+0000 7f075184e700 1 -- 172.21.15.3:0/3267382969 shutdown_connections 2024-05-08T21:23:26.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.452+0000 7f075184e700 1 --2- 172.21.15.3:0/3267382969 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f072c06ac90 0x7f072c06d150 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.452+0000 7f075184e700 1 --2- 172.21.15.3:0/3267382969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f074c15d9c0 0x7f074c0b4f40 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.452+0000 7f075184e700 1 --2- 172.21.15.3:0/3267382969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f074c06b810 0x7f074c15be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.452+0000 7f075184e700 1 --2- 172.21.15.3:0/3267382969 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f074c06bc70 0x7f074c0b61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.452+0000 7f075184e700 1 -- 172.21.15.3:0/3267382969 >> 172.21.15.3:0/3267382969 conn(0x7f074c08afe0 msgr2=0x7f074c15dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:26.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.453+0000 7f075184e700 1 -- 172.21.15.3:0/3267382969 shutdown_connections 2024-05-08T21:23:26.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.453+0000 7f075184e700 1 -- 172.21.15.3:0/3267382969 wait complete. 2024-05-08T21:23:26.455 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 26 2024-05-08T21:23:26.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:23:26.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.602+0000 7fcb48ef2700 1 Processor -- start 2024-05-08T21:23:26.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.602+0000 7fcb48ef2700 1 -- start start 2024-05-08T21:23:26.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.603+0000 7fcb48ef2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcb440b29e0 0x7fcb440cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:26.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.603+0000 7fcb48ef2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb440d0f50 0x7fcb4415fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:26.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.603+0000 7fcb48ef2700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4415ba40 con 0x7fcb440c6740 2024-05-08T21:23:26.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.603+0000 7fcb48ef2700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb440ca960 con 0x7fcb440b29e0 2024-05-08T21:23:26.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.603+0000 7fcb48ef2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb441600a0 con 0x7fcb440d0f50 2024-05-08T21:23:26.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.603+0000 7fcb4259c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcb440b29e0 0x7fcb440cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:26.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.603+0000 7fcb42d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fcb440c6740 0x7fcb440b1180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50504/0 (socket says 172.21.15.3:50504) 2024-05-08T21:23:26.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.603+0000 7fcb42d9d700 1 -- 172.21.15.3:0/1896837099 learned_addr learned my addr 172.21.15.3:0/1896837099 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:26.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.604+0000 7fcb41d9b700 1 --2- 172.21.15.3:0/1896837099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb440d0f50 0x7fcb4415fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:26.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.604+0000 7fcb4259c700 1 -- 172.21.15.3:0/1896837099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb440d0f50 msgr2=0x7fcb4415fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:26.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.604+0000 7fcb4259c700 1 --2- 172.21.15.3:0/1896837099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb440d0f50 0x7fcb4415fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.604+0000 7fcb4259c700 1 -- 172.21.15.3:0/1896837099 >> v1:172.21.15.3:6789/0 conn(0x7fcb440c6740 legacy=0x7fcb440b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:26.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.604+0000 7fcb4159a700 1 -- 172.21.15.3:0/1896837099 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954584508 0 0) 0x7fcb4415ba40 con 0x7fcb440c6740 2024-05-08T21:23:26.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.604+0000 7fcb4259c700 1 -- 172.21.15.3:0/1896837099 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb440cabb0 con 0x7fcb440b29e0 2024-05-08T21:23:26.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.604+0000 7fcb4259c700 1 --2- 172.21.15.3:0/1896837099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb440b29e0 0x7fcb440cbc30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcb34009840 tx=0x7fcb34009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc6ae5d1aa83b32a server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:26.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.604+0000 7fcb4159a700 1 -- 172.21.15.3:0/1896837099 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34011830 con 0x7fcb440b29e0 2024-05-08T21:23:26.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.604+0000 7fcb4159a700 1 -- 172.21.15.3:0/1896837099 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb34011990 con 0x7fcb440b29e0 2024-05-08T21:23:26.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.604+0000 7fcb4159a700 1 -- 172.21.15.3:0/1896837099 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34011d20 con 0x7fcb440b29e0 2024-05-08T21:23:26.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.605+0000 7fcb48ef2700 1 -- 172.21.15.3:0/1896837099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb440b29e0 msgr2=0x7fcb440cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:26.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.605+0000 7fcb48ef2700 1 --2- 172.21.15.3:0/1896837099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb440b29e0 0x7fcb440cbc30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcb34009840 tx=0x7fcb34009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:26.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.605+0000 7fcb48ef2700 1 -- 172.21.15.3:0/1896837099 shutdown_connections 2024-05-08T21:23:26.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.605+0000 7fcb48ef2700 1 --2- 172.21.15.3:0/1896837099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb440b29e0 0x7fcb440cbc30 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.605+0000 7fcb48ef2700 1 --2- 172.21.15.3:0/1896837099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb440d0f50 0x7fcb4415fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.605+0000 7fcb48ef2700 1 -- 172.21.15.3:0/1896837099 >> 172.21.15.3:0/1896837099 conn(0x7fcb440d03a0 msgr2=0x7fcb440d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:26.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.605+0000 7fcb48ef2700 1 -- 172.21.15.3:0/1896837099 shutdown_connections 2024-05-08T21:23:26.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.605+0000 7fcb48ef2700 1 -- 172.21.15.3:0/1896837099 wait complete. 2024-05-08T21:23:26.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.607+0000 7fcb48ef2700 1 Processor -- start 2024-05-08T21:23:26.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.607+0000 7fcb48ef2700 1 -- start start 2024-05-08T21:23:26.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.607+0000 7fcb48ef2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb440b29e0 0x7fcb4415f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:26.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.607+0000 7fcb48ef2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb440c6740 0x7fcb441f1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:26.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.607+0000 7fcb48ef2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb440d0f50 0x7fcb440c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:26.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.607+0000 7fcb48ef2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb440b4ab0 con 0x7fcb440c6740 2024-05-08T21:23:26.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.607+0000 7fcb48ef2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb441f1af0 con 0x7fcb440d0f50 2024-05-08T21:23:26.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.607+0000 7fcb48ef2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb441f1dd0 con 0x7fcb440b29e0 2024-05-08T21:23:26.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.607+0000 7fcb41d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb440c6740 0x7fcb441f1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.608+0000 7fcb41d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb440c6740 0x7fcb441f1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42120/0 (socket says 172.21.15.3:42120) 2024-05-08T21:23:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.608+0000 7fcb41d9b700 1 -- 172.21.15.3:0/3833975399 learned_addr learned my addr 172.21.15.3:0/3833975399 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.608+0000 7fcb4259c700 1 --2- 172.21.15.3:0/3833975399 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb440b29e0 0x7fcb4415f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.608+0000 7fcb41d9b700 1 -- 172.21.15.3:0/3833975399 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb440b29e0 msgr2=0x7fcb4415f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.608+0000 7fcb41d9b700 1 --2- 172.21.15.3:0/3833975399 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb440b29e0 0x7fcb4415f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.608+0000 7fcb41d9b700 1 -- 172.21.15.3:0/3833975399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb440d0f50 msgr2=0x7fcb440c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.608+0000 7fcb41d9b700 1 --2- 172.21.15.3:0/3833975399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb440d0f50 0x7fcb440c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.608+0000 7fcb41d9b700 1 -- 172.21.15.3:0/3833975399 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb340094a0 con 0x7fcb440c6740 2024-05-08T21:23:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.608+0000 7fcb41d9b700 1 --2- 172.21.15.3:0/3833975399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb440c6740 0x7fcb441f1050 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800e7f0 tx=0x7fcb3800ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.608+0000 7fcb32ffd700 1 -- 172.21.15.3:0/3833975399 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38009550 con 0x7fcb440c6740 2024-05-08T21:23:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.608+0000 7fcb32ffd700 1 -- 172.21.15.3:0/3833975399 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb38013680 con 0x7fcb440c6740 2024-05-08T21:23:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.608+0000 7fcb48ef2700 1 -- 172.21.15.3:0/3833975399 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb441f2370 con 0x7fcb440c6740 2024-05-08T21:23:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.608+0000 7fcb32ffd700 1 -- 172.21.15.3:0/3833975399 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38013a10 con 0x7fcb440c6740 2024-05-08T21:23:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.608+0000 7fcb48ef2700 1 -- 172.21.15.3:0/3833975399 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb440b3720 con 0x7fcb440c6740 2024-05-08T21:23:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.609+0000 7fcb48ef2700 1 -- 172.21.15.3:0/3833975399 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb441f2890 con 0x7fcb440c6740 2024-05-08T21:23:26.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.610+0000 7fcb32ffd700 1 -- 172.21.15.3:0/3833975399 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcb38013b70 con 0x7fcb440c6740 2024-05-08T21:23:26.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.610+0000 7fcb32ffd700 1 --2- 172.21.15.3:0/3833975399 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb2406ac90 0x7fcb2406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:26.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.610+0000 7fcb32ffd700 1 -- 172.21.15.3:0/3833975399 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7fcb38089430 con 0x7fcb440c6740 2024-05-08T21:23:26.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.611+0000 7fcb4259c700 1 --2- 172.21.15.3:0/3833975399 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb2406ac90 0x7fcb2406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:26.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.611+0000 7fcb4259c700 1 --2- 172.21.15.3:0/3833975399 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb2406ac90 0x7fcb2406d150 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcb34009fd0 tx=0x7fcb34008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:26.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.615+0000 7fcb32ffd700 1 -- 172.21.15.3:0/3833975399 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fcb441f2890 con 0x7fcb440c6740 2024-05-08T21:23:26.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.880+0000 7fcb48ef2700 1 -- 172.21.15.3:0/3833975399 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb440b0dc0 con 0x7fcb440c6740 2024-05-08T21:23:26.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.881+0000 7fcb32ffd700 1 -- 172.21.15.3:0/3833975399 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5936 (secure 0 0 0) 0x7fcb38054480 con 0x7fcb440c6740 2024-05-08T21:23:26.882 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:26.882 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6833/2104267343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2104267343},{"type":"v1","addr":"172.21.15.3:6833","nonce":2104267343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/208321708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":208321708},{"type":"v1","addr":"172.21.15.3:6835","nonce":208321708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/1591386131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1591386131},{"type":"v1","addr":"172.21.15.186:6836","nonce":1591386131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:22:50.184245+0000","modified":"2024-05-08T21:23:01.563810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:23:05.475355+0000","modified":"2024-05-08T21:23:15.662101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6837/2000072997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":2000072997},{"type":"v1","addr":"172.21.15.186:6837","nonce":2000072997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":6,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-08T21:23:26.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.883+0000 7fcb48ef2700 1 -- 172.21.15.3:0/3833975399 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb2406ac90 msgr2=0x7fcb2406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:26.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.883+0000 7fcb48ef2700 1 --2- 172.21.15.3:0/3833975399 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb2406ac90 0x7fcb2406d150 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcb34009fd0 tx=0x7fcb34008030 comp rx=0 tx=0).stop 2024-05-08T21:23:26.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.883+0000 7fcb48ef2700 1 -- 172.21.15.3:0/3833975399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb440c6740 msgr2=0x7fcb441f1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:26.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.883+0000 7fcb48ef2700 1 --2- 172.21.15.3:0/3833975399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb440c6740 0x7fcb441f1050 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800e7f0 tx=0x7fcb3800ebb0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.883+0000 7fcb48ef2700 1 -- 172.21.15.3:0/3833975399 shutdown_connections 2024-05-08T21:23:26.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.883+0000 7fcb48ef2700 1 --2- 172.21.15.3:0/3833975399 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb440b29e0 0x7fcb4415f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.883+0000 7fcb48ef2700 1 --2- 172.21.15.3:0/3833975399 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fcb2406ac90 0x7fcb2406d150 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.883+0000 7fcb48ef2700 1 --2- 172.21.15.3:0/3833975399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb440d0f50 0x7fcb440c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.883+0000 7fcb48ef2700 1 --2- 172.21.15.3:0/3833975399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb440c6740 0x7fcb441f1050 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:26.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.883+0000 7fcb48ef2700 1 -- 172.21.15.3:0/3833975399 >> 172.21.15.3:0/3833975399 conn(0x7fcb440d03a0 msgr2=0x7fcb44159900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:26.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.883+0000 7fcb48ef2700 1 -- 172.21.15.3:0/3833975399 shutdown_connections 2024-05-08T21:23:26.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:26.883+0000 7fcb48ef2700 1 -- 172.21.15.3:0/3833975399 wait complete. 2024-05-08T21:23:26.886 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 26 2024-05-08T21:23:26.898 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:23:27.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.034+0000 7ff32b1c6700 1 Processor -- start 2024-05-08T21:23:27.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.034+0000 7ff32b1c6700 1 -- start start 2024-05-08T21:23:27.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff32b1c6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32415d780 0x7ff3240b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:27.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff32b1c6700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3240b4430 con 0x7ff3240d13e0 2024-05-08T21:23:27.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff32b1c6700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3240b3820 con 0x7ff3240d22f0 2024-05-08T21:23:27.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff32b1c6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240d45c0 con 0x7ff32415d780 2024-05-08T21:23:27.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff31bfff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff3240d13e0 0x7ff3240d1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50520/0 (socket says 172.21.15.3:50520) 2024-05-08T21:23:27.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff31bfff700 1 -- 172.21.15.3:0/3155293572 learned_addr learned my addr 172.21.15.3:0/3155293572 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:27.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff328f62700 1 --2- 172.21.15.3:0/3155293572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32415d780 0x7ff3240b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff31b7fe700 1 -- 172.21.15.3:0/3155293572 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175485496 0 0) 0x7ff3240b4430 con 0x7ff3240d13e0 2024-05-08T21:23:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff31b7fe700 1 -- 172.21.15.3:0/3155293572 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff30c0035b0 con 0x7ff3240d13e0 2024-05-08T21:23:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff31b7fe700 1 -- 172.21.15.3:0/3155293572 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126652110 0 0) 0x7ff3240b3820 con 0x7ff3240d22f0 2024-05-08T21:23:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff31b7fe700 1 -- 172.21.15.3:0/3155293572 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3240b4430 con 0x7ff3240d22f0 2024-05-08T21:23:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff31b7fe700 1 -- 172.21.15.3:0/3155293572 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1750159942 0 0) 0x7ff30c0035b0 con 0x7ff3240d13e0 2024-05-08T21:23:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff31b7fe700 1 -- 172.21.15.3:0/3155293572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32415d780 msgr2=0x7ff3240b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff31b7fe700 1 --2- 172.21.15.3:0/3155293572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32415d780 0x7ff3240b48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff31b7fe700 1 -- 172.21.15.3:0/3155293572 >> v1:172.21.15.186:6789/0 conn(0x7ff3240d22f0 legacy=0x7ff3240ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff31b7fe700 1 -- 172.21.15.3:0/3155293572 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3240b3b00 con 0x7ff3240d13e0 2024-05-08T21:23:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff31b7fe700 1 -- 172.21.15.3:0/3155293572 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7ff314001670 con 0x7ff3240d13e0 2024-05-08T21:23:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff328f62700 1 --2- 172.21.15.3:0/3155293572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32415d780 0x7ff3240b48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.035+0000 7ff31b7fe700 1 -- 172.21.15.3:0/3155293572 >> v1:172.21.15.3:6789/0 conn(0x7ff3240d13e0 legacy=0x7ff3240d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff31b7fe700 1 --2- 172.21.15.3:0/3155293572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32415d780 0x7ff30c003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff31b7fe700 1 --2- 172.21.15.3:0/3155293572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff30c004150 0x7ff30c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff31b7fe700 1 --2- 172.21.15.3:0/3155293572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff30c004d90 0x7ff30c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff31b7fe700 1 -- 172.21.15.3:0/3155293572 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240d45c0 con 0x7ff30c004150 2024-05-08T21:23:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff31b7fe700 1 -- 172.21.15.3:0/3155293572 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c00bb20 con 0x7ff30c004d90 2024-05-08T21:23:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff31b7fe700 1 -- 172.21.15.3:0/3155293572 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c00bce0 con 0x7ff32415d780 2024-05-08T21:23:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff31bfff700 1 --2- 172.21.15.3:0/3155293572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff30c004150 0x7ff30c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff328f62700 1 --2- 172.21.15.3:0/3155293572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32415d780 0x7ff30c003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff329763700 1 --2- 172.21.15.3:0/3155293572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff30c004d90 0x7ff30c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff31bfff700 1 -- 172.21.15.3:0/3155293572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32415d780 msgr2=0x7ff30c003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff31bfff700 1 --2- 172.21.15.3:0/3155293572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32415d780 0x7ff30c003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff31bfff700 1 -- 172.21.15.3:0/3155293572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff30c004d90 msgr2=0x7ff30c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff31bfff700 1 --2- 172.21.15.3:0/3155293572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff30c004d90 0x7ff30c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff31bfff700 1 -- 172.21.15.3:0/3155293572 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff320009430 con 0x7ff30c004150 2024-05-08T21:23:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff31bfff700 1 --2- 172.21.15.3:0/3155293572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff30c004150 0x7ff30c004850 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff31400c0e0 tx=0x7ff30c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51a938a3b2e7641f server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff31b7fe700 1 -- 172.21.15.3:0/3155293572 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31400cc20 con 0x7ff30c004150 2024-05-08T21:23:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff31b7fe700 1 -- 172.21.15.3:0/3155293572 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31400cda0 con 0x7ff30c004150 2024-05-08T21:23:27.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.036+0000 7ff31b7fe700 1 -- 172.21.15.3:0/3155293572 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314005ba0 con 0x7ff30c004150 2024-05-08T21:23:27.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.037+0000 7ff32b1c6700 1 -- 172.21.15.3:0/3155293572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff30c004150 msgr2=0x7ff30c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:27.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.037+0000 7ff32b1c6700 1 --2- 172.21.15.3:0/3155293572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff30c004150 0x7ff30c004850 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff31400c0e0 tx=0x7ff30c005880 comp rx=0 tx=0).stop 2024-05-08T21:23:27.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.037+0000 7ff329763700 1 -- 172.21.15.3:0/3155293572 reap_dead start 2024-05-08T21:23:27.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.037+0000 7ff32b1c6700 1 -- 172.21.15.3:0/3155293572 shutdown_connections 2024-05-08T21:23:27.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.037+0000 7ff32b1c6700 1 -- 172.21.15.3:0/3155293572 >> 172.21.15.3:0/3155293572 conn(0x7ff32408afe0 msgr2=0x7ff3240d3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:27.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.037+0000 7ff32b1c6700 1 -- 172.21.15.3:0/3155293572 shutdown_connections 2024-05-08T21:23:27.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.037+0000 7ff32b1c6700 1 -- 172.21.15.3:0/3155293572 wait complete. 2024-05-08T21:23:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.039+0000 7ff32b1c6700 1 Processor -- start 2024-05-08T21:23:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.039+0000 7ff32b1c6700 1 -- start start 2024-05-08T21:23:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.040+0000 7ff32b1c6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff30c004150 0x7ff3240b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.040+0000 7ff32b1c6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3241f6740 0x7ff32415db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.040+0000 7ff32b1c6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff32415e0a0 0x7ff32415fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.040+0000 7ff32b1c6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240b7c50 con 0x7ff32415e0a0 2024-05-08T21:23:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.040+0000 7ff32b1c6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32415e520 con 0x7ff3241f6740 2024-05-08T21:23:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.040+0000 7ff32b1c6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff324160aa0 con 0x7ff30c004150 2024-05-08T21:23:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.040+0000 7ff329763700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff32415e0a0 0x7ff32415fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.040+0000 7ff329763700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff32415e0a0 0x7ff32415fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42128/0 (socket says 172.21.15.3:42128) 2024-05-08T21:23:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.040+0000 7ff329763700 1 -- 172.21.15.3:0/1196890006 learned_addr learned my addr 172.21.15.3:0/1196890006 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.040+0000 7ff328f62700 1 --2- 172.21.15.3:0/1196890006 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff30c004150 0x7ff3240b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:27.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.040+0000 7ff31bfff700 1 --2- 172.21.15.3:0/1196890006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3241f6740 0x7ff32415db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:27.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.041+0000 7ff329763700 1 -- 172.21.15.3:0/1196890006 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff30c004150 msgr2=0x7ff3240b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:27.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.041+0000 7ff329763700 1 --2- 172.21.15.3:0/1196890006 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff30c004150 0x7ff3240b0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.041+0000 7ff329763700 1 -- 172.21.15.3:0/1196890006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3241f6740 msgr2=0x7ff32415db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:27.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.041+0000 7ff329763700 1 --2- 172.21.15.3:0/1196890006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3241f6740 0x7ff32415db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.041+0000 7ff329763700 1 -- 172.21.15.3:0/1196890006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3240b72f0 con 0x7ff32415e0a0 2024-05-08T21:23:27.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.041+0000 7ff328f62700 1 --2- 172.21.15.3:0/1196890006 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff30c004150 0x7ff3240b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:27.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.041+0000 7ff329763700 1 --2- 172.21.15.3:0/1196890006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff32415e0a0 0x7ff32415fd70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff32408b550 tx=0x7ff31c00d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:27.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.041+0000 7ff3197fa700 1 -- 172.21.15.3:0/1196890006 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c00db60 con 0x7ff32415e0a0 2024-05-08T21:23:27.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.041+0000 7ff3197fa700 1 -- 172.21.15.3:0/1196890006 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31c00dcc0 con 0x7ff32415e0a0 2024-05-08T21:23:27.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.041+0000 7ff32b1c6700 1 -- 172.21.15.3:0/1196890006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3240b7590 con 0x7ff32415e0a0 2024-05-08T21:23:27.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.041+0000 7ff3197fa700 1 -- 172.21.15.3:0/1196890006 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c003a00 con 0x7ff32415e0a0 2024-05-08T21:23:27.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.041+0000 7ff32b1c6700 1 -- 172.21.15.3:0/1196890006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3241f6bc0 con 0x7ff32415e0a0 2024-05-08T21:23:27.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.042+0000 7ff32b1c6700 1 -- 172.21.15.3:0/1196890006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3240d1a50 con 0x7ff32415e0a0 2024-05-08T21:23:27.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.042+0000 7ff3197fa700 1 -- 172.21.15.3:0/1196890006 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff31c003b80 con 0x7ff32415e0a0 2024-05-08T21:23:27.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.043+0000 7ff3197fa700 1 --2- 172.21.15.3:0/1196890006 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff31006ac90 0x7ff31006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:27.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.043+0000 7ff3197fa700 1 -- 172.21.15.3:0/1196890006 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7ff31c089740 con 0x7ff32415e0a0 2024-05-08T21:23:27.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.043+0000 7ff328f62700 1 --2- 172.21.15.3:0/1196890006 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff31006ac90 0x7ff31006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:27.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.044+0000 7ff328f62700 1 --2- 172.21.15.3:0/1196890006 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff31006ac90 0x7ff31006d150 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff320009cf0 tx=0x7ff320008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:27.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.048+0000 7ff3197fa700 1 -- 172.21.15.3:0/1196890006 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff31c054790 con 0x7ff32415e0a0 2024-05-08T21:23:27.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.265+0000 7ff32b1c6700 1 -- 172.21.15.3:0/1196890006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3240b7d90 con 0x7ff32415e0a0 2024-05-08T21:23:27.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.266+0000 7ff3197fa700 1 -- 172.21.15.3:0/1196890006 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+18029 (secure 0 0 0) 0x7ff3240b7d90 con 0x7ff32415e0a0 2024-05-08T21:23:27.268 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:27.268 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":42,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:23:06.476868+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-08T21:22:47.853786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-08T21:22:49.153759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"recovery_fs_metadata","create_time":"2024-05-08T21:23:03.921695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6834/3599481370":"2024-05-09T21:23:01.563392+0000","172.21.15.3:6832/3599481370":"2024-05-09T21:23:01.563392+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:23:27.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.268+0000 7ff32b1c6700 1 -- 172.21.15.3:0/1196890006 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff31006ac90 msgr2=0x7ff31006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:27.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.268+0000 7ff32b1c6700 1 --2- 172.21.15.3:0/1196890006 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff31006ac90 0x7ff31006d150 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff320009cf0 tx=0x7ff320008040 comp rx=0 tx=0).stop 2024-05-08T21:23:27.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.268+0000 7ff32b1c6700 1 -- 172.21.15.3:0/1196890006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff32415e0a0 msgr2=0x7ff32415fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:27.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.268+0000 7ff32b1c6700 1 --2- 172.21.15.3:0/1196890006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff32415e0a0 0x7ff32415fd70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff32408b550 tx=0x7ff31c00d660 comp rx=0 tx=0).stop 2024-05-08T21:23:27.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.268+0000 7ff32b1c6700 1 -- 172.21.15.3:0/1196890006 shutdown_connections 2024-05-08T21:23:27.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.268+0000 7ff32b1c6700 1 --2- 172.21.15.3:0/1196890006 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff30c004150 0x7ff3240b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.268+0000 7ff32b1c6700 1 --2- 172.21.15.3:0/1196890006 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff31006ac90 0x7ff31006d150 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.268+0000 7ff32b1c6700 1 --2- 172.21.15.3:0/1196890006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff32415e0a0 0x7ff32415fd70 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.268+0000 7ff32b1c6700 1 --2- 172.21.15.3:0/1196890006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3241f6740 0x7ff32415db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.268+0000 7ff32b1c6700 1 -- 172.21.15.3:0/1196890006 >> 172.21.15.3:0/1196890006 conn(0x7ff32408afe0 msgr2=0x7ff3240cfa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:27.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.268+0000 7ff32b1c6700 1 -- 172.21.15.3:0/1196890006 shutdown_connections 2024-05-08T21:23:27.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.268+0000 7ff32b1c6700 1 -- 172.21.15.3:0/1196890006 wait complete. 2024-05-08T21:23:27.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-08T21:23:27.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.419+0000 7f9bf7a22700 1 Processor -- start 2024-05-08T21:23:27.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.419+0000 7f9bf7a22700 1 -- start start 2024-05-08T21:23:27.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.420+0000 7f9bf7a22700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9bf015b900 0x7f9bf00cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:27.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.420+0000 7f9bf7a22700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf009a610 0x7f9bf00b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:27.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.420+0000 7f9bf7a22700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf00b1410 con 0x7f9bf0099a50 2024-05-08T21:23:27.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.420+0000 7f9bf7a22700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf00b0e30 con 0x7f9bf015b900 2024-05-08T21:23:27.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.420+0000 7f9bf7a22700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf00cffd0 con 0x7f9bf009a610 2024-05-08T21:23:27.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.420+0000 7f9bf57be700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9bf015b900 0x7f9bf00cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:27.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.420+0000 7f9bf4fbd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9bf0099a50 0x7f9bf0099f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50524/0 (socket says 172.21.15.3:50524) 2024-05-08T21:23:27.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.420+0000 7f9bf4fbd700 1 -- 172.21.15.3:0/2779887942 learned_addr learned my addr 172.21.15.3:0/2779887942 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:27.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.421+0000 7f9bf5fbf700 1 --2- 172.21.15.3:0/2779887942 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf009a610 0x7f9bf00b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:27.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.421+0000 7f9bf57be700 1 -- 172.21.15.3:0/2779887942 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf009a610 msgr2=0x7f9bf00b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:27.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.421+0000 7f9bf57be700 1 --2- 172.21.15.3:0/2779887942 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf009a610 0x7f9bf00b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.421+0000 7f9bf57be700 1 -- 172.21.15.3:0/2779887942 >> v1:172.21.15.3:6789/0 conn(0x7f9bf0099a50 legacy=0x7f9bf0099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:27.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.421+0000 7f9be7fff700 1 -- 172.21.15.3:0/2779887942 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531338997 0 0) 0x7f9bf00b1410 con 0x7f9bf0099a50 2024-05-08T21:23:27.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.421+0000 7f9bf57be700 1 -- 172.21.15.3:0/2779887942 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf00b10a0 con 0x7f9bf015b900 2024-05-08T21:23:27.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.421+0000 7f9bf57be700 1 --2- 172.21.15.3:0/2779887942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf015b900 0x7f9bf00cf9e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9bec009840 tx=0x7f9bec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c93896e49f95ea71 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:27.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.421+0000 7f9be7fff700 1 -- 172.21.15.3:0/2779887942 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec011830 con 0x7f9bf015b900 2024-05-08T21:23:27.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.421+0000 7f9be7fff700 1 -- 172.21.15.3:0/2779887942 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bec011990 con 0x7f9bf015b900 2024-05-08T21:23:27.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.421+0000 7f9be7fff700 1 -- 172.21.15.3:0/2779887942 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec011d20 con 0x7f9bf015b900 2024-05-08T21:23:27.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.422+0000 7f9bf7a22700 1 -- 172.21.15.3:0/2779887942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf015b900 msgr2=0x7f9bf00cf9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:27.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.422+0000 7f9bf7a22700 1 --2- 172.21.15.3:0/2779887942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf015b900 0x7f9bf00cf9e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9bec009840 tx=0x7f9bec009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:27.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.422+0000 7f9bf7a22700 1 -- 172.21.15.3:0/2779887942 shutdown_connections 2024-05-08T21:23:27.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.422+0000 7f9bf7a22700 1 --2- 172.21.15.3:0/2779887942 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf009a610 0x7f9bf00b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.422+0000 7f9bf7a22700 1 --2- 172.21.15.3:0/2779887942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf015b900 0x7f9bf00cf9e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.422+0000 7f9bf7a22700 1 -- 172.21.15.3:0/2779887942 >> 172.21.15.3:0/2779887942 conn(0x7f9bf00b2f20 msgr2=0x7f9bf00b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:27.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.422+0000 7f9bf7a22700 1 -- 172.21.15.3:0/2779887942 shutdown_connections 2024-05-08T21:23:27.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.422+0000 7f9bf7a22700 1 -- 172.21.15.3:0/2779887942 wait complete. 2024-05-08T21:23:27.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.423+0000 7f9bf7a22700 1 Processor -- start 2024-05-08T21:23:27.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.423+0000 7f9bf7a22700 1 -- start start 2024-05-08T21:23:27.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.424+0000 7f9bf7a22700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf0099a50 0x7f9bf014c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:27.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.424+0000 7f9bf7a22700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf009a610 0x7f9bf006ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:27.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.424+0000 7f9bf7a22700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf015b900 0x7f9bf00b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:27.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.424+0000 7f9bf7a22700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf006d480 con 0x7f9bf0099a50 2024-05-08T21:23:27.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.424+0000 7f9bf7a22700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf014c970 con 0x7f9bf009a610 2024-05-08T21:23:27.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.424+0000 7f9bf7a22700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf006c3e0 con 0x7f9bf015b900 2024-05-08T21:23:27.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.424+0000 7f9bf57be700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf0099a50 0x7f9bf014c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:27.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.424+0000 7f9bf57be700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf0099a50 0x7f9bf014c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42142/0 (socket says 172.21.15.3:42142) 2024-05-08T21:23:27.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.424+0000 7f9bf57be700 1 -- 172.21.15.3:0/1603994209 learned_addr learned my addr 172.21.15.3:0/1603994209 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:27.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.424+0000 7f9bf57be700 1 -- 172.21.15.3:0/1603994209 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf015b900 msgr2=0x7f9bf00b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:27.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.424+0000 7f9bf5fbf700 1 --2- 172.21.15.3:0/1603994209 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf015b900 0x7f9bf00b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:27.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.424+0000 7f9bf4fbd700 1 --2- 172.21.15.3:0/1603994209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf009a610 0x7f9bf006ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:27.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.424+0000 7f9bf57be700 1 --2- 172.21.15.3:0/1603994209 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf015b900 0x7f9bf00b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.424+0000 7f9bf57be700 1 -- 172.21.15.3:0/1603994209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf009a610 msgr2=0x7f9bf006ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:27.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.424+0000 7f9bf57be700 1 --2- 172.21.15.3:0/1603994209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf009a610 0x7f9bf006ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.424+0000 7f9bf57be700 1 -- 172.21.15.3:0/1603994209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bec0094a0 con 0x7f9bf0099a50 2024-05-08T21:23:27.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.425+0000 7f9bf57be700 1 --2- 172.21.15.3:0/1603994209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf0099a50 0x7f9bf014c430 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0090b0 tx=0x7f9bec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:27.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.425+0000 7f9be5ffb700 1 -- 172.21.15.3:0/1603994209 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec0038a0 con 0x7f9bf0099a50 2024-05-08T21:23:27.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.425+0000 7f9be5ffb700 1 -- 172.21.15.3:0/1603994209 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bec01e070 con 0x7f9bf0099a50 2024-05-08T21:23:27.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.425+0000 7f9bf7a22700 1 -- 172.21.15.3:0/1603994209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf006c6c0 con 0x7f9bf0099a50 2024-05-08T21:23:27.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.425+0000 7f9be5ffb700 1 -- 172.21.15.3:0/1603994209 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec003c20 con 0x7f9bf0099a50 2024-05-08T21:23:27.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.425+0000 7f9bf7a22700 1 -- 172.21.15.3:0/1603994209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf006cc80 con 0x7f9bf0099a50 2024-05-08T21:23:27.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.426+0000 7f9bf7a22700 1 -- 172.21.15.3:0/1603994209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf0068000 con 0x7f9bf0099a50 2024-05-08T21:23:27.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.427+0000 7f9be5ffb700 1 -- 172.21.15.3:0/1603994209 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9bec02d450 con 0x7f9bf0099a50 2024-05-08T21:23:27.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.427+0000 7f9be5ffb700 1 --2- 172.21.15.3:0/1603994209 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9bd406ac90 0x7f9bd406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:27.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.427+0000 7f9be5ffb700 1 -- 172.21.15.3:0/1603994209 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f9bec08a5a0 con 0x7f9bf0099a50 2024-05-08T21:23:27.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.428+0000 7f9bf4fbd700 1 --2- 172.21.15.3:0/1603994209 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9bd406ac90 0x7f9bd406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:27.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.428+0000 7f9bf4fbd700 1 --2- 172.21.15.3:0/1603994209 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9bd406ac90 0x7f9bd406d150 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00b06d0 tx=0x7f9be0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:27.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.432+0000 7f9be5ffb700 1 -- 172.21.15.3:0/1603994209 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9bec0555f0 con 0x7f9bf0099a50 2024-05-08T21:23:27.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.699+0000 7f9bf7a22700 1 -- 172.21.15.3:0/1603994209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9bf0073ec0 con 0x7f9bf0099a50 2024-05-08T21:23:27.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.835+0000 7f9be5ffb700 1 -- 172.21.15.3:0/1603994209 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9bec058c10 con 0x7f9bf0099a50 2024-05-08T21:23:27.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.838+0000 7f9bf7a22700 1 -- 172.21.15.3:0/1603994209 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9bd406ac90 msgr2=0x7f9bd406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:27.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.838+0000 7f9bf7a22700 1 --2- 172.21.15.3:0/1603994209 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9bd406ac90 0x7f9bd406d150 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00b06d0 tx=0x7f9be0009040 comp rx=0 tx=0).stop 2024-05-08T21:23:27.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.838+0000 7f9bf7a22700 1 -- 172.21.15.3:0/1603994209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf0099a50 msgr2=0x7f9bf014c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:27.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.838+0000 7f9bf7a22700 1 --2- 172.21.15.3:0/1603994209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf0099a50 0x7f9bf014c430 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0090b0 tx=0x7f9bec004750 comp rx=0 tx=0).stop 2024-05-08T21:23:27.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.838+0000 7f9bf7a22700 1 -- 172.21.15.3:0/1603994209 shutdown_connections 2024-05-08T21:23:27.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.838+0000 7f9bf7a22700 1 --2- 172.21.15.3:0/1603994209 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf015b900 0x7f9bf00b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.838+0000 7f9bf7a22700 1 --2- 172.21.15.3:0/1603994209 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9bd406ac90 0x7f9bd406d150 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.838+0000 7f9bf7a22700 1 --2- 172.21.15.3:0/1603994209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf0099a50 0x7f9bf014c430 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.838+0000 7f9bf7a22700 1 --2- 172.21.15.3:0/1603994209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf009a610 0x7f9bf006ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:27.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.838+0000 7f9bf7a22700 1 -- 172.21.15.3:0/1603994209 >> 172.21.15.3:0/1603994209 conn(0x7f9bf00b2f20 msgr2=0x7f9bf00cbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:27.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.838+0000 7f9bf7a22700 1 -- 172.21.15.3:0/1603994209 shutdown_connections 2024-05-08T21:23:27.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.838+0000 7f9bf7a22700 1 -- 172.21.15.3:0/1603994209 wait complete. 2024-05-08T21:23:27.841 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-08T21:23:27.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-08T21:23:27.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.995+0000 7f59546ec700 1 Processor -- start 2024-05-08T21:23:27.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.995+0000 7f59546ec700 1 -- start start 2024-05-08T21:23:27.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.996+0000 7f59546ec700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f594c06a960 0x7f594c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:27.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.996+0000 7f59546ec700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f594c0ce230 0x7f594c0ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:27.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.996+0000 7f59546ec700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594c0cffb0 con 0x7f594c15ba10 2024-05-08T21:23:27.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.996+0000 7f59546ec700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594c06b350 con 0x7f594c0ce230 2024-05-08T21:23:27.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.996+0000 7f59546ec700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594c15f3e0 con 0x7f594c06a960 2024-05-08T21:23:27.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.996+0000 7f5952c89700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f594c0ce230 0x7f594c0ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:27.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.996+0000 7f5952c89700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f594c0ce230 0x7f594c0ce680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42152/0 (socket says 172.21.15.3:42152) 2024-05-08T21:23:27.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.996+0000 7f5952c89700 1 -- 172.21.15.3:0/1075196678 learned_addr learned my addr 172.21.15.3:0/1075196678 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:27.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.996+0000 7f5951c87700 1 --2- 172.21.15.3:0/1075196678 >> v2:172.21.15.186:3300/0 conn(0x7f594c06a960 0x7f594c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:28.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.996+0000 7f5952c89700 1 -- 172.21.15.3:0/1075196678 >> v2:172.21.15.186:3300/0 conn(0x7f594c06a960 msgr2=0x7f594c06ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:28.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.996+0000 7f5952c89700 1 --2- 172.21.15.3:0/1075196678 >> v2:172.21.15.186:3300/0 conn(0x7f594c06a960 0x7f594c06ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:28.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.996+0000 7f5952c89700 1 -- 172.21.15.3:0/1075196678 >> v1:172.21.15.186:6789/0 conn(0x7f594c15ba10 legacy=0x7f594c0cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:28.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.996+0000 7f5952c89700 1 -- 172.21.15.3:0/1075196678 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c15f5f0 con 0x7f594c0ce230 2024-05-08T21:23:28.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.996+0000 7f5952c89700 1 --2- 172.21.15.3:0/1075196678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f594c0ce230 0x7f594c0ce680 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f5944009e50 tx=0x7f594400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe1f6bdb097fef5 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:28.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.997+0000 7f5951486700 1 -- 172.21.15.3:0/1075196678 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594400e070 con 0x7f594c0ce230 2024-05-08T21:23:28.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.997+0000 7f5951486700 1 -- 172.21.15.3:0/1075196678 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594400b8d0 con 0x7f594c0ce230 2024-05-08T21:23:28.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.997+0000 7f5951486700 1 -- 172.21.15.3:0/1075196678 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594400bc80 con 0x7f594c0ce230 2024-05-08T21:23:28.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.997+0000 7f59546ec700 1 -- 172.21.15.3:0/1075196678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f594c0ce230 msgr2=0x7f594c0ce680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:28.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.997+0000 7f59546ec700 1 --2- 172.21.15.3:0/1075196678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f594c0ce230 0x7f594c0ce680 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f5944009e50 tx=0x7f594400b330 comp rx=0 tx=0).stop 2024-05-08T21:23:28.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.997+0000 7f59546ec700 1 -- 172.21.15.3:0/1075196678 shutdown_connections 2024-05-08T21:23:28.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.997+0000 7f59546ec700 1 --2- 172.21.15.3:0/1075196678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f594c0ce230 0x7f594c0ce680 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:28.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.997+0000 7f59546ec700 1 --2- 172.21.15.3:0/1075196678 >> v2:172.21.15.186:3300/0 conn(0x7f594c06a960 0x7f594c06ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:28.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.997+0000 7f59546ec700 1 -- 172.21.15.3:0/1075196678 >> 172.21.15.3:0/1075196678 conn(0x7f594c099a50 msgr2=0x7f594c0d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:28.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.997+0000 7f59546ec700 1 -- 172.21.15.3:0/1075196678 shutdown_connections 2024-05-08T21:23:28.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.997+0000 7f59546ec700 1 -- 172.21.15.3:0/1075196678 wait complete. 2024-05-08T21:23:28.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.999+0000 7f59546ec700 1 Processor -- start 2024-05-08T21:23:28.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.999+0000 7f59546ec700 1 -- start start 2024-05-08T21:23:28.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:27.999+0000 7f59546ec700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f594c06a960 0x7f594c155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:28.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f59546ec700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f594c0ce230 0x7f594c15a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:28.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f59546ec700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f594c15ba10 0x7f594c15a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:28.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f59546ec700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c0c80f0 con 0x7f594c06a960 2024-05-08T21:23:28.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f59546ec700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c0c83d0 con 0x7f594c0ce230 2024-05-08T21:23:28.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f59546ec700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594c0c86b0 con 0x7f594c15ba10 2024-05-08T21:23:28.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f5952488700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f594c06a960 0x7f594c155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:28.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f5952488700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f594c06a960 0x7f594c155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42162/0 (socket says 172.21.15.3:42162) 2024-05-08T21:23:28.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f5952488700 1 -- 172.21.15.3:0/4276844361 learned_addr learned my addr 172.21.15.3:0/4276844361 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:28.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f5952c89700 1 --2- 172.21.15.3:0/4276844361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f594c15ba10 0x7f594c15a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:28.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f5951c87700 1 --2- 172.21.15.3:0/4276844361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f594c0ce230 0x7f594c15a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:28.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f5952488700 1 -- 172.21.15.3:0/4276844361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f594c15ba10 msgr2=0x7f594c15a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:28.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f5952488700 1 --2- 172.21.15.3:0/4276844361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f594c15ba10 0x7f594c15a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:28.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f5952488700 1 -- 172.21.15.3:0/4276844361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f594c0ce230 msgr2=0x7f594c15a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:28.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f5952488700 1 --2- 172.21.15.3:0/4276844361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f594c0ce230 0x7f594c15a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:28.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f5952488700 1 -- 172.21.15.3:0/4276844361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5944009b30 con 0x7f594c06a960 2024-05-08T21:23:28.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f5952488700 1 --2- 172.21.15.3:0/4276844361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f594c06a960 0x7f594c155b20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f594800c490 tx=0x7f594800c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:28.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f5942ffd700 1 -- 172.21.15.3:0/4276844361 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594800cea0 con 0x7f594c06a960 2024-05-08T21:23:28.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f5942ffd700 1 -- 172.21.15.3:0/4276844361 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59480052f0 con 0x7f594c06a960 2024-05-08T21:23:28.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f59546ec700 1 -- 172.21.15.3:0/4276844361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594c0c8ce0 con 0x7f594c06a960 2024-05-08T21:23:28.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.000+0000 7f5942ffd700 1 -- 172.21.15.3:0/4276844361 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5948006860 con 0x7f594c06a960 2024-05-08T21:23:28.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.001+0000 7f59546ec700 1 -- 172.21.15.3:0/4276844361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594c0c8990 con 0x7f594c06a960 2024-05-08T21:23:28.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.002+0000 7f59546ec700 1 -- 172.21.15.3:0/4276844361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f594c074bc0 con 0x7f594c06a960 2024-05-08T21:23:28.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.002+0000 7f5942ffd700 1 -- 172.21.15.3:0/4276844361 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5948003680 con 0x7f594c06a960 2024-05-08T21:23:28.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.002+0000 7f5942ffd700 1 --2- 172.21.15.3:0/4276844361 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f593006ac90 0x7f593006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:28.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.003+0000 7f5942ffd700 1 -- 172.21.15.3:0/4276844361 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7f5948088790 con 0x7f594c06a960 2024-05-08T21:23:28.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.003+0000 7f5951c87700 1 --2- 172.21.15.3:0/4276844361 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f593006ac90 0x7f593006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:28.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.003+0000 7f5951c87700 1 --2- 172.21.15.3:0/4276844361 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f593006ac90 0x7f593006d150 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f594c1f46a0 tx=0x7f593c009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:28.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.007+0000 7f5942ffd700 1 -- 172.21.15.3:0/4276844361 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f59480536b0 con 0x7f594c06a960 2024-05-08T21:23:28.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.274+0000 7f59546ec700 1 -- 172.21.15.3:0/4276844361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f594c0c6b30 con 0x7f594c06a960 2024-05-08T21:23:28.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.841+0000 7f5942ffd700 1 -- 172.21.15.3:0/4276844361 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5948056cd0 con 0x7f594c06a960 2024-05-08T21:23:28.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.844+0000 7f59546ec700 1 -- 172.21.15.3:0/4276844361 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f593006ac90 msgr2=0x7f593006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.844+0000 7f59546ec700 1 --2- 172.21.15.3:0/4276844361 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f593006ac90 0x7f593006d150 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f594c1f46a0 tx=0x7f593c009210 comp rx=0 tx=0).stop 2024-05-08T21:23:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.844+0000 7f59546ec700 1 -- 172.21.15.3:0/4276844361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f594c06a960 msgr2=0x7f594c155b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.844+0000 7f59546ec700 1 --2- 172.21.15.3:0/4276844361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f594c06a960 0x7f594c155b20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f594800c490 tx=0x7f594800c7a0 comp rx=0 tx=0).stop 2024-05-08T21:23:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.844+0000 7f59546ec700 1 -- 172.21.15.3:0/4276844361 shutdown_connections 2024-05-08T21:23:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.844+0000 7f59546ec700 1 --2- 172.21.15.3:0/4276844361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f594c15ba10 0x7f594c15a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.844+0000 7f59546ec700 1 --2- 172.21.15.3:0/4276844361 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f593006ac90 0x7f593006d150 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.844+0000 7f59546ec700 1 --2- 172.21.15.3:0/4276844361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f594c06a960 0x7f594c155b20 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.844+0000 7f59546ec700 1 --2- 172.21.15.3:0/4276844361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f594c0ce230 0x7f594c15a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.844+0000 7f59546ec700 1 -- 172.21.15.3:0/4276844361 >> 172.21.15.3:0/4276844361 conn(0x7f594c099a50 msgr2=0x7f594c0b1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.844+0000 7f59546ec700 1 -- 172.21.15.3:0/4276844361 shutdown_connections 2024-05-08T21:23:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:28.844+0000 7f59546ec700 1 -- 172.21.15.3:0/4276844361 wait complete. 2024-05-08T21:23:28.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-08T21:23:29.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.003+0000 7faf3c428700 1 Processor -- start 2024-05-08T21:23:29.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.003+0000 7faf3c428700 1 -- start start 2024-05-08T21:23:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.004+0000 7faf3c428700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faf3415a7c0 0x7faf340cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.005+0000 7faf3c428700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf340cd7c0 con 0x7faf340ca130 2024-05-08T21:23:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.005+0000 7faf3c428700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf340cd1b0 con 0x7faf340727b0 2024-05-08T21:23:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.005+0000 7faf3c428700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf3415aba0 con 0x7faf3415a7c0 2024-05-08T21:23:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.005+0000 7faf3a9c5700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7faf340ca130 0x7faf340b24f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50540/0 (socket says 172.21.15.3:50540) 2024-05-08T21:23:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.005+0000 7faf3a9c5700 1 -- 172.21.15.3:0/268174835 learned_addr learned my addr 172.21.15.3:0/268174835 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.005+0000 7faf3a1c4700 1 --2- 172.21.15.3:0/268174835 >> v2:172.21.15.186:3300/0 conn(0x7faf3415a7c0 0x7faf340cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:29.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.005+0000 7faf391c2700 1 -- 172.21.15.3:0/268174835 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583394986 0 0) 0x7faf340cd7c0 con 0x7faf340ca130 2024-05-08T21:23:29.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.005+0000 7faf391c2700 1 -- 172.21.15.3:0/268174835 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf200035b0 con 0x7faf340ca130 2024-05-08T21:23:29.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.005+0000 7faf3a1c4700 1 -- 172.21.15.3:0/268174835 >> v1:172.21.15.186:6789/0 conn(0x7faf340727b0 legacy=0x7faf340c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:29.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.005+0000 7faf3a1c4700 1 -- 172.21.15.3:0/268174835 >> v1:172.21.15.3:6789/0 conn(0x7faf340ca130 legacy=0x7faf340b24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:29.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.005+0000 7faf3a1c4700 1 -- 172.21.15.3:0/268174835 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf340b3e70 con 0x7faf3415a7c0 2024-05-08T21:23:29.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.005+0000 7faf3a1c4700 1 --2- 172.21.15.3:0/268174835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf3415a7c0 0x7faf340cdc50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7faf30009840 tx=0x7faf30009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d78f2b2412795201 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:29.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.006+0000 7faf391c2700 1 -- 172.21.15.3:0/268174835 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf30011830 con 0x7faf3415a7c0 2024-05-08T21:23:29.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.006+0000 7faf391c2700 1 -- 172.21.15.3:0/268174835 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf30011990 con 0x7faf3415a7c0 2024-05-08T21:23:29.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.006+0000 7faf391c2700 1 -- 172.21.15.3:0/268174835 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf30011d20 con 0x7faf3415a7c0 2024-05-08T21:23:29.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.006+0000 7faf3c428700 1 -- 172.21.15.3:0/268174835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf3415a7c0 msgr2=0x7faf340cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:29.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.006+0000 7faf3c428700 1 --2- 172.21.15.3:0/268174835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf3415a7c0 0x7faf340cdc50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7faf30009840 tx=0x7faf30009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:29.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.006+0000 7faf3c428700 1 -- 172.21.15.3:0/268174835 shutdown_connections 2024-05-08T21:23:29.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.006+0000 7faf3c428700 1 --2- 172.21.15.3:0/268174835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf3415a7c0 0x7faf340cdc50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:29.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.006+0000 7faf3c428700 1 -- 172.21.15.3:0/268174835 >> 172.21.15.3:0/268174835 conn(0x7faf3406a960 msgr2=0x7faf3406ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:29.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.006+0000 7faf3c428700 1 -- 172.21.15.3:0/268174835 shutdown_connections 2024-05-08T21:23:29.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.006+0000 7faf3c428700 1 -- 172.21.15.3:0/268174835 wait complete. 2024-05-08T21:23:29.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.008+0000 7faf3c428700 1 Processor -- start 2024-05-08T21:23:29.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.008+0000 7faf3c428700 1 -- start start 2024-05-08T21:23:29.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.008+0000 7faf3c428700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf340727b0 0x7faf340b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:29.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.008+0000 7faf3c428700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf340ca130 0x7faf340aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:29.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.008+0000 7faf3c428700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf3415a7c0 0x7faf341574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:29.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.008+0000 7faf3c428700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf340aff30 con 0x7faf340727b0 2024-05-08T21:23:29.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.008+0000 7faf3c428700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf340af050 con 0x7faf340ca130 2024-05-08T21:23:29.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.008+0000 7faf3c428700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf340af320 con 0x7faf3415a7c0 2024-05-08T21:23:29.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.009+0000 7faf3a1c4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf340727b0 0x7faf340b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:29.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.009+0000 7faf3a1c4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf340727b0 0x7faf340b03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42170/0 (socket says 172.21.15.3:42170) 2024-05-08T21:23:29.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.009+0000 7faf3a1c4700 1 -- 172.21.15.3:0/3971061376 learned_addr learned my addr 172.21.15.3:0/3971061376 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:29.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.009+0000 7faf3a9c5700 1 --2- 172.21.15.3:0/3971061376 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf3415a7c0 0x7faf341574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:29.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.009+0000 7faf399c3700 1 --2- 172.21.15.3:0/3971061376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf340ca130 0x7faf340aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:29.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.009+0000 7faf3a1c4700 1 -- 172.21.15.3:0/3971061376 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf3415a7c0 msgr2=0x7faf341574d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:29.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.009+0000 7faf3a1c4700 1 --2- 172.21.15.3:0/3971061376 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf3415a7c0 0x7faf341574d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:29.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.009+0000 7faf3a1c4700 1 -- 172.21.15.3:0/3971061376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf340ca130 msgr2=0x7faf340aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:29.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.009+0000 7faf3a1c4700 1 --2- 172.21.15.3:0/3971061376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf340ca130 0x7faf340aeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:29.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.009+0000 7faf3a1c4700 1 -- 172.21.15.3:0/3971061376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf300094a0 con 0x7faf340727b0 2024-05-08T21:23:29.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.009+0000 7faf3a1c4700 1 --2- 172.21.15.3:0/3971061376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf340727b0 0x7faf340b03c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7faf30009b90 tx=0x7faf30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:29.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.009+0000 7faf2affd700 1 -- 172.21.15.3:0/3971061376 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf300038a0 con 0x7faf340727b0 2024-05-08T21:23:29.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.009+0000 7faf2affd700 1 -- 172.21.15.3:0/3971061376 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3001e070 con 0x7faf340727b0 2024-05-08T21:23:29.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.009+0000 7faf3c428700 1 -- 172.21.15.3:0/3971061376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf340af600 con 0x7faf340727b0 2024-05-08T21:23:29.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.009+0000 7faf2affd700 1 -- 172.21.15.3:0/3971061376 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf30003c40 con 0x7faf340727b0 2024-05-08T21:23:29.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.009+0000 7faf3c428700 1 -- 172.21.15.3:0/3971061376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf340afbc0 con 0x7faf340727b0 2024-05-08T21:23:29.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.010+0000 7faf3c428700 1 -- 172.21.15.3:0/3971061376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf34074bc0 con 0x7faf340727b0 2024-05-08T21:23:29.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.011+0000 7faf2affd700 1 -- 172.21.15.3:0/3971061376 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faf3002d450 con 0x7faf340727b0 2024-05-08T21:23:29.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.011+0000 7faf2affd700 1 --2- 172.21.15.3:0/3971061376 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faf1806ac90 0x7faf1806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:29.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.011+0000 7faf2affd700 1 -- 172.21.15.3:0/3971061376 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6248+0+0 (secure 0 0 0) 0x7faf3008a4b0 con 0x7faf340727b0 2024-05-08T21:23:29.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.012+0000 7faf399c3700 1 --2- 172.21.15.3:0/3971061376 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faf1806ac90 0x7faf1806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:29.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.012+0000 7faf399c3700 1 --2- 172.21.15.3:0/3971061376 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faf1806ac90 0x7faf1806d150 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7faf341f69c0 tx=0x7faf24009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:29.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.016+0000 7faf2affd700 1 -- 172.21.15.3:0/3971061376 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7faf30055500 con 0x7faf340727b0 2024-05-08T21:23:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.231+0000 7faf3c428700 1 -- 172.21.15.3:0/3971061376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faf340ced00 con 0x7faf340727b0 2024-05-08T21:23:29.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.843+0000 7faf2affd700 1 -- 172.21.15.3:0/3971061376 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v43) v1 ==== 181+0+0 (secure 0 0 0) 0x7faf30058b20 con 0x7faf340727b0 2024-05-08T21:23:29.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.848+0000 7faf3c428700 1 -- 172.21.15.3:0/3971061376 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faf1806ac90 msgr2=0x7faf1806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:29.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.848+0000 7faf3c428700 1 --2- 172.21.15.3:0/3971061376 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faf1806ac90 0x7faf1806d150 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7faf341f69c0 tx=0x7faf24009040 comp rx=0 tx=0).stop 2024-05-08T21:23:29.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.848+0000 7faf3c428700 1 -- 172.21.15.3:0/3971061376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf340727b0 msgr2=0x7faf340b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:29.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.848+0000 7faf3c428700 1 --2- 172.21.15.3:0/3971061376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf340727b0 0x7faf340b03c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7faf30009b90 tx=0x7faf30004750 comp rx=0 tx=0).stop 2024-05-08T21:23:29.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.848+0000 7faf3c428700 1 -- 172.21.15.3:0/3971061376 shutdown_connections 2024-05-08T21:23:29.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.848+0000 7faf3c428700 1 --2- 172.21.15.3:0/3971061376 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf3415a7c0 0x7faf341574d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:29.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.848+0000 7faf3c428700 1 --2- 172.21.15.3:0/3971061376 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faf1806ac90 0x7faf1806d150 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:29.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.848+0000 7faf3c428700 1 --2- 172.21.15.3:0/3971061376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf340727b0 0x7faf340b03c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:29.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.848+0000 7faf3c428700 1 --2- 172.21.15.3:0/3971061376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf340ca130 0x7faf340aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:29.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.848+0000 7faf3c428700 1 -- 172.21.15.3:0/3971061376 >> 172.21.15.3:0/3971061376 conn(0x7faf3406a960 msgr2=0x7faf3406ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:29.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.849+0000 7faf3c428700 1 -- 172.21.15.3:0/3971061376 shutdown_connections 2024-05-08T21:23:29.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:29.849+0000 7faf3c428700 1 -- 172.21.15.3:0/3971061376 wait complete. 2024-05-08T21:23:29.852 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2024-05-08T21:23:29.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-08T21:23:30.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.018+0000 7f984729e700 1 Processor -- start 2024-05-08T21:23:30.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.019+0000 7f984729e700 1 -- start start 2024-05-08T21:23:30.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.019+0000 7f984729e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98400d1e50 0x7f98400ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.019+0000 7f984729e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f98400d2230 0x7f9840160c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.019+0000 7f984729e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98400cf560 con 0x7f98400b2610 2024-05-08T21:23:30.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.019+0000 7f984729e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f984006cae0 con 0x7f98400d2230 2024-05-08T21:23:30.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.019+0000 7f984729e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98400ce950 con 0x7f98400d1e50 2024-05-08T21:23:30.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.020+0000 7f984583b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f98400b2610 0x7f98400d0000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50556/0 (socket says 172.21.15.3:50556) 2024-05-08T21:23:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.020+0000 7f984583b700 1 -- 172.21.15.3:0/1033055449 learned_addr learned my addr 172.21.15.3:0/1033055449 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.020+0000 7f9844839700 1 --2- 172.21.15.3:0/1033055449 >> v2:172.21.15.186:3300/0 conn(0x7f98400d2230 0x7f9840160c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.020+0000 7f984503a700 1 --2- 172.21.15.3:0/1033055449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98400d1e50 0x7f98400ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.020+0000 7f9837fff700 1 -- 172.21.15.3:0/1033055449 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052112079 0 0) 0x7f98400cf560 con 0x7f98400b2610 2024-05-08T21:23:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.020+0000 7f9837fff700 1 -- 172.21.15.3:0/1033055449 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f982c0035b0 con 0x7f98400b2610 2024-05-08T21:23:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.020+0000 7f9844839700 1 -- 172.21.15.3:0/1033055449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98400d1e50 msgr2=0x7f98400ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.020+0000 7f9844839700 1 --2- 172.21.15.3:0/1033055449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98400d1e50 0x7f98400ce2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.020+0000 7f9844839700 1 -- 172.21.15.3:0/1033055449 >> v1:172.21.15.3:6789/0 conn(0x7f98400b2610 legacy=0x7f98400d0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.020+0000 7f9844839700 1 -- 172.21.15.3:0/1033055449 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98400cec30 con 0x7f98400d2230 2024-05-08T21:23:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.020+0000 7f984503a700 1 --2- 172.21.15.3:0/1033055449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98400d1e50 0x7f98400ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.020+0000 7f9844839700 1 --2- 172.21.15.3:0/1033055449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98400d2230 0x7f9840160c10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9830009840 tx=0x7f9830009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26682535e525cca8 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.021+0000 7f9837fff700 1 -- 172.21.15.3:0/1033055449 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9830011830 con 0x7f98400d2230 2024-05-08T21:23:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.021+0000 7f9837fff700 1 -- 172.21.15.3:0/1033055449 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9830011990 con 0x7f98400d2230 2024-05-08T21:23:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.021+0000 7f9837fff700 1 -- 172.21.15.3:0/1033055449 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9830011d20 con 0x7f98400d2230 2024-05-08T21:23:30.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.021+0000 7f984729e700 1 -- 172.21.15.3:0/1033055449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98400d2230 msgr2=0x7f9840160c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:30.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.021+0000 7f984729e700 1 --2- 172.21.15.3:0/1033055449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98400d2230 0x7f9840160c10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9830009840 tx=0x7f9830009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:30.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.021+0000 7f984729e700 1 -- 172.21.15.3:0/1033055449 shutdown_connections 2024-05-08T21:23:30.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.021+0000 7f984729e700 1 --2- 172.21.15.3:0/1033055449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98400d1e50 0x7f98400ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.021+0000 7f984729e700 1 --2- 172.21.15.3:0/1033055449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98400d2230 0x7f9840160c10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.021+0000 7f984729e700 1 -- 172.21.15.3:0/1033055449 >> 172.21.15.3:0/1033055449 conn(0x7f984008afe0 msgr2=0x7f98400d35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:30.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.021+0000 7f984729e700 1 -- 172.21.15.3:0/1033055449 shutdown_connections 2024-05-08T21:23:30.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.021+0000 7f984729e700 1 -- 172.21.15.3:0/1033055449 wait complete. 2024-05-08T21:23:30.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.023+0000 7f984729e700 1 Processor -- start 2024-05-08T21:23:30.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.023+0000 7f984729e700 1 -- start start 2024-05-08T21:23:30.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f984729e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98400b2610 0x7f984014b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f984729e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98400d1e50 0x7f98401f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f984729e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98400d2230 0x7f98400d3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f984729e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984014bee0 con 0x7f98400d1e50 2024-05-08T21:23:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f984729e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98401558c0 con 0x7f98400d2230 2024-05-08T21:23:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f984729e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98400c8a70 con 0x7f98400b2610 2024-05-08T21:23:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f9844839700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98400d1e50 0x7f98401f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f9844839700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98400d1e50 0x7f98401f3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42180/0 (socket says 172.21.15.3:42180) 2024-05-08T21:23:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f9844839700 1 -- 172.21.15.3:0/2274220329 learned_addr learned my addr 172.21.15.3:0/2274220329 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f984583b700 1 --2- 172.21.15.3:0/2274220329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98400d2230 0x7f98400d3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f984503a700 1 --2- 172.21.15.3:0/2274220329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98400b2610 0x7f984014b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f9844839700 1 -- 172.21.15.3:0/2274220329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98400b2610 msgr2=0x7f984014b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f9844839700 1 --2- 172.21.15.3:0/2274220329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98400b2610 0x7f984014b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f9844839700 1 -- 172.21.15.3:0/2274220329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98400d2230 msgr2=0x7f98400d3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f9844839700 1 --2- 172.21.15.3:0/2274220329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98400d2230 0x7f98400d3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f9844839700 1 -- 172.21.15.3:0/2274220329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983c0094a0 con 0x7f98400d1e50 2024-05-08T21:23:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f984503a700 1 --2- 172.21.15.3:0/2274220329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98400b2610 0x7f984014b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f9844839700 1 --2- 172.21.15.3:0/2274220329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98400d1e50 0x7f98401f3b90 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9830009fd0 tx=0x7f9830004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f9835ffb700 1 -- 172.21.15.3:0/2274220329 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98300036a0 con 0x7f98400d1e50 2024-05-08T21:23:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f9835ffb700 1 -- 172.21.15.3:0/2274220329 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983001e070 con 0x7f98400d1e50 2024-05-08T21:23:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f984729e700 1 -- 172.21.15.3:0/2274220329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98300094a0 con 0x7f98400d1e50 2024-05-08T21:23:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.024+0000 7f9835ffb700 1 -- 172.21.15.3:0/2274220329 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9830003a60 con 0x7f98400d1e50 2024-05-08T21:23:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.025+0000 7f984729e700 1 -- 172.21.15.3:0/2274220329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98400c9030 con 0x7f98400d1e50 2024-05-08T21:23:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.026+0000 7f984729e700 1 -- 172.21.15.3:0/2274220329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f984006b220 con 0x7f98400d1e50 2024-05-08T21:23:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.026+0000 7f9835ffb700 1 -- 172.21.15.3:0/2274220329 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9830003bc0 con 0x7f98400d1e50 2024-05-08T21:23:30.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.027+0000 7f9835ffb700 1 --2- 172.21.15.3:0/2274220329 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f982406ac90 0x7f982406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.027+0000 7f9835ffb700 1 -- 172.21.15.3:0/2274220329 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5810+0+0 (secure 0 0 0) 0x7f983008a300 con 0x7f98400d1e50 2024-05-08T21:23:30.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.027+0000 7f984503a700 1 --2- 172.21.15.3:0/2274220329 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f982406ac90 0x7f982406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.027+0000 7f984503a700 1 --2- 172.21.15.3:0/2274220329 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f982406ac90 0x7f982406d150 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f983c009cf0 tx=0x7f983c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:30.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.031+0000 7f9835ffb700 1 -- 172.21.15.3:0/2274220329 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9830055470 con 0x7f98400d1e50 2024-05-08T21:23:30.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.251+0000 7f984729e700 1 -- 172.21.15.3:0/2274220329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9840074bc0 con 0x7f98400d1e50 2024-05-08T21:23:30.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.251+0000 7f9835ffb700 1 -- 172.21.15.3:0/2274220329 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=-16 pool 'cephfs_data' is in use by CephFS v43) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9830058a90 con 0x7f98400d1e50 2024-05-08T21:23:30.254 INFO:teuthology.orchestra.run.smithi003.stderr:Error EBUSY: pool 'cephfs_data' is in use by CephFS 2024-05-08T21:23:30.258 DEBUG:teuthology.orchestra.run:got remote process result: 16 2024-05-08T21:23:30.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:23:30.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.394+0000 7f9bfb31b700 1 Processor -- start 2024-05-08T21:23:30.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.394+0000 7f9bfb31b700 1 -- start start 2024-05-08T21:23:30.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.395+0000 7f9bfb31b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9bf40c32f0 0x7f9bf4166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.395+0000 7f9bfb31b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf4162180 0x7f9bf40c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.395+0000 7f9bfb31b700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf4166770 con 0x7f9bf40b0240 2024-05-08T21:23:30.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.395+0000 7f9bfb31b700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf4163210 con 0x7f9bf40c32f0 2024-05-08T21:23:30.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.395+0000 7f9bfb31b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf4162970 con 0x7f9bf4162180 2024-05-08T21:23:30.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.396+0000 7f9bf90b7700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9bf40c32f0 0x7f9bf4166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.396+0000 7f9bf88b6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf4162180 0x7f9bf40c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.396+0000 7f9bf90b7700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9bf40c32f0 0x7f9bf4166230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.3:42580/0 (socket says 172.21.15.3:42580) 2024-05-08T21:23:30.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.396+0000 7f9bf90b7700 1 -- 172.21.15.3:0/1735901477 learned_addr learned my addr 172.21.15.3:0/1735901477 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:30.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.396+0000 7f9bf90b7700 1 -- 172.21.15.3:0/1735901477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf4162180 msgr2=0x7f9bf40c36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.396+0000 7f9bf90b7700 1 --2- 172.21.15.3:0/1735901477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf4162180 0x7f9bf40c36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.396+0000 7f9bf90b7700 1 -- 172.21.15.3:0/1735901477 >> v1:172.21.15.186:6789/0 conn(0x7f9bf40b0240 legacy=0x7f9bf40cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.396+0000 7f9bf90b7700 1 -- 172.21.15.3:0/1735901477 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf4162c50 con 0x7f9bf40c32f0 2024-05-08T21:23:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.396+0000 7f9bf90b7700 1 --2- 172.21.15.3:0/1735901477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf40c32f0 0x7f9bf4166230 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00098f0 tx=0x7f9bf0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa1628b1213e6a79 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.397+0000 7f9bebfff700 1 -- 172.21.15.3:0/1735901477 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0011830 con 0x7f9bf40c32f0 2024-05-08T21:23:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.397+0000 7f9bebfff700 1 -- 172.21.15.3:0/1735901477 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf0011990 con 0x7f9bf40c32f0 2024-05-08T21:23:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.397+0000 7f9bebfff700 1 -- 172.21.15.3:0/1735901477 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0011d20 con 0x7f9bf40c32f0 2024-05-08T21:23:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.397+0000 7f9bfb31b700 1 -- 172.21.15.3:0/1735901477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf40c32f0 msgr2=0x7f9bf4166230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.397+0000 7f9bfb31b700 1 --2- 172.21.15.3:0/1735901477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf40c32f0 0x7f9bf4166230 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00098f0 tx=0x7f9bf0009c00 comp rx=0 tx=0).stop 2024-05-08T21:23:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.397+0000 7f9bfb31b700 1 -- 172.21.15.3:0/1735901477 shutdown_connections 2024-05-08T21:23:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.397+0000 7f9bfb31b700 1 --2- 172.21.15.3:0/1735901477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf40c32f0 0x7f9bf4166230 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.397+0000 7f9bfb31b700 1 --2- 172.21.15.3:0/1735901477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf4162180 0x7f9bf40c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.397+0000 7f9bfb31b700 1 -- 172.21.15.3:0/1735901477 >> 172.21.15.3:0/1735901477 conn(0x7f9bf4099a50 msgr2=0x7f9bf40d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.397+0000 7f9bfb31b700 1 -- 172.21.15.3:0/1735901477 shutdown_connections 2024-05-08T21:23:30.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.397+0000 7f9bfb31b700 1 -- 172.21.15.3:0/1735901477 wait complete. 2024-05-08T21:23:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.399+0000 7f9bfb31b700 1 Processor -- start 2024-05-08T21:23:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.399+0000 7f9bfb31b700 1 -- start start 2024-05-08T21:23:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.399+0000 7f9bfb31b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf40b0240 0x7f9bf40c66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9bfb31b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf40c32f0 0x7f9bf406aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9bfb31b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf4162180 0x7f9bf406afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9bfb31b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf409dc00 con 0x7f9bf40c32f0 2024-05-08T21:23:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9bfb31b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf40c6c00 con 0x7f9bf40b0240 2024-05-08T21:23:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9bfb31b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf409cff0 con 0x7f9bf4162180 2024-05-08T21:23:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9bf88b6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf40c32f0 0x7f9bf406aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9bf88b6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf40c32f0 0x7f9bf406aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42194/0 (socket says 172.21.15.3:42194) 2024-05-08T21:23:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9bf88b6700 1 -- 172.21.15.3:0/2730237357 learned_addr learned my addr 172.21.15.3:0/2730237357 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9bf90b7700 1 --2- 172.21.15.3:0/2730237357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf40b0240 0x7f9bf40c66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9bf98b8700 1 --2- 172.21.15.3:0/2730237357 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf4162180 0x7f9bf406afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9bf88b6700 1 -- 172.21.15.3:0/2730237357 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf4162180 msgr2=0x7f9bf406afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:30.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9bf88b6700 1 --2- 172.21.15.3:0/2730237357 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf4162180 0x7f9bf406afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9bf88b6700 1 -- 172.21.15.3:0/2730237357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf40b0240 msgr2=0x7f9bf40c66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:30.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9bf88b6700 1 --2- 172.21.15.3:0/2730237357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf40b0240 0x7f9bf40c66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9bf88b6700 1 -- 172.21.15.3:0/2730237357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf0009550 con 0x7f9bf40c32f0 2024-05-08T21:23:30.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9bf88b6700 1 --2- 172.21.15.3:0/2730237357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf40c32f0 0x7f9bf406aa60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9be400e870 tx=0x7f9be400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:30.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.400+0000 7f9be9ffb700 1 -- 172.21.15.3:0/2730237357 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be4009590 con 0x7f9bf40c32f0 2024-05-08T21:23:30.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.401+0000 7f9be9ffb700 1 -- 172.21.15.3:0/2730237357 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be4013680 con 0x7f9bf40c32f0 2024-05-08T21:23:30.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.401+0000 7f9be9ffb700 1 -- 172.21.15.3:0/2730237357 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be40139f0 con 0x7f9bf40c32f0 2024-05-08T21:23:30.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.401+0000 7f9bfb31b700 1 -- 172.21.15.3:0/2730237357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf409d2d0 con 0x7f9bf40c32f0 2024-05-08T21:23:30.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.401+0000 7f9bfb31b700 1 -- 172.21.15.3:0/2730237357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf409d890 con 0x7f9bf40c32f0 2024-05-08T21:23:30.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.402+0000 7f9bfb31b700 1 -- 172.21.15.3:0/2730237357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf4067a80 con 0x7f9bf40c32f0 2024-05-08T21:23:30.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.402+0000 7f9be9ffb700 1 -- 172.21.15.3:0/2730237357 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9be4013b50 con 0x7f9bf40c32f0 2024-05-08T21:23:30.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.403+0000 7f9be9ffb700 1 --2- 172.21.15.3:0/2730237357 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9bd806ac90 0x7f9bd806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.403+0000 7f9be9ffb700 1 -- 172.21.15.3:0/2730237357 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5810+0+0 (secure 0 0 0) 0x7f9be4089310 con 0x7f9bf40c32f0 2024-05-08T21:23:30.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.403+0000 7f9bf90b7700 1 --2- 172.21.15.3:0/2730237357 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9bd806ac90 0x7f9bd806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.403+0000 7f9bf90b7700 1 --2- 172.21.15.3:0/2730237357 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9bd806ac90 0x7f9bd806d150 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0014040 tx=0x7f9bf0008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:30.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.408+0000 7f9be9ffb700 1 -- 172.21.15.3:0/2730237357 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9be4054360 con 0x7f9bf40c32f0 2024-05-08T21:23:30.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.629+0000 7f9bfb31b700 1 -- 172.21.15.3:0/2730237357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bf4068000 con 0x7f9bf40c32f0 2024-05-08T21:23:30.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.631+0000 7f9be9ffb700 1 -- 172.21.15.3:0/2730237357 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+16195 (secure 0 0 0) 0x7f9be4057980 con 0x7f9bf40c32f0 2024-05-08T21:23:30.632 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:30.632 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":43,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:23:29.839993+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-08T21:22:49.153759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"recovery_fs_metadata","create_time":"2024-05-08T21:23:03.921695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6834/3599481370":"2024-05-09T21:23:01.563392+0000","172.21.15.3:6832/3599481370":"2024-05-09T21:23:01.563392+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:23:30.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.633+0000 7f9bfb31b700 1 -- 172.21.15.3:0/2730237357 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9bd806ac90 msgr2=0x7f9bd806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:30.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.633+0000 7f9bfb31b700 1 --2- 172.21.15.3:0/2730237357 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9bd806ac90 0x7f9bd806d150 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0014040 tx=0x7f9bf0008030 comp rx=0 tx=0).stop 2024-05-08T21:23:30.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.633+0000 7f9bfb31b700 1 -- 172.21.15.3:0/2730237357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf40c32f0 msgr2=0x7f9bf406aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:30.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.633+0000 7f9bfb31b700 1 --2- 172.21.15.3:0/2730237357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf40c32f0 0x7f9bf406aa60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9be400e870 tx=0x7f9be400eb80 comp rx=0 tx=0).stop 2024-05-08T21:23:30.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.633+0000 7f9bfb31b700 1 -- 172.21.15.3:0/2730237357 shutdown_connections 2024-05-08T21:23:30.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.633+0000 7f9bfb31b700 1 --2- 172.21.15.3:0/2730237357 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf4162180 0x7f9bf406afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.633+0000 7f9bfb31b700 1 --2- 172.21.15.3:0/2730237357 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9bd806ac90 0x7f9bd806d150 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.633+0000 7f9bfb31b700 1 --2- 172.21.15.3:0/2730237357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf40b0240 0x7f9bf40c66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.633+0000 7f9bfb31b700 1 --2- 172.21.15.3:0/2730237357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf40c32f0 0x7f9bf406aa60 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.633+0000 7f9bfb31b700 1 -- 172.21.15.3:0/2730237357 >> 172.21.15.3:0/2730237357 conn(0x7f9bf4099a50 msgr2=0x7f9bf40cd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:30.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.633+0000 7f9bfb31b700 1 -- 172.21.15.3:0/2730237357 shutdown_connections 2024-05-08T21:23:30.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.633+0000 7f9bfb31b700 1 -- 172.21.15.3:0/2730237357 wait complete. 2024-05-08T21:23:30.649 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T21:21:40.595836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-05-08T21:22:49.153759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'recovery_fs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'recovery_fs_metadata', 'create_time': '2024-05-08T21:23:03.921695+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'recovery_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T21:23:30.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T21:23:30.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.787+0000 7f97e7fda700 1 Processor -- start 2024-05-08T21:23:30.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.787+0000 7f97e7fda700 1 -- start start 2024-05-08T21:23:30.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.788+0000 7f97e7fda700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f97e015b4e0 0x7f97e00b1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.788+0000 7f97e7fda700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97e00d1910 0x7f97e00cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.788+0000 7f97e7fda700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e00cf560 con 0x7f97e015b8c0 2024-05-08T21:23:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.788+0000 7f97e7fda700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e00cefe0 con 0x7f97e015b4e0 2024-05-08T21:23:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.788+0000 7f97e7fda700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e00ccff0 con 0x7f97e00d1910 2024-05-08T21:23:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.788+0000 7f97e5d76700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f97e015b4e0 0x7f97e00b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.788+0000 7f97e5575700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f97e015b8c0 0x7f97e00d1200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50570/0 (socket says 172.21.15.3:50570) 2024-05-08T21:23:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.788+0000 7f97e5575700 1 -- 172.21.15.3:0/3926710888 learned_addr learned my addr 172.21.15.3:0/3926710888 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.788+0000 7f97e6577700 1 --2- 172.21.15.3:0/3926710888 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97e00d1910 0x7f97e00cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.788+0000 7f97e5d76700 1 -- 172.21.15.3:0/3926710888 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97e00d1910 msgr2=0x7f97e00cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.788+0000 7f97e5d76700 1 --2- 172.21.15.3:0/3926710888 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97e00d1910 0x7f97e00cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.788+0000 7f97e5d76700 1 -- 172.21.15.3:0/3926710888 >> v1:172.21.15.3:6789/0 conn(0x7f97e015b8c0 legacy=0x7f97e00d1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.788+0000 7f97e5d76700 1 -- 172.21.15.3:0/3926710888 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e00cd2d0 con 0x7f97e015b4e0 2024-05-08T21:23:30.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.788+0000 7f97e5d76700 1 --2- 172.21.15.3:0/3926710888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97e015b4e0 0x7f97e00b1200 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009840 tx=0x7f97dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef5a64e6f1c7276d server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:30.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.789+0000 7f97e4d74700 1 -- 172.21.15.3:0/3926710888 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc011830 con 0x7f97e015b4e0 2024-05-08T21:23:30.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.789+0000 7f97e4d74700 1 -- 172.21.15.3:0/3926710888 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97dc011990 con 0x7f97e015b4e0 2024-05-08T21:23:30.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.789+0000 7f97e4d74700 1 -- 172.21.15.3:0/3926710888 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc011d20 con 0x7f97e015b4e0 2024-05-08T21:23:30.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.789+0000 7f97e7fda700 1 -- 172.21.15.3:0/3926710888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97e015b4e0 msgr2=0x7f97e00b1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:30.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.789+0000 7f97e7fda700 1 --2- 172.21.15.3:0/3926710888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97e015b4e0 0x7f97e00b1200 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009840 tx=0x7f97dc009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:30.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.789+0000 7f97e7fda700 1 -- 172.21.15.3:0/3926710888 shutdown_connections 2024-05-08T21:23:30.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.789+0000 7f97e7fda700 1 --2- 172.21.15.3:0/3926710888 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97e00d1910 0x7f97e00cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.789+0000 7f97e7fda700 1 --2- 172.21.15.3:0/3926710888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97e015b4e0 0x7f97e00b1200 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.789+0000 7f97e7fda700 1 -- 172.21.15.3:0/3926710888 >> 172.21.15.3:0/3926710888 conn(0x7f97e008afe0 msgr2=0x7f97e008b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:30.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.789+0000 7f97e7fda700 1 -- 172.21.15.3:0/3926710888 shutdown_connections 2024-05-08T21:23:30.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.789+0000 7f97e7fda700 1 -- 172.21.15.3:0/3926710888 wait complete. 2024-05-08T21:23:30.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.790+0000 7f97e7fda700 1 Processor -- start 2024-05-08T21:23:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.791+0000 7f97e7fda700 1 -- start start 2024-05-08T21:23:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.791+0000 7f97e7fda700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97e00d1910 0x7f97e00cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.791+0000 7f97e7fda700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97e015b4e0 0x7f97e00cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.791+0000 7f97e7fda700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97e015b8c0 0x7f97e0147930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.791+0000 7f97e7fda700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e0146370 con 0x7f97e015b4e0 2024-05-08T21:23:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.791+0000 7f97e7fda700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e015da50 con 0x7f97e00d1910 2024-05-08T21:23:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.791+0000 7f97e7fda700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e01459d0 con 0x7f97e015b8c0 2024-05-08T21:23:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.791+0000 7f97e5575700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97e015b4e0 0x7f97e00cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.791+0000 7f97e5575700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97e015b4e0 0x7f97e00cb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42214/0 (socket says 172.21.15.3:42214) 2024-05-08T21:23:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.791+0000 7f97e5575700 1 -- 172.21.15.3:0/2920374190 learned_addr learned my addr 172.21.15.3:0/2920374190 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.792+0000 7f97e6577700 1 --2- 172.21.15.3:0/2920374190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97e015b8c0 0x7f97e0147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.792+0000 7f97e5575700 1 -- 172.21.15.3:0/2920374190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97e015b8c0 msgr2=0x7f97e0147930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.792+0000 7f97e5d76700 1 --2- 172.21.15.3:0/2920374190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97e00d1910 0x7f97e00cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.792+0000 7f97e5575700 1 --2- 172.21.15.3:0/2920374190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97e015b8c0 0x7f97e0147930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.792+0000 7f97e5575700 1 -- 172.21.15.3:0/2920374190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97e00d1910 msgr2=0x7f97e00cabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.792+0000 7f97e5575700 1 --2- 172.21.15.3:0/2920374190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97e00d1910 0x7f97e00cabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.792+0000 7f97e5575700 1 -- 172.21.15.3:0/2920374190 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc0094a0 con 0x7f97e015b4e0 2024-05-08T21:23:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.792+0000 7f97e6577700 1 --2- 172.21.15.3:0/2920374190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97e015b8c0 0x7f97e0147930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.792+0000 7f97e5d76700 1 --2- 172.21.15.3:0/2920374190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97e00d1910 0x7f97e00cabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.792+0000 7f97e5575700 1 --2- 172.21.15.3:0/2920374190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97e015b4e0 0x7f97e00cb120 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f97d000c430 tx=0x7f97d000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.792+0000 7f97d67fc700 1 -- 172.21.15.3:0/2920374190 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0005d70 con 0x7f97e015b4e0 2024-05-08T21:23:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.792+0000 7f97d67fc700 1 -- 172.21.15.3:0/2920374190 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d00070d0 con 0x7f97e015b4e0 2024-05-08T21:23:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.792+0000 7f97e7fda700 1 -- 172.21.15.3:0/2920374190 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e0145cb0 con 0x7f97e015b4e0 2024-05-08T21:23:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.792+0000 7f97d67fc700 1 -- 172.21.15.3:0/2920374190 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0007460 con 0x7f97e015b4e0 2024-05-08T21:23:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.792+0000 7f97e7fda700 1 -- 172.21.15.3:0/2920374190 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e00c84e0 con 0x7f97e015b4e0 2024-05-08T21:23:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.793+0000 7f97e7fda700 1 -- 172.21.15.3:0/2920374190 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e01464b0 con 0x7f97e015b4e0 2024-05-08T21:23:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.794+0000 7f97d67fc700 1 -- 172.21.15.3:0/2920374190 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f97d00075c0 con 0x7f97e015b4e0 2024-05-08T21:23:30.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.794+0000 7f97d67fc700 1 --2- 172.21.15.3:0/2920374190 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f97c406ac90 0x7f97c406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:30.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.794+0000 7f97d67fc700 1 -- 172.21.15.3:0/2920374190 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5810+0+0 (secure 0 0 0) 0x7f97d0088760 con 0x7f97e015b4e0 2024-05-08T21:23:30.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.794+0000 7f97e5d76700 1 --2- 172.21.15.3:0/2920374190 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f97c406ac90 0x7f97c406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:30.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.795+0000 7f97e5d76700 1 --2- 172.21.15.3:0/2920374190 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f97c406ac90 0x7f97c406d150 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0090b0 tx=0x7f97dc008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:30.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:30.799+0000 7f97d67fc700 1 -- 172.21.15.3:0/2920374190 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f97e01464b0 con 0x7f97e015b4e0 2024-05-08T21:23:31.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.015+0000 7f97e7fda700 1 -- 172.21.15.3:0/2920374190 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f97e01464b0 con 0x7f97e015b4e0 2024-05-08T21:23:31.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.016+0000 7f97d67fc700 1 -- 172.21.15.3:0/2920374190 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) v1 ==== 93+0+10 (secure 0 0 0) 0x7f97e01464b0 con 0x7f97e015b4e0 2024-05-08T21:23:31.017 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-05-08T21:23:31.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.018+0000 7f97e7fda700 1 -- 172.21.15.3:0/2920374190 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f97c406ac90 msgr2=0x7f97c406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.018+0000 7f97e7fda700 1 --2- 172.21.15.3:0/2920374190 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f97c406ac90 0x7f97c406d150 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0090b0 tx=0x7f97dc008030 comp rx=0 tx=0).stop 2024-05-08T21:23:31.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.018+0000 7f97e7fda700 1 -- 172.21.15.3:0/2920374190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97e015b4e0 msgr2=0x7f97e00cb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.018+0000 7f97e7fda700 1 --2- 172.21.15.3:0/2920374190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97e015b4e0 0x7f97e00cb120 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f97d000c430 tx=0x7f97d000c7f0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.018+0000 7f97e7fda700 1 -- 172.21.15.3:0/2920374190 shutdown_connections 2024-05-08T21:23:31.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.018+0000 7f97e7fda700 1 --2- 172.21.15.3:0/2920374190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97e015b8c0 0x7f97e0147930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.018+0000 7f97e7fda700 1 --2- 172.21.15.3:0/2920374190 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f97c406ac90 0x7f97c406d150 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.018+0000 7f97e7fda700 1 --2- 172.21.15.3:0/2920374190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97e00d1910 0x7f97e00cabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.018+0000 7f97e7fda700 1 --2- 172.21.15.3:0/2920374190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97e015b4e0 0x7f97e00cb120 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.018+0000 7f97e7fda700 1 -- 172.21.15.3:0/2920374190 >> 172.21.15.3:0/2920374190 conn(0x7f97e008afe0 msgr2=0x7f97e00cd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:31.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.018+0000 7f97e7fda700 1 -- 172.21.15.3:0/2920374190 shutdown_connections 2024-05-08T21:23:31.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.018+0000 7f97e7fda700 1 -- 172.21.15.3:0/2920374190 wait complete. 2024-05-08T21:23:31.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-08T21:23:31.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.170+0000 7f711d90d700 1 Processor -- start 2024-05-08T21:23:31.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.170+0000 7f711d90d700 1 -- start start 2024-05-08T21:23:31.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.171+0000 7f711d90d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71180d0f10 0x7f711815e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.171+0000 7f711d90d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71180cf970 con 0x7f71180cbf50 2024-05-08T21:23:31.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.171+0000 7f711d90d700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71180cfc50 con 0x7f71180af5a0 2024-05-08T21:23:31.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.171+0000 7f711d90d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71180cff30 con 0x7f71180d0f10 2024-05-08T21:23:31.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.172+0000 7f7116ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f71180cbf50 0x7f7118161d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50586/0 (socket says 172.21.15.3:50586) 2024-05-08T21:23:31.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.172+0000 7f7116ffd700 1 -- 172.21.15.3:0/1218378698 learned_addr learned my addr 172.21.15.3:0/1218378698 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:31.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.172+0000 7f71167fc700 1 --2- 172.21.15.3:0/1218378698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71180d0f10 0x7f711815e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.172+0000 7f7115ffb700 1 -- 172.21.15.3:0/1218378698 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40053172 0 0) 0x7f71180cf970 con 0x7f71180cbf50 2024-05-08T21:23:31.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.172+0000 7f7115ffb700 1 -- 172.21.15.3:0/1218378698 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71000035b0 con 0x7f71180cbf50 2024-05-08T21:23:31.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.172+0000 7f7115ffb700 1 -- 172.21.15.3:0/1218378698 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106600803 0 0) 0x7f71180cfc50 con 0x7f71180af5a0 2024-05-08T21:23:31.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.172+0000 7f7115ffb700 1 -- 172.21.15.3:0/1218378698 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71180cf970 con 0x7f71180af5a0 2024-05-08T21:23:31.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.172+0000 7f71167fc700 1 -- 172.21.15.3:0/1218378698 >> v1:172.21.15.186:6789/0 conn(0x7f71180af5a0 legacy=0x7f711806a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.172+0000 7f71167fc700 1 -- 172.21.15.3:0/1218378698 >> v1:172.21.15.3:6789/0 conn(0x7f71180cbf50 legacy=0x7f7118161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.172+0000 7f71167fc700 1 -- 172.21.15.3:0/1218378698 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71181617e0 con 0x7f71180d0f10 2024-05-08T21:23:31.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.172+0000 7f71167fc700 1 --2- 172.21.15.3:0/1218378698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71180d0f10 0x7f711815e050 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f710c009870 tx=0x7f710c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34809a9c7c4b7fe4 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:31.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.173+0000 7f7115ffb700 1 -- 172.21.15.3:0/1218378698 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c011830 con 0x7f71180d0f10 2024-05-08T21:23:31.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.173+0000 7f7115ffb700 1 -- 172.21.15.3:0/1218378698 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f710c011990 con 0x7f71180d0f10 2024-05-08T21:23:31.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.173+0000 7f7115ffb700 1 -- 172.21.15.3:0/1218378698 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c011d20 con 0x7f71180d0f10 2024-05-08T21:23:31.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.173+0000 7f711d90d700 1 -- 172.21.15.3:0/1218378698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71180d0f10 msgr2=0x7f711815e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.173+0000 7f711d90d700 1 --2- 172.21.15.3:0/1218378698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71180d0f10 0x7f711815e050 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f710c009870 tx=0x7f710c009b80 comp rx=0 tx=0).stop 2024-05-08T21:23:31.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.173+0000 7f711d90d700 1 -- 172.21.15.3:0/1218378698 shutdown_connections 2024-05-08T21:23:31.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.173+0000 7f711d90d700 1 --2- 172.21.15.3:0/1218378698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71180d0f10 0x7f711815e050 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.173+0000 7f711d90d700 1 -- 172.21.15.3:0/1218378698 >> 172.21.15.3:0/1218378698 conn(0x7f7118099a50 msgr2=0x7f71180d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:31.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.173+0000 7f711d90d700 1 -- 172.21.15.3:0/1218378698 shutdown_connections 2024-05-08T21:23:31.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.173+0000 7f711d90d700 1 -- 172.21.15.3:0/1218378698 wait complete. 2024-05-08T21:23:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.175+0000 7f711d90d700 1 Processor -- start 2024-05-08T21:23:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.175+0000 7f711d90d700 1 -- start start 2024-05-08T21:23:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.176+0000 7f711d90d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71180af5a0 0x7f71180b3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.176+0000 7f711d90d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71180cbf50 0x7f71181e9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.176+0000 7f711d90d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71180d0f10 0x7f7118159860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.176+0000 7f711d90d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71181e9b80 con 0x7f71180af5a0 2024-05-08T21:23:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.176+0000 7f711d90d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71180b3fc0 con 0x7f71180cbf50 2024-05-08T21:23:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.176+0000 7f711d90d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71180b42a0 con 0x7f71180d0f10 2024-05-08T21:23:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.176+0000 7f7116ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71180af5a0 0x7f71180b3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.176+0000 7f7116ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71180af5a0 0x7f71180b3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42230/0 (socket says 172.21.15.3:42230) 2024-05-08T21:23:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.176+0000 7f7116ffd700 1 -- 172.21.15.3:0/3940602217 learned_addr learned my addr 172.21.15.3:0/3940602217 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.177+0000 7f71167fc700 1 --2- 172.21.15.3:0/3940602217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71180cbf50 0x7f71181e9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.177+0000 7f71177fe700 1 --2- 172.21.15.3:0/3940602217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71180d0f10 0x7f7118159860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.177+0000 7f7116ffd700 1 -- 172.21.15.3:0/3940602217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71180d0f10 msgr2=0x7f7118159860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.177+0000 7f7116ffd700 1 --2- 172.21.15.3:0/3940602217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71180d0f10 0x7f7118159860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.177+0000 7f7116ffd700 1 -- 172.21.15.3:0/3940602217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71180cbf50 msgr2=0x7f71181e9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.177+0000 7f7116ffd700 1 --2- 172.21.15.3:0/3940602217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71180cbf50 0x7f71181e9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.177+0000 7f7116ffd700 1 -- 172.21.15.3:0/3940602217 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710c0094a0 con 0x7f71180af5a0 2024-05-08T21:23:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.177+0000 7f71177fe700 1 --2- 172.21.15.3:0/3940602217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71180d0f10 0x7f7118159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.177+0000 7f7116ffd700 1 --2- 172.21.15.3:0/3940602217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71180af5a0 0x7f71180b3900 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f710800c4d0 tx=0x7f710800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:31.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.177+0000 7f70fffff700 1 -- 172.21.15.3:0/3940602217 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7108005d70 con 0x7f71180af5a0 2024-05-08T21:23:31.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.177+0000 7f70fffff700 1 -- 172.21.15.3:0/3940602217 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71080070d0 con 0x7f71180af5a0 2024-05-08T21:23:31.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.177+0000 7f711d90d700 1 -- 172.21.15.3:0/3940602217 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71180b4580 con 0x7f71180af5a0 2024-05-08T21:23:31.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.177+0000 7f70fffff700 1 -- 172.21.15.3:0/3940602217 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7108007480 con 0x7f71180af5a0 2024-05-08T21:23:31.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.177+0000 7f711d90d700 1 -- 172.21.15.3:0/3940602217 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71180b0a10 con 0x7f71180af5a0 2024-05-08T21:23:31.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.178+0000 7f711d90d700 1 -- 172.21.15.3:0/3940602217 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71180b0870 con 0x7f71180af5a0 2024-05-08T21:23:31.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.179+0000 7f70fffff700 1 -- 172.21.15.3:0/3940602217 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f71080075e0 con 0x7f71180af5a0 2024-05-08T21:23:31.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.179+0000 7f70fffff700 1 --2- 172.21.15.3:0/3940602217 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f710406ac90 0x7f710406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.179+0000 7f70fffff700 1 -- 172.21.15.3:0/3940602217 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5810+0+0 (secure 0 0 0) 0x7f7108088960 con 0x7f71180af5a0 2024-05-08T21:23:31.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.179+0000 7f71167fc700 1 --2- 172.21.15.3:0/3940602217 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f710406ac90 0x7f710406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.180+0000 7f71167fc700 1 --2- 172.21.15.3:0/3940602217 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f710406ac90 0x7f710406d150 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f710c0097c0 tx=0x7f710c01c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:31.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.184+0000 7f70fffff700 1 -- 172.21.15.3:0/3940602217 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f71180b0870 con 0x7f71180af5a0 2024-05-08T21:23:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.399+0000 7f711d90d700 1 -- 172.21.15.3:0/3940602217 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f71180b0870 con 0x7f71180af5a0 2024-05-08T21:23:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.400+0000 7f70fffff700 1 -- 172.21.15.3:0/3940602217 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) v1 ==== 100+0+11 (secure 0 0 0) 0x7f71180b0870 con 0x7f71180af5a0 2024-05-08T21:23:31.401 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-05-08T21:23:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.402+0000 7f711d90d700 1 -- 172.21.15.3:0/3940602217 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f710406ac90 msgr2=0x7f710406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.402+0000 7f711d90d700 1 --2- 172.21.15.3:0/3940602217 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f710406ac90 0x7f710406d150 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f710c0097c0 tx=0x7f710c01c040 comp rx=0 tx=0).stop 2024-05-08T21:23:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.402+0000 7f711d90d700 1 -- 172.21.15.3:0/3940602217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71180af5a0 msgr2=0x7f71180b3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.402+0000 7f711d90d700 1 --2- 172.21.15.3:0/3940602217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71180af5a0 0x7f71180b3900 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f710800c4d0 tx=0x7f710800c890 comp rx=0 tx=0).stop 2024-05-08T21:23:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.402+0000 7f711d90d700 1 -- 172.21.15.3:0/3940602217 shutdown_connections 2024-05-08T21:23:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.402+0000 7f711d90d700 1 --2- 172.21.15.3:0/3940602217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71180d0f10 0x7f7118159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.402+0000 7f711d90d700 1 --2- 172.21.15.3:0/3940602217 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f710406ac90 0x7f710406d150 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.402+0000 7f711d90d700 1 --2- 172.21.15.3:0/3940602217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71180af5a0 0x7f71180b3900 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.402+0000 7f711d90d700 1 --2- 172.21.15.3:0/3940602217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71180cbf50 0x7f71181e9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.402+0000 7f711d90d700 1 -- 172.21.15.3:0/3940602217 >> 172.21.15.3:0/3940602217 conn(0x7f7118099a50 msgr2=0x7f711815bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.402+0000 7f711d90d700 1 -- 172.21.15.3:0/3940602217 shutdown_connections 2024-05-08T21:23:31.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.402+0000 7f711d90d700 1 -- 172.21.15.3:0/3940602217 wait complete. 2024-05-08T21:23:31.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get recovery_fs_metadata pg_num 2024-05-08T21:23:31.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.555+0000 7f10199af700 1 Processor -- start 2024-05-08T21:23:31.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.556+0000 7f10199af700 1 -- start start 2024-05-08T21:23:31.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.556+0000 7f10199af700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10140cff30 0x7f10140d0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.556+0000 7f10199af700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f10140b4a90 0x7f10140ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.556+0000 7f10199af700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10140d0850 con 0x7f10140d3d20 2024-05-08T21:23:31.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.556+0000 7f10199af700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10140cf610 con 0x7f10140b4a90 2024-05-08T21:23:31.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.556+0000 7f10199af700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10140d1fc0 con 0x7f10140cff30 2024-05-08T21:23:31.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.557+0000 7f10127fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f10140d3d20 0x7f10140ce260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:36350/0 (socket says 172.21.15.3:36350) 2024-05-08T21:23:31.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.557+0000 7f10127fc700 1 -- 172.21.15.3:0/3271744938 learned_addr learned my addr 172.21.15.3:0/3271744938 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:31.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.557+0000 7f10137fe700 1 --2- 172.21.15.3:0/3271744938 >> v2:172.21.15.186:3300/0 conn(0x7f10140b4a90 0x7f10140ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.557+0000 7f1012ffd700 1 --2- 172.21.15.3:0/3271744938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10140cff30 0x7f10140d0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.557+0000 7f1011ffb700 1 -- 172.21.15.3:0/3271744938 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619076906 0 0) 0x7f10140d0850 con 0x7f10140d3d20 2024-05-08T21:23:31.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.557+0000 7f1011ffb700 1 -- 172.21.15.3:0/3271744938 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ffc0035b0 con 0x7f10140d3d20 2024-05-08T21:23:31.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.557+0000 7f10137fe700 1 -- 172.21.15.3:0/3271744938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10140cff30 msgr2=0x7f10140d0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:31.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.557+0000 7f10137fe700 1 --2- 172.21.15.3:0/3271744938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10140cff30 0x7f10140d0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.557+0000 7f1011ffb700 1 -- 172.21.15.3:0/3271744938 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3365421158 0 0) 0x7f0ffc0035b0 con 0x7f10140d3d20 2024-05-08T21:23:31.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.557+0000 7f10137fe700 1 -- 172.21.15.3:0/3271744938 >> v1:172.21.15.3:6789/0 conn(0x7f10140d3d20 legacy=0x7f10140ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.557+0000 7f10137fe700 1 -- 172.21.15.3:0/3271744938 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10140d22a0 con 0x7f10140b4a90 2024-05-08T21:23:31.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.557+0000 7f10137fe700 1 --2- 172.21.15.3:0/3271744938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10140b4a90 0x7f10140ce970 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f100c009840 tx=0x7f100c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9f0d4d43c89b2f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:31.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.558+0000 7f1011ffb700 1 -- 172.21.15.3:0/3271744938 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008001670 con 0x7f10140b4a90 2024-05-08T21:23:31.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.558+0000 7f1011ffb700 1 -- 172.21.15.3:0/3271744938 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100c004030 con 0x7f10140b4a90 2024-05-08T21:23:31.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.558+0000 7f1011ffb700 1 -- 172.21.15.3:0/3271744938 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c0043c0 con 0x7f10140b4a90 2024-05-08T21:23:31.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.558+0000 7f10199af700 1 -- 172.21.15.3:0/3271744938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10140b4a90 msgr2=0x7f10140ce970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.558+0000 7f10199af700 1 --2- 172.21.15.3:0/3271744938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10140b4a90 0x7f10140ce970 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f100c009840 tx=0x7f100c009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:31.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.558+0000 7f10199af700 1 -- 172.21.15.3:0/3271744938 shutdown_connections 2024-05-08T21:23:31.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.558+0000 7f10199af700 1 --2- 172.21.15.3:0/3271744938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10140b4a90 0x7f10140ce970 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.558+0000 7f10199af700 1 --2- 172.21.15.3:0/3271744938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10140cff30 0x7f10140d0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.558+0000 7f10199af700 1 -- 172.21.15.3:0/3271744938 >> 172.21.15.3:0/3271744938 conn(0x7f101408afe0 msgr2=0x7f10140d3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:31.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.558+0000 7f10199af700 1 -- 172.21.15.3:0/3271744938 shutdown_connections 2024-05-08T21:23:31.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.558+0000 7f10199af700 1 -- 172.21.15.3:0/3271744938 wait complete. 2024-05-08T21:23:31.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.560+0000 7f10199af700 1 Processor -- start 2024-05-08T21:23:31.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.560+0000 7f10199af700 1 -- start start 2024-05-08T21:23:31.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f10199af700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10140b4a90 0x7f1014156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f10199af700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10140cff30 0x7f101415a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f10199af700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10140d3d20 0x7f101415b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f10199af700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10140c83f0 con 0x7f10140b4a90 2024-05-08T21:23:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f10199af700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1014157090 con 0x7f10140d3d20 2024-05-08T21:23:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f10199af700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10140b6d00 con 0x7f10140cff30 2024-05-08T21:23:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f1012ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10140b4a90 0x7f1014156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f1012ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10140b4a90 0x7f1014156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35010/0 (socket says 172.21.15.3:35010) 2024-05-08T21:23:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f1012ffd700 1 -- 172.21.15.3:0/1039010745 learned_addr learned my addr 172.21.15.3:0/1039010745 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f10127fc700 1 --2- 172.21.15.3:0/1039010745 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10140cff30 0x7f101415a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f10137fe700 1 --2- 172.21.15.3:0/1039010745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10140d3d20 0x7f101415b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f1012ffd700 1 -- 172.21.15.3:0/1039010745 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10140cff30 msgr2=0x7f101415a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f1012ffd700 1 --2- 172.21.15.3:0/1039010745 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10140cff30 0x7f101415a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f1012ffd700 1 -- 172.21.15.3:0/1039010745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10140d3d20 msgr2=0x7f101415b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f1012ffd700 1 --2- 172.21.15.3:0/1039010745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10140d3d20 0x7f101415b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f1012ffd700 1 -- 172.21.15.3:0/1039010745 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100c0094a0 con 0x7f10140b4a90 2024-05-08T21:23:31.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.561+0000 7f1012ffd700 1 --2- 172.21.15.3:0/1039010745 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10140b4a90 0x7f1014156b50 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f100400efc0 tx=0x7f100400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:31.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.562+0000 7f0ffbfff700 1 -- 172.21.15.3:0/1039010745 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100400cdc0 con 0x7f10140b4a90 2024-05-08T21:23:31.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.562+0000 7f0ffbfff700 1 -- 172.21.15.3:0/1039010745 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100400f040 con 0x7f10140b4a90 2024-05-08T21:23:31.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.562+0000 7f10199af700 1 -- 172.21.15.3:0/1039010745 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10140b6fe0 con 0x7f10140b4a90 2024-05-08T21:23:31.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.562+0000 7f0ffbfff700 1 -- 172.21.15.3:0/1039010745 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10040146a0 con 0x7f10140b4a90 2024-05-08T21:23:31.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.562+0000 7f10199af700 1 -- 172.21.15.3:0/1039010745 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10140c8010 con 0x7f10140b4a90 2024-05-08T21:23:31.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.563+0000 7f10199af700 1 -- 172.21.15.3:0/1039010745 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10140c8660 con 0x7f10140b4a90 2024-05-08T21:23:31.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.563+0000 7f0ffbfff700 1 -- 172.21.15.3:0/1039010745 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1004014800 con 0x7f10140b4a90 2024-05-08T21:23:31.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.564+0000 7f0ffbfff700 1 --2- 172.21.15.3:0/1039010745 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f100006ac90 0x7f100006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.564+0000 7f0ffbfff700 1 -- 172.21.15.3:0/1039010745 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5810+0+0 (secure 0 0 0) 0x7f1004089fe0 con 0x7f10140b4a90 2024-05-08T21:23:31.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.564+0000 7f10127fc700 1 --2- 172.21.15.3:0/1039010745 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f100006ac90 0x7f100006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.565+0000 7f10127fc700 1 --2- 172.21.15.3:0/1039010745 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f100006ac90 0x7f100006d150 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1008001eb0 tx=0x7f1008001ce0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:31.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.568+0000 7f0ffbfff700 1 -- 172.21.15.3:0/1039010745 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f10040551e0 con 0x7f10140b4a90 2024-05-08T21:23:31.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.786+0000 7f10199af700 1 -- 172.21.15.3:0/1039010745 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f10140d4160 con 0x7f10140b4a90 2024-05-08T21:23:31.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.787+0000 7f0ffbfff700 1 -- 172.21.15.3:0/1039010745 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"}]=0 v43) v1 ==== 109+0+11 (secure 0 0 0) 0x7f1004058800 con 0x7f10140b4a90 2024-05-08T21:23:31.788 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-05-08T21:23:31.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.789+0000 7f10199af700 1 -- 172.21.15.3:0/1039010745 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f100006ac90 msgr2=0x7f100006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.789+0000 7f10199af700 1 --2- 172.21.15.3:0/1039010745 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f100006ac90 0x7f100006d150 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1008001eb0 tx=0x7f1008001ce0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.789+0000 7f10199af700 1 -- 172.21.15.3:0/1039010745 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10140b4a90 msgr2=0x7f1014156b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.789+0000 7f10199af700 1 --2- 172.21.15.3:0/1039010745 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10140b4a90 0x7f1014156b50 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f100400efc0 tx=0x7f100400c3a0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.789+0000 7f10199af700 1 -- 172.21.15.3:0/1039010745 shutdown_connections 2024-05-08T21:23:31.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.789+0000 7f10199af700 1 --2- 172.21.15.3:0/1039010745 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f100006ac90 0x7f100006d150 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.789+0000 7f10199af700 1 --2- 172.21.15.3:0/1039010745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10140d3d20 0x7f101415b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.789+0000 7f10199af700 1 --2- 172.21.15.3:0/1039010745 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10140b4a90 0x7f1014156b50 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.789+0000 7f10199af700 1 --2- 172.21.15.3:0/1039010745 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10140cff30 0x7f101415a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.789+0000 7f10199af700 1 -- 172.21.15.3:0/1039010745 >> 172.21.15.3:0/1039010745 conn(0x7f101408afe0 msgr2=0x7f101415ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:31.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.789+0000 7f10199af700 1 -- 172.21.15.3:0/1039010745 shutdown_connections 2024-05-08T21:23:31.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.789+0000 7f10199af700 1 -- 172.21.15.3:0/1039010745 wait complete. 2024-05-08T21:23:31.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:23:31.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.939+0000 7f020517e700 1 Processor -- start 2024-05-08T21:23:31.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.939+0000 7f020517e700 1 -- start start 2024-05-08T21:23:31.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.940+0000 7f020517e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f020006bd80 0x7f02000d0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.940+0000 7f020517e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f020006c3d0 0x7f020006c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.940+0000 7f020517e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02000cf990 con 0x7f020015da10 2024-05-08T21:23:31.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.940+0000 7f020517e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f020009ab40 con 0x7f020006bd80 2024-05-08T21:23:31.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.940+0000 7f020517e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020006cda0 con 0x7f020006c3d0 2024-05-08T21:23:31.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.940+0000 7f01fe59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f020006bd80 0x7f02000d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.940+0000 7f01fe59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f020006bd80 0x7f02000d0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35012/0 (socket says 172.21.15.3:35012) 2024-05-08T21:23:31.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.940+0000 7f01fe59c700 1 -- 172.21.15.3:0/276598082 learned_addr learned my addr 172.21.15.3:0/276598082 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:31.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.940+0000 7f01ff59e700 1 --2- 172.21.15.3:0/276598082 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f020006c3d0 0x7f020006c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.940+0000 7f01fe59c700 1 -- 172.21.15.3:0/276598082 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f020006c3d0 msgr2=0x7f020006c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:31.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.940+0000 7f01fe59c700 1 --2- 172.21.15.3:0/276598082 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f020006c3d0 0x7f020006c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.940+0000 7f01fe59c700 1 -- 172.21.15.3:0/276598082 >> v1:172.21.15.186:6789/0 conn(0x7f020015da10 legacy=0x7f020009a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:31.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.941+0000 7f01fe59c700 1 -- 172.21.15.3:0/276598082 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02000d04b0 con 0x7f020006bd80 2024-05-08T21:23:31.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.941+0000 7f01ff59e700 1 --2- 172.21.15.3:0/276598082 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f020006c3d0 0x7f020006c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:31.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.941+0000 7f01fe59c700 1 --2- 172.21.15.3:0/276598082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020006bd80 0x7f02000d0940 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009920 tx=0x7f01f4009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d7ff536e18d5da server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:31.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.941+0000 7f01fdd9b700 1 -- 172.21.15.3:0/276598082 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f4004230 con 0x7f020006bd80 2024-05-08T21:23:31.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.941+0000 7f01fdd9b700 1 -- 172.21.15.3:0/276598082 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f4004390 con 0x7f020006bd80 2024-05-08T21:23:31.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.941+0000 7f01fdd9b700 1 -- 172.21.15.3:0/276598082 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f4009e70 con 0x7f020006bd80 2024-05-08T21:23:31.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.941+0000 7f020517e700 1 -- 172.21.15.3:0/276598082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020006bd80 msgr2=0x7f02000d0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.941+0000 7f020517e700 1 --2- 172.21.15.3:0/276598082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020006bd80 0x7f02000d0940 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009920 tx=0x7f01f4009c30 comp rx=0 tx=0).stop 2024-05-08T21:23:31.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.941+0000 7f020517e700 1 -- 172.21.15.3:0/276598082 shutdown_connections 2024-05-08T21:23:31.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.941+0000 7f020517e700 1 --2- 172.21.15.3:0/276598082 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f020006c3d0 0x7f020006c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.941+0000 7f020517e700 1 --2- 172.21.15.3:0/276598082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020006bd80 0x7f02000d0940 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.941+0000 7f020517e700 1 -- 172.21.15.3:0/276598082 >> 172.21.15.3:0/276598082 conn(0x7f02000b3010 msgr2=0x7f02000b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:31.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.942+0000 7f020517e700 1 -- 172.21.15.3:0/276598082 shutdown_connections 2024-05-08T21:23:31.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.942+0000 7f020517e700 1 -- 172.21.15.3:0/276598082 wait complete. 2024-05-08T21:23:31.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.943+0000 7f020517e700 1 Processor -- start 2024-05-08T21:23:31.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.943+0000 7f020517e700 1 -- start start 2024-05-08T21:23:31.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.944+0000 7f020517e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f020006bd80 0x7f020009ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.944+0000 7f020517e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020006c3d0 0x7f02000c7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.944+0000 7f020517e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f020015da10 0x7f02000c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.944+0000 7f020517e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02000b57f0 con 0x7f020006c3d0 2024-05-08T21:23:31.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.944+0000 7f020517e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02000b4ee0 con 0x7f020006bd80 2024-05-08T21:23:31.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.944+0000 7f020517e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02000c7e80 con 0x7f020015da10 2024-05-08T21:23:31.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.944+0000 7f01fe59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020006c3d0 0x7f02000c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.944+0000 7f01fe59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020006c3d0 0x7f02000c7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35028/0 (socket says 172.21.15.3:35028) 2024-05-08T21:23:31.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.944+0000 7f01fe59c700 1 -- 172.21.15.3:0/145363863 learned_addr learned my addr 172.21.15.3:0/145363863 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:31.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.944+0000 7f01fed9d700 1 --2- 172.21.15.3:0/145363863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f020006bd80 0x7f020009ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.944+0000 7f01ff59e700 1 --2- 172.21.15.3:0/145363863 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f020015da10 0x7f02000c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.945+0000 7f01fe59c700 1 -- 172.21.15.3:0/145363863 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f020015da10 msgr2=0x7f02000c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.945+0000 7f01fe59c700 1 --2- 172.21.15.3:0/145363863 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f020015da10 0x7f02000c7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.945+0000 7f01fe59c700 1 -- 172.21.15.3:0/145363863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f020006bd80 msgr2=0x7f020009ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:31.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.945+0000 7f01fe59c700 1 --2- 172.21.15.3:0/145363863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f020006bd80 0x7f020009ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:31.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.945+0000 7f01fe59c700 1 -- 172.21.15.3:0/145363863 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f4009550 con 0x7f020006c3d0 2024-05-08T21:23:31.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.945+0000 7f01fe59c700 1 --2- 172.21.15.3:0/145363863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020006c3d0 0x7f02000c7400 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009130 tx=0x7f01f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:31.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.945+0000 7f01e77fe700 1 -- 172.21.15.3:0/145363863 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f400be30 con 0x7f020006c3d0 2024-05-08T21:23:31.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.945+0000 7f01e77fe700 1 -- 172.21.15.3:0/145363863 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f40036a0 con 0x7f020006c3d0 2024-05-08T21:23:31.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.945+0000 7f020517e700 1 -- 172.21.15.3:0/145363863 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02000cb1b0 con 0x7f020006c3d0 2024-05-08T21:23:31.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.945+0000 7f01e77fe700 1 -- 172.21.15.3:0/145363863 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f4003a30 con 0x7f020006c3d0 2024-05-08T21:23:31.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.945+0000 7f020517e700 1 -- 172.21.15.3:0/145363863 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02000b5410 con 0x7f020006c3d0 2024-05-08T21:23:31.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.946+0000 7f020517e700 1 -- 172.21.15.3:0/145363863 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02000b55a0 con 0x7f020006c3d0 2024-05-08T21:23:31.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.947+0000 7f01e77fe700 1 -- 172.21.15.3:0/145363863 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f01f4003b90 con 0x7f020006c3d0 2024-05-08T21:23:31.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.947+0000 7f01e77fe700 1 --2- 172.21.15.3:0/145363863 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f01ec06ac90 0x7f01ec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:31.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.947+0000 7f01e77fe700 1 -- 172.21.15.3:0/145363863 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5810+0+0 (secure 0 0 0) 0x7f01f408ab90 con 0x7f020006c3d0 2024-05-08T21:23:31.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.948+0000 7f01fed9d700 1 --2- 172.21.15.3:0/145363863 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f01ec06ac90 0x7f01ec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:31.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.948+0000 7f01fed9d700 1 --2- 172.21.15.3:0/145363863 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f01ec06ac90 0x7f01ec06d150 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f01f0007fd0 tx=0x7f01f0007ec0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:31.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:31.952+0000 7f01e77fe700 1 -- 172.21.15.3:0/145363863 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f01f4055d90 con 0x7f020006c3d0 2024-05-08T21:23:32.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.220+0000 7f020517e700 1 -- 172.21.15.3:0/145363863 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0200068000 con 0x7f020006c3d0 2024-05-08T21:23:32.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.220+0000 7f01e77fe700 1 -- 172.21.15.3:0/145363863 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f01f40593b0 con 0x7f020006c3d0 2024-05-08T21:23:32.222 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:32.222 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":28,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6833/2104267343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2104267343},{"type":"v1","addr":"172.21.15.3:6833","nonce":2104267343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/208321708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":208321708},{"type":"v1","addr":"172.21.15.3:6835","nonce":208321708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/1591386131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1591386131},{"type":"v1","addr":"172.21.15.186:6836","nonce":1591386131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:23:05.475355+0000","modified":"2024-05-08T21:23:15.662101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6837/2000072997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":2000072997},{"type":"v1","addr":"172.21.15.186:6837","nonce":2000072997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":6,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-08T21:23:32.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.223+0000 7f020517e700 1 -- 172.21.15.3:0/145363863 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f01ec06ac90 msgr2=0x7f01ec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:32.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.223+0000 7f020517e700 1 --2- 172.21.15.3:0/145363863 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f01ec06ac90 0x7f01ec06d150 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f01f0007fd0 tx=0x7f01f0007ec0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.223+0000 7f020517e700 1 -- 172.21.15.3:0/145363863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020006c3d0 msgr2=0x7f02000c7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:32.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.223+0000 7f020517e700 1 --2- 172.21.15.3:0/145363863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020006c3d0 0x7f02000c7400 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009130 tx=0x7f01f4004750 comp rx=0 tx=0).stop 2024-05-08T21:23:32.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.223+0000 7f020517e700 1 -- 172.21.15.3:0/145363863 shutdown_connections 2024-05-08T21:23:32.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.223+0000 7f020517e700 1 --2- 172.21.15.3:0/145363863 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f020015da10 0x7f02000c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.223+0000 7f020517e700 1 --2- 172.21.15.3:0/145363863 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f01ec06ac90 0x7f01ec06d150 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.223+0000 7f020517e700 1 --2- 172.21.15.3:0/145363863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f020006bd80 0x7f020009ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.223+0000 7f020517e700 1 --2- 172.21.15.3:0/145363863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020006c3d0 0x7f02000c7400 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.223+0000 7f020517e700 1 -- 172.21.15.3:0/145363863 >> 172.21.15.3:0/145363863 conn(0x7f02000b3010 msgr2=0x7f02000cc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:32.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.223+0000 7f020517e700 1 -- 172.21.15.3:0/145363863 shutdown_connections 2024-05-08T21:23:32.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.223+0000 7f020517e700 1 -- 172.21.15.3:0/145363863 wait complete. 2024-05-08T21:23:32.226 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 28 2024-05-08T21:23:32.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:23:32.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.372+0000 7fb82f442700 1 Processor -- start 2024-05-08T21:23:32.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.373+0000 7fb82f442700 1 -- start start 2024-05-08T21:23:32.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.373+0000 7fb82f442700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb8280b2480 0x7fb82809a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:32.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.373+0000 7fb82f442700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8280d0840 con 0x7fb828161030 2024-05-08T21:23:32.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.373+0000 7fb82f442700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82806c600 con 0x7fb8280cbbc0 2024-05-08T21:23:32.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.373+0000 7fb82f442700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82806ccc0 con 0x7fb8280b2480 2024-05-08T21:23:32.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.374+0000 7fb82d1de700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb828161030 0x7fb828099a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:36366/0 (socket says 172.21.15.3:36366) 2024-05-08T21:23:32.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.374+0000 7fb82d1de700 1 -- 172.21.15.3:0/2742428645 learned_addr learned my addr 172.21.15.3:0/2742428645 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:32.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.374+0000 7fb82c9dd700 1 --2- 172.21.15.3:0/2742428645 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb8280b2480 0x7fb82809a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:32.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.374+0000 7fb81ffff700 1 -- 172.21.15.3:0/2742428645 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424112068 0 0) 0x7fb8280d0840 con 0x7fb828161030 2024-05-08T21:23:32.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.374+0000 7fb81ffff700 1 -- 172.21.15.3:0/2742428645 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8140035b0 con 0x7fb828161030 2024-05-08T21:23:32.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.374+0000 7fb81ffff700 1 -- 172.21.15.3:0/2742428645 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301144435 0 0) 0x7fb82806c600 con 0x7fb8280cbbc0 2024-05-08T21:23:32.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.374+0000 7fb81ffff700 1 -- 172.21.15.3:0/2742428645 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8280d0840 con 0x7fb8280cbbc0 2024-05-08T21:23:32.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.374+0000 7fb82c9dd700 1 -- 172.21.15.3:0/2742428645 >> v1:172.21.15.186:6789/0 conn(0x7fb8280cbbc0 legacy=0x7fb8280cc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:32.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.374+0000 7fb82c9dd700 1 -- 172.21.15.3:0/2742428645 >> v1:172.21.15.3:6789/0 conn(0x7fb828161030 legacy=0x7fb828099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:32.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.374+0000 7fb82c9dd700 1 -- 172.21.15.3:0/2742428645 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82806c8e0 con 0x7fb8280b2480 2024-05-08T21:23:32.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.374+0000 7fb82c9dd700 1 --2- 172.21.15.3:0/2742428645 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb8280b2480 0x7fb82809a160 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb818009870 tx=0x7fb818009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5c4e04189ae883 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:32.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.375+0000 7fb81ffff700 1 -- 172.21.15.3:0/2742428645 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb818011830 con 0x7fb8280b2480 2024-05-08T21:23:32.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.375+0000 7fb81ffff700 1 -- 172.21.15.3:0/2742428645 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb818011990 con 0x7fb8280b2480 2024-05-08T21:23:32.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.375+0000 7fb81ffff700 1 -- 172.21.15.3:0/2742428645 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb818011d20 con 0x7fb8280b2480 2024-05-08T21:23:32.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.375+0000 7fb82f442700 1 -- 172.21.15.3:0/2742428645 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb8280b2480 msgr2=0x7fb82809a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:32.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.375+0000 7fb82f442700 1 --2- 172.21.15.3:0/2742428645 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb8280b2480 0x7fb82809a160 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb818009870 tx=0x7fb818009b80 comp rx=0 tx=0).stop 2024-05-08T21:23:32.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.375+0000 7fb82f442700 1 -- 172.21.15.3:0/2742428645 shutdown_connections 2024-05-08T21:23:32.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.375+0000 7fb82f442700 1 --2- 172.21.15.3:0/2742428645 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb8280b2480 0x7fb82809a160 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.375+0000 7fb82f442700 1 -- 172.21.15.3:0/2742428645 >> 172.21.15.3:0/2742428645 conn(0x7fb8280b2f20 msgr2=0x7fb8280b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:32.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.375+0000 7fb82f442700 1 -- 172.21.15.3:0/2742428645 shutdown_connections 2024-05-08T21:23:32.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.375+0000 7fb82f442700 1 -- 172.21.15.3:0/2742428645 wait complete. 2024-05-08T21:23:32.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.377+0000 7fb82f442700 1 Processor -- start 2024-05-08T21:23:32.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.377+0000 7fb82f442700 1 -- start start 2024-05-08T21:23:32.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.377+0000 7fb82f442700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8280b2480 0x7fb8280b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:32.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.377+0000 7fb82f442700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb8280cbbc0 0x7fb8280b6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:32.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb82f442700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb828161030 0x7fb828153b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:32.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb82f442700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8280d0660 con 0x7fb8280b2480 2024-05-08T21:23:32.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb82f442700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8280b6cb0 con 0x7fb828161030 2024-05-08T21:23:32.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb82f442700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8280c5580 con 0x7fb8280cbbc0 2024-05-08T21:23:32.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb82d1de700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8280b2480 0x7fb8280b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:32.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb82d1de700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8280b2480 0x7fb8280b3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35030/0 (socket says 172.21.15.3:35030) 2024-05-08T21:23:32.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb82d1de700 1 -- 172.21.15.3:0/1999552247 learned_addr learned my addr 172.21.15.3:0/1999552247 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:32.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb82c9dd700 1 --2- 172.21.15.3:0/1999552247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb8280cbbc0 0x7fb8280b6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:32.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb82d9df700 1 --2- 172.21.15.3:0/1999552247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb828161030 0x7fb828153b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:32.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb82d1de700 1 -- 172.21.15.3:0/1999552247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb8280cbbc0 msgr2=0x7fb8280b6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:32.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb82d1de700 1 --2- 172.21.15.3:0/1999552247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb8280cbbc0 0x7fb8280b6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb82d1de700 1 -- 172.21.15.3:0/1999552247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb828161030 msgr2=0x7fb828153b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:32.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb82d1de700 1 --2- 172.21.15.3:0/1999552247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb828161030 0x7fb828153b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb82d1de700 1 -- 172.21.15.3:0/1999552247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8180094a0 con 0x7fb8280b2480 2024-05-08T21:23:32.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb82d1de700 1 --2- 172.21.15.3:0/1999552247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8280b2480 0x7fb8280b3e00 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb82400c4d0 tx=0x7fb82400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:32.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb81dffb700 1 -- 172.21.15.3:0/1999552247 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82400cea0 con 0x7fb8280b2480 2024-05-08T21:23:32.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.378+0000 7fb81dffb700 1 -- 172.21.15.3:0/1999552247 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb824005d70 con 0x7fb8280b2480 2024-05-08T21:23:32.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.379+0000 7fb82f442700 1 -- 172.21.15.3:0/1999552247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8280cab30 con 0x7fb8280b2480 2024-05-08T21:23:32.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.379+0000 7fb81dffb700 1 -- 172.21.15.3:0/1999552247 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8240072f0 con 0x7fb8280b2480 2024-05-08T21:23:32.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.379+0000 7fb82f442700 1 -- 172.21.15.3:0/1999552247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8280c4f30 con 0x7fb8280b2480 2024-05-08T21:23:32.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.380+0000 7fb82f442700 1 -- 172.21.15.3:0/1999552247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb828068000 con 0x7fb8280b2480 2024-05-08T21:23:32.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.380+0000 7fb81dffb700 1 -- 172.21.15.3:0/1999552247 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb824007450 con 0x7fb8280b2480 2024-05-08T21:23:32.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.381+0000 7fb81dffb700 1 --2- 172.21.15.3:0/1999552247 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb80c06ac90 0x7fb80c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:32.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.381+0000 7fb81dffb700 1 -- 172.21.15.3:0/1999552247 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5810+0+0 (secure 0 0 0) 0x7fb8240886f0 con 0x7fb8280b2480 2024-05-08T21:23:32.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.381+0000 7fb82c9dd700 1 --2- 172.21.15.3:0/1999552247 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb80c06ac90 0x7fb80c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:32.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.382+0000 7fb82c9dd700 1 --2- 172.21.15.3:0/1999552247 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb80c06ac90 0x7fb80c06d150 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb8180097c0 tx=0x7fb81801c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:32.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.386+0000 7fb81dffb700 1 -- 172.21.15.3:0/1999552247 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb8240538f0 con 0x7fb8280b2480 2024-05-08T21:23:32.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.606+0000 7fb82f442700 1 -- 172.21.15.3:0/1999552247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8280d0f50 con 0x7fb8280b2480 2024-05-08T21:23:32.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.607+0000 7fb81dffb700 1 -- 172.21.15.3:0/1999552247 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+16195 (secure 0 0 0) 0x7fb824056f10 con 0x7fb8280b2480 2024-05-08T21:23:32.609 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:32.609 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":43,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:23:29.839993+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-08T21:22:49.153759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"recovery_fs_metadata","create_time":"2024-05-08T21:23:03.921695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6834/3599481370":"2024-05-09T21:23:01.563392+0000","172.21.15.3:6832/3599481370":"2024-05-09T21:23:01.563392+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:23:32.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.610+0000 7fb82f442700 1 -- 172.21.15.3:0/1999552247 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb80c06ac90 msgr2=0x7fb80c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:32.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.610+0000 7fb82f442700 1 --2- 172.21.15.3:0/1999552247 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb80c06ac90 0x7fb80c06d150 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb8180097c0 tx=0x7fb81801c040 comp rx=0 tx=0).stop 2024-05-08T21:23:32.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.610+0000 7fb82f442700 1 -- 172.21.15.3:0/1999552247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8280b2480 msgr2=0x7fb8280b3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:32.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.610+0000 7fb82f442700 1 --2- 172.21.15.3:0/1999552247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8280b2480 0x7fb8280b3e00 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb82400c4d0 tx=0x7fb82400c7e0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.610+0000 7fb82f442700 1 -- 172.21.15.3:0/1999552247 shutdown_connections 2024-05-08T21:23:32.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.610+0000 7fb82f442700 1 --2- 172.21.15.3:0/1999552247 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb80c06ac90 0x7fb80c06d150 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.610+0000 7fb82f442700 1 --2- 172.21.15.3:0/1999552247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb828161030 0x7fb828153b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.610+0000 7fb82f442700 1 --2- 172.21.15.3:0/1999552247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8280b2480 0x7fb8280b3e00 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.610+0000 7fb82f442700 1 --2- 172.21.15.3:0/1999552247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb8280cbbc0 0x7fb8280b6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.610+0000 7fb82f442700 1 -- 172.21.15.3:0/1999552247 >> 172.21.15.3:0/1999552247 conn(0x7fb8280b2f20 msgr2=0x7fb82806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:32.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.610+0000 7fb82f442700 1 -- 172.21.15.3:0/1999552247 shutdown_connections 2024-05-08T21:23:32.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.610+0000 7fb82f442700 1 -- 172.21.15.3:0/1999552247 wait complete. 2024-05-08T21:23:32.626 INFO:tasks.cephfs.filesystem:Destroying file system recovery_fs and related pools 2024-05-08T21:23:32.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:23:32.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.761+0000 7ff111a70700 1 Processor -- start 2024-05-08T21:23:32.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.761+0000 7ff111a70700 1 -- start start 2024-05-08T21:23:32.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.762+0000 7ff111a70700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff10c0d13e0 0x7ff10c0d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:32.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.762+0000 7ff111a70700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff10c15db60 0x7ff10c0d2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:32.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.762+0000 7ff111a70700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff10c0b4430 con 0x7ff10c15d780 2024-05-08T21:23:32.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.762+0000 7ff111a70700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff10c0d45c0 con 0x7ff10c15db60 2024-05-08T21:23:32.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.762+0000 7ff111a70700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c0b3820 con 0x7ff10c0d13e0 2024-05-08T21:23:32.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.762+0000 7ff10b7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff10c15d780 0x7ff10c0b48c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:36378/0 (socket says 172.21.15.3:36378) 2024-05-08T21:23:32.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.762+0000 7ff10b7fe700 1 -- 172.21.15.3:0/2537290630 learned_addr learned my addr 172.21.15.3:0/2537290630 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:32.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.763+0000 7ff10bfff700 1 --2- 172.21.15.3:0/2537290630 >> v2:172.21.15.186:3300/0 conn(0x7ff10c15db60 0x7ff10c0d2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:32.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.763+0000 7ff10affd700 1 --2- 172.21.15.3:0/2537290630 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff10c0d13e0 0x7ff10c0d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:32.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.763+0000 7ff10a7fc700 1 -- 172.21.15.3:0/2537290630 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390149646 0 0) 0x7ff10c0b4430 con 0x7ff10c15d780 2024-05-08T21:23:32.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.763+0000 7ff10a7fc700 1 -- 172.21.15.3:0/2537290630 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0f40035b0 con 0x7ff10c15d780 2024-05-08T21:23:32.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.763+0000 7ff10bfff700 1 -- 172.21.15.3:0/2537290630 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff10c0d13e0 msgr2=0x7ff10c0d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:32.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.763+0000 7ff10bfff700 1 --2- 172.21.15.3:0/2537290630 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff10c0d13e0 0x7ff10c0d1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.763+0000 7ff10bfff700 1 -- 172.21.15.3:0/2537290630 >> v1:172.21.15.3:6789/0 conn(0x7ff10c15d780 legacy=0x7ff10c0b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:32.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.763+0000 7ff10bfff700 1 -- 172.21.15.3:0/2537290630 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff10c0b3b00 con 0x7ff10c15db60 2024-05-08T21:23:32.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.763+0000 7ff10affd700 1 --2- 172.21.15.3:0/2537290630 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff10c0d13e0 0x7ff10c0d1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:32.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.763+0000 7ff10bfff700 1 --2- 172.21.15.3:0/2537290630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff10c15db60 0x7ff10c0d2120 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff104009d90 tx=0x7ff10400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=364b55253e598855 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:32.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.763+0000 7ff10a7fc700 1 -- 172.21.15.3:0/2537290630 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10400e070 con 0x7ff10c15db60 2024-05-08T21:23:32.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.763+0000 7ff10a7fc700 1 -- 172.21.15.3:0/2537290630 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10400bb20 con 0x7ff10c15db60 2024-05-08T21:23:32.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.764+0000 7ff10a7fc700 1 -- 172.21.15.3:0/2537290630 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff104012430 con 0x7ff10c15db60 2024-05-08T21:23:32.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.764+0000 7ff111a70700 1 -- 172.21.15.3:0/2537290630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff10c15db60 msgr2=0x7ff10c0d2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:32.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.764+0000 7ff111a70700 1 --2- 172.21.15.3:0/2537290630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff10c15db60 0x7ff10c0d2120 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff104009d90 tx=0x7ff10400b330 comp rx=0 tx=0).stop 2024-05-08T21:23:32.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.764+0000 7ff111a70700 1 -- 172.21.15.3:0/2537290630 shutdown_connections 2024-05-08T21:23:32.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.764+0000 7ff111a70700 1 --2- 172.21.15.3:0/2537290630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff10c15db60 0x7ff10c0d2120 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.764+0000 7ff111a70700 1 --2- 172.21.15.3:0/2537290630 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff10c0d13e0 0x7ff10c0d1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.764+0000 7ff111a70700 1 -- 172.21.15.3:0/2537290630 >> 172.21.15.3:0/2537290630 conn(0x7ff10c08afe0 msgr2=0x7ff10c0d3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:32.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.764+0000 7ff111a70700 1 -- 172.21.15.3:0/2537290630 shutdown_connections 2024-05-08T21:23:32.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.764+0000 7ff111a70700 1 -- 172.21.15.3:0/2537290630 wait complete. 2024-05-08T21:23:32.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.765+0000 7ff111a70700 1 Processor -- start 2024-05-08T21:23:32.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.766+0000 7ff111a70700 1 -- start start 2024-05-08T21:23:32.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.766+0000 7ff111a70700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff10c0d13e0 0x7ff10c09d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:32.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.766+0000 7ff111a70700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff10c15d780 0x7ff10c15ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:32.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.766+0000 7ff111a70700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff10c15db60 0x7ff10c0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:32.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.766+0000 7ff111a70700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c1fb490 con 0x7ff10c15db60 2024-05-08T21:23:32.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.766+0000 7ff111a70700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c1fb8b0 con 0x7ff10c0d13e0 2024-05-08T21:23:32.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.766+0000 7ff111a70700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c09d830 con 0x7ff10c15d780 2024-05-08T21:23:32.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.767+0000 7ff10bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff10c15db60 0x7ff10c0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:32.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.767+0000 7ff10bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff10c15db60 0x7ff10c0b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35046/0 (socket says 172.21.15.3:35046) 2024-05-08T21:23:32.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.767+0000 7ff10bfff700 1 -- 172.21.15.3:0/3590442976 learned_addr learned my addr 172.21.15.3:0/3590442976 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:32.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.767+0000 7ff10b7fe700 1 --2- 172.21.15.3:0/3590442976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff10c0d13e0 0x7ff10c09d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:32.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.767+0000 7ff10affd700 1 --2- 172.21.15.3:0/3590442976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff10c15d780 0x7ff10c15ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:32.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.767+0000 7ff10bfff700 1 -- 172.21.15.3:0/3590442976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff10c15d780 msgr2=0x7ff10c15ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:32.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.767+0000 7ff10bfff700 1 --2- 172.21.15.3:0/3590442976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff10c15d780 0x7ff10c15ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.767+0000 7ff10bfff700 1 -- 172.21.15.3:0/3590442976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff10c0d13e0 msgr2=0x7ff10c09d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:32.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.767+0000 7ff10bfff700 1 --2- 172.21.15.3:0/3590442976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff10c0d13e0 0x7ff10c09d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:32.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.767+0000 7ff10bfff700 1 -- 172.21.15.3:0/3590442976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1000094a0 con 0x7ff10c15db60 2024-05-08T21:23:32.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.767+0000 7ff10bfff700 1 --2- 172.21.15.3:0/3590442976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff10c15db60 0x7ff10c0b1b40 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff104014040 tx=0x7ff104004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:32.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.767+0000 7ff0f3fff700 1 -- 172.21.15.3:0/3590442976 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff104012a70 con 0x7ff10c15db60 2024-05-08T21:23:32.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.767+0000 7ff111a70700 1 -- 172.21.15.3:0/3590442976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff104009a70 con 0x7ff10c15db60 2024-05-08T21:23:32.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.767+0000 7ff0f3fff700 1 -- 172.21.15.3:0/3590442976 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff104020070 con 0x7ff10c15db60 2024-05-08T21:23:32.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.767+0000 7ff0f3fff700 1 -- 172.21.15.3:0/3590442976 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff104012e30 con 0x7ff10c15db60 2024-05-08T21:23:32.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.767+0000 7ff111a70700 1 -- 172.21.15.3:0/3590442976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff10c1fadd0 con 0x7ff10c15db60 2024-05-08T21:23:32.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.768+0000 7ff111a70700 1 -- 172.21.15.3:0/3590442976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff10c0d1a50 con 0x7ff10c15db60 2024-05-08T21:23:32.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.769+0000 7ff0f3fff700 1 -- 172.21.15.3:0/3590442976 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff104004040 con 0x7ff10c15db60 2024-05-08T21:23:32.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.770+0000 7ff0f3fff700 1 --2- 172.21.15.3:0/3590442976 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff0f806ac90 0x7ff0f806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:32.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.770+0000 7ff0f3fff700 1 -- 172.21.15.3:0/3590442976 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5810+0+0 (secure 0 0 0) 0x7ff104022070 con 0x7ff10c15db60 2024-05-08T21:23:32.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.770+0000 7ff10b7fe700 1 --2- 172.21.15.3:0/3590442976 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff0f806ac90 0x7ff0f806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:32.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.770+0000 7ff10b7fe700 1 --2- 172.21.15.3:0/3590442976 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff0f806ac90 0x7ff0f806d150 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc007fd0 tx=0x7ff0fc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:32.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:32.775+0000 7ff0f3fff700 1 -- 172.21.15.3:0/3590442976 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff104059d20 con 0x7ff10c15db60 2024-05-08T21:23:33.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.048+0000 7ff111a70700 1 -- 172.21.15.3:0/3590442976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff10c1fb5d0 con 0x7ff10c15db60 2024-05-08T21:23:33.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.049+0000 7ff0f3fff700 1 -- 172.21.15.3:0/3590442976 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4737 (secure 0 0 0) 0x7ff1040598b0 con 0x7ff10c15db60 2024-05-08T21:23:33.051 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:33.051 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":28,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6833/2104267343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2104267343},{"type":"v1","addr":"172.21.15.3:6833","nonce":2104267343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/208321708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":208321708},{"type":"v1","addr":"172.21.15.3:6835","nonce":208321708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/1591386131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1591386131},{"type":"v1","addr":"172.21.15.186:6836","nonce":1591386131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:23:05.475355+0000","modified":"2024-05-08T21:23:15.662101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6837/2000072997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":2000072997},{"type":"v1","addr":"172.21.15.186:6837","nonce":2000072997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":6,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-08T21:23:33.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.051+0000 7ff111a70700 1 -- 172.21.15.3:0/3590442976 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff0f806ac90 msgr2=0x7ff0f806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:33.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.051+0000 7ff111a70700 1 --2- 172.21.15.3:0/3590442976 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff0f806ac90 0x7ff0f806d150 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc007fd0 tx=0x7ff0fc009040 comp rx=0 tx=0).stop 2024-05-08T21:23:33.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.051+0000 7ff111a70700 1 -- 172.21.15.3:0/3590442976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff10c15db60 msgr2=0x7ff10c0b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:33.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.051+0000 7ff111a70700 1 --2- 172.21.15.3:0/3590442976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff10c15db60 0x7ff10c0b1b40 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff104014040 tx=0x7ff104004ad0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.052+0000 7ff111a70700 1 -- 172.21.15.3:0/3590442976 shutdown_connections 2024-05-08T21:23:33.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.052+0000 7ff111a70700 1 --2- 172.21.15.3:0/3590442976 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff0f806ac90 0x7ff0f806d150 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.052+0000 7ff111a70700 1 --2- 172.21.15.3:0/3590442976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff10c15db60 0x7ff10c0b1b40 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.052+0000 7ff111a70700 1 --2- 172.21.15.3:0/3590442976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff10c0d13e0 0x7ff10c09d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.052+0000 7ff111a70700 1 --2- 172.21.15.3:0/3590442976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff10c15d780 0x7ff10c15ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.052+0000 7ff111a70700 1 -- 172.21.15.3:0/3590442976 >> 172.21.15.3:0/3590442976 conn(0x7ff10c08afe0 msgr2=0x7ff10c0b0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:33.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.052+0000 7ff111a70700 1 -- 172.21.15.3:0/3590442976 shutdown_connections 2024-05-08T21:23:33.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.052+0000 7ff111a70700 1 -- 172.21.15.3:0/3590442976 wait complete. 2024-05-08T21:23:33.054 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 28 2024-05-08T21:23:33.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:23:33.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.201+0000 7f131d881700 1 Processor -- start 2024-05-08T21:23:33.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.201+0000 7f131d881700 1 -- start start 2024-05-08T21:23:33.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.202+0000 7f131d881700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1318072af0 0x7f131806cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:33.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.202+0000 7f131d881700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f13180cfef0 0x7f131806d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:33.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.202+0000 7f131d881700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131815dc20 con 0x7f131815d840 2024-05-08T21:23:33.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.202+0000 7f131d881700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1318071420 con 0x7f13180cfef0 2024-05-08T21:23:33.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.202+0000 7f131d881700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13180b2380 con 0x7f1318072af0 2024-05-08T21:23:33.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.202+0000 7f13177fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f13180cfef0 0x7f131806d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:33.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.202+0000 7f13177fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f13180cfef0 0x7f131806d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35060/0 (socket says 172.21.15.3:35060) 2024-05-08T21:23:33.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.202+0000 7f13177fe700 1 -- 172.21.15.3:0/3955224457 learned_addr learned my addr 172.21.15.3:0/3955224457 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:33.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.202+0000 7f13167fc700 1 --2- 172.21.15.3:0/3955224457 >> v2:172.21.15.186:3300/0 conn(0x7f1318072af0 0x7f131806cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:33.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.202+0000 7f13177fe700 1 -- 172.21.15.3:0/3955224457 >> v2:172.21.15.186:3300/0 conn(0x7f1318072af0 msgr2=0x7f131806cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:33.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.202+0000 7f13177fe700 1 --2- 172.21.15.3:0/3955224457 >> v2:172.21.15.186:3300/0 conn(0x7f1318072af0 0x7f131806cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.202+0000 7f13177fe700 1 -- 172.21.15.3:0/3955224457 >> v1:172.21.15.186:6789/0 conn(0x7f131815d840 legacy=0x7f13180ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:33.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.202+0000 7f13177fe700 1 -- 172.21.15.3:0/3955224457 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13180b2660 con 0x7f13180cfef0 2024-05-08T21:23:33.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.202+0000 7f13167fc700 1 --2- 172.21.15.3:0/3955224457 >> v2:172.21.15.186:3300/0 conn(0x7f1318072af0 0x7f131806cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:33.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.203+0000 7f13177fe700 1 --2- 172.21.15.3:0/3955224457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13180cfef0 0x7f131806d0d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f131806fc70 tx=0x7f131000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b09cec115a3884b server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:33.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.203+0000 7f1315ffb700 1 -- 172.21.15.3:0/3955224457 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131000e070 con 0x7f13180cfef0 2024-05-08T21:23:33.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.203+0000 7f1315ffb700 1 -- 172.21.15.3:0/3955224457 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1310003a20 con 0x7f13180cfef0 2024-05-08T21:23:33.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.203+0000 7f1315ffb700 1 -- 172.21.15.3:0/3955224457 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1310003dd0 con 0x7f13180cfef0 2024-05-08T21:23:33.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.203+0000 7f131d881700 1 -- 172.21.15.3:0/3955224457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13180cfef0 msgr2=0x7f131806d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:33.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.203+0000 7f131d881700 1 --2- 172.21.15.3:0/3955224457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13180cfef0 0x7f131806d0d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f131806fc70 tx=0x7f131000b330 comp rx=0 tx=0).stop 2024-05-08T21:23:33.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.203+0000 7f131d881700 1 -- 172.21.15.3:0/3955224457 shutdown_connections 2024-05-08T21:23:33.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.203+0000 7f131d881700 1 --2- 172.21.15.3:0/3955224457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13180cfef0 0x7f131806d0d0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.203+0000 7f131d881700 1 --2- 172.21.15.3:0/3955224457 >> v2:172.21.15.186:3300/0 conn(0x7f1318072af0 0x7f131806cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.203+0000 7f131d881700 1 -- 172.21.15.3:0/3955224457 >> 172.21.15.3:0/3955224457 conn(0x7f13180cf270 msgr2=0x7f13180cf650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:33.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.204+0000 7f131d881700 1 -- 172.21.15.3:0/3955224457 shutdown_connections 2024-05-08T21:23:33.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.204+0000 7f131d881700 1 -- 172.21.15.3:0/3955224457 wait complete. 2024-05-08T21:23:33.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.205+0000 7f131d881700 1 Processor -- start 2024-05-08T21:23:33.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.205+0000 7f131d881700 1 -- start start 2024-05-08T21:23:33.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.206+0000 7f131d881700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1318072af0 0x7f1318156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:33.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.206+0000 7f131d881700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13180cfef0 0x7f131814d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:33.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.206+0000 7f131d881700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f131815d840 0x7f1318151ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:33.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.206+0000 7f131d881700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13181f6bf0 con 0x7f1318072af0 2024-05-08T21:23:33.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.206+0000 7f131d881700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13181f6e40 con 0x7f13180cfef0 2024-05-08T21:23:33.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.206+0000 7f131d881700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13181f7120 con 0x7f131815d840 2024-05-08T21:23:33.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f1316ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1318072af0 0x7f1318156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:33.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f1316ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1318072af0 0x7f1318156f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35072/0 (socket says 172.21.15.3:35072) 2024-05-08T21:23:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f1316ffd700 1 -- 172.21.15.3:0/2400664407 learned_addr learned my addr 172.21.15.3:0/2400664407 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f13167fc700 1 --2- 172.21.15.3:0/2400664407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13180cfef0 0x7f131814d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f13177fe700 1 --2- 172.21.15.3:0/2400664407 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f131815d840 0x7f1318151ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f1316ffd700 1 -- 172.21.15.3:0/2400664407 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f131815d840 msgr2=0x7f1318151ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f1316ffd700 1 --2- 172.21.15.3:0/2400664407 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f131815d840 0x7f1318151ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f1316ffd700 1 -- 172.21.15.3:0/2400664407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13180cfef0 msgr2=0x7f131814d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f1316ffd700 1 --2- 172.21.15.3:0/2400664407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13180cfef0 0x7f131814d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f1316ffd700 1 -- 172.21.15.3:0/2400664407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1310009b30 con 0x7f1318072af0 2024-05-08T21:23:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f13177fe700 1 --2- 172.21.15.3:0/2400664407 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f131815d840 0x7f1318151ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f13167fc700 1 --2- 172.21.15.3:0/2400664407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13180cfef0 0x7f131814d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f1316ffd700 1 --2- 172.21.15.3:0/2400664407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1318072af0 0x7f1318156f70 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f130800b4a0 tx=0x7f130800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f12fffff700 1 -- 172.21.15.3:0/2400664407 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1308011830 con 0x7f1318072af0 2024-05-08T21:23:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f12fffff700 1 -- 172.21.15.3:0/2400664407 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1308011990 con 0x7f1318072af0 2024-05-08T21:23:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f131d881700 1 -- 172.21.15.3:0/2400664407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13181f7400 con 0x7f1318072af0 2024-05-08T21:23:33.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f12fffff700 1 -- 172.21.15.3:0/2400664407 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1308011d40 con 0x7f1318072af0 2024-05-08T21:23:33.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.207+0000 7f131d881700 1 -- 172.21.15.3:0/2400664407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13181574b0 con 0x7f1318072af0 2024-05-08T21:23:33.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.209+0000 7f131d881700 1 -- 172.21.15.3:0/2400664407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13181f7920 con 0x7f1318072af0 2024-05-08T21:23:33.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.209+0000 7f12fffff700 1 -- 172.21.15.3:0/2400664407 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f130801a400 con 0x7f1318072af0 2024-05-08T21:23:33.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.209+0000 7f12fffff700 1 --2- 172.21.15.3:0/2400664407 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f130406ac90 0x7f130406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:33.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.210+0000 7f12fffff700 1 -- 172.21.15.3:0/2400664407 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5810+0+0 (secure 0 0 0) 0x7f13080890c0 con 0x7f1318072af0 2024-05-08T21:23:33.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.210+0000 7f13167fc700 1 --2- 172.21.15.3:0/2400664407 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f130406ac90 0x7f130406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:33.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.210+0000 7f13167fc700 1 --2- 172.21.15.3:0/2400664407 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f130406ac90 0x7f130406d150 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1318152790 tx=0x7f130c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:33.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.214+0000 7f12fffff700 1 -- 172.21.15.3:0/2400664407 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f13181f7920 con 0x7f1318072af0 2024-05-08T21:23:33.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.481+0000 7f131d881700 1 -- 172.21.15.3:0/2400664407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13181f7920 con 0x7f1318072af0 2024-05-08T21:23:33.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.481+0000 7f12fffff700 1 -- 172.21.15.3:0/2400664407 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f13181f7920 con 0x7f1318072af0 2024-05-08T21:23:33.483 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:33.483 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":28,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6833/2104267343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2104267343},{"type":"v1","addr":"172.21.15.3:6833","nonce":2104267343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/208321708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":208321708},{"type":"v1","addr":"172.21.15.3:6835","nonce":208321708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/1591386131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1591386131},{"type":"v1","addr":"172.21.15.186:6836","nonce":1591386131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T21:23:05.475355+0000","modified":"2024-05-08T21:23:15.662101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"d","rank":0,"incarnation":23,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6837/2000072997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":2000072997},{"type":"v1","addr":"172.21.15.186:6837","nonce":2000072997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":6,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-08T21:23:33.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.483+0000 7f131d881700 1 -- 172.21.15.3:0/2400664407 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f130406ac90 msgr2=0x7f130406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:33.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.483+0000 7f131d881700 1 --2- 172.21.15.3:0/2400664407 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f130406ac90 0x7f130406d150 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1318152790 tx=0x7f130c008040 comp rx=0 tx=0).stop 2024-05-08T21:23:33.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.483+0000 7f131d881700 1 -- 172.21.15.3:0/2400664407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1318072af0 msgr2=0x7f1318156f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:33.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.483+0000 7f131d881700 1 --2- 172.21.15.3:0/2400664407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1318072af0 0x7f1318156f70 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f130800b4a0 tx=0x7f130800b860 comp rx=0 tx=0).stop 2024-05-08T21:23:33.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.483+0000 7f131d881700 1 -- 172.21.15.3:0/2400664407 shutdown_connections 2024-05-08T21:23:33.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.484+0000 7f131d881700 1 --2- 172.21.15.3:0/2400664407 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f131815d840 0x7f1318151ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.484+0000 7f131d881700 1 --2- 172.21.15.3:0/2400664407 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f130406ac90 0x7f130406d150 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.484+0000 7f131d881700 1 --2- 172.21.15.3:0/2400664407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1318072af0 0x7f1318156f70 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.484+0000 7f131d881700 1 --2- 172.21.15.3:0/2400664407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13180cfef0 0x7f131814d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.484+0000 7f131d881700 1 -- 172.21.15.3:0/2400664407 >> 172.21.15.3:0/2400664407 conn(0x7f13180cf270 msgr2=0x7f1318153c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:33.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.484+0000 7f131d881700 1 -- 172.21.15.3:0/2400664407 shutdown_connections 2024-05-08T21:23:33.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.484+0000 7f131d881700 1 -- 172.21.15.3:0/2400664407 wait complete. 2024-05-08T21:23:33.487 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 28 2024-05-08T21:23:33.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:23:33.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.634+0000 7feaf4a81700 1 Processor -- start 2024-05-08T21:23:33.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.634+0000 7feaf4a81700 1 -- start start 2024-05-08T21:23:33.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.635+0000 7feaf4a81700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feaf0071c90 0x7feaf015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:33.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.635+0000 7feaf4a81700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feaf006cb90 0x7feaf006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:33.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.635+0000 7feaf4a81700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf01547e0 con 0x7feaf0154400 2024-05-08T21:23:33.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.635+0000 7feaf4a81700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaf015b290 con 0x7feaf006cb90 2024-05-08T21:23:33.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.635+0000 7feaf4a81700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf006dae0 con 0x7feaf0071c90 2024-05-08T21:23:33.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.635+0000 7feaeed9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feaf006cb90 0x7feaf006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:33.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.635+0000 7feaeed9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feaf006cb90 0x7feaf006cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35082/0 (socket says 172.21.15.3:35082) 2024-05-08T21:23:33.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.635+0000 7feaeed9d700 1 -- 172.21.15.3:0/2026761073 learned_addr learned my addr 172.21.15.3:0/2026761073 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:33.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.635+0000 7feaedd9b700 1 --2- 172.21.15.3:0/2026761073 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feaf0071c90 0x7feaf015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:33.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.635+0000 7feaeed9d700 1 -- 172.21.15.3:0/2026761073 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feaf0071c90 msgr2=0x7feaf015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:33.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.635+0000 7feaeed9d700 1 --2- 172.21.15.3:0/2026761073 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feaf0071c90 0x7feaf015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.635+0000 7feaeed9d700 1 -- 172.21.15.3:0/2026761073 >> v1:172.21.15.186:6789/0 conn(0x7feaf0154400 legacy=0x7feaf0071580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:33.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.635+0000 7feaeed9d700 1 -- 172.21.15.3:0/2026761073 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf006ddc0 con 0x7feaf006cb90 2024-05-08T21:23:33.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.636+0000 7feaeed9d700 1 --2- 172.21.15.3:0/2026761073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feaf006cb90 0x7feaf006cfa0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7feae8009e50 tx=0x7feae800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e4d1645f4c033f server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:33.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.636+0000 7feaed59a700 1 -- 172.21.15.3:0/2026761073 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae800e070 con 0x7feaf006cb90 2024-05-08T21:23:33.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.636+0000 7feaed59a700 1 -- 172.21.15.3:0/2026761073 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae800b9c0 con 0x7feaf006cb90 2024-05-08T21:23:33.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.636+0000 7feaed59a700 1 -- 172.21.15.3:0/2026761073 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae800bd70 con 0x7feaf006cb90 2024-05-08T21:23:33.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.636+0000 7feaf4a81700 1 -- 172.21.15.3:0/2026761073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feaf006cb90 msgr2=0x7feaf006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:33.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.636+0000 7feaf4a81700 1 --2- 172.21.15.3:0/2026761073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feaf006cb90 0x7feaf006cfa0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7feae8009e50 tx=0x7feae800b330 comp rx=0 tx=0).stop 2024-05-08T21:23:33.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.636+0000 7feaf4a81700 1 -- 172.21.15.3:0/2026761073 shutdown_connections 2024-05-08T21:23:33.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.636+0000 7feaf4a81700 1 --2- 172.21.15.3:0/2026761073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feaf006cb90 0x7feaf006cfa0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.636+0000 7feaf4a81700 1 --2- 172.21.15.3:0/2026761073 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feaf0071c90 0x7feaf015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.636+0000 7feaf4a81700 1 -- 172.21.15.3:0/2026761073 >> 172.21.15.3:0/2026761073 conn(0x7feaf00cf270 msgr2=0x7feaf00cf650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:33.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.636+0000 7feaf4a81700 1 -- 172.21.15.3:0/2026761073 shutdown_connections 2024-05-08T21:23:33.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.637+0000 7feaf4a81700 1 -- 172.21.15.3:0/2026761073 wait complete. 2024-05-08T21:23:33.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.638+0000 7feaf4a81700 1 Processor -- start 2024-05-08T21:23:33.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.638+0000 7feaf4a81700 1 -- start start 2024-05-08T21:23:33.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.639+0000 7feaf4a81700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feaf006cb90 0x7feaf00989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:33.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.639+0000 7feaf4a81700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feaf0071c90 0x7feaf01549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:33.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.639+0000 7feaf4a81700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feaf0154400 0x7feaf006a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:33.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.639+0000 7feaf4a81700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf0157790 con 0x7feaf006cb90 2024-05-08T21:23:33.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.639+0000 7feaf4a81700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf0098ee0 con 0x7feaf0071c90 2024-05-08T21:23:33.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.639+0000 7feaf4a81700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf0156b10 con 0x7feaf0154400 2024-05-08T21:23:33.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.640+0000 7feaee59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feaf006cb90 0x7feaf00989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:33.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.640+0000 7feaee59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feaf006cb90 0x7feaf00989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35092/0 (socket says 172.21.15.3:35092) 2024-05-08T21:23:33.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.640+0000 7feaee59c700 1 -- 172.21.15.3:0/2143278339 learned_addr learned my addr 172.21.15.3:0/2143278339 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:33.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.640+0000 7feaeed9d700 1 --2- 172.21.15.3:0/2143278339 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feaf0154400 0x7feaf006a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:33.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.640+0000 7feaedd9b700 1 --2- 172.21.15.3:0/2143278339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feaf0071c90 0x7feaf01549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:33.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.640+0000 7feaee59c700 1 -- 172.21.15.3:0/2143278339 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feaf0154400 msgr2=0x7feaf006a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:33.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.640+0000 7feaee59c700 1 --2- 172.21.15.3:0/2143278339 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feaf0154400 0x7feaf006a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.640+0000 7feaee59c700 1 -- 172.21.15.3:0/2143278339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feaf0071c90 msgr2=0x7feaf01549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:33.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.640+0000 7feaee59c700 1 --2- 172.21.15.3:0/2143278339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feaf0071c90 0x7feaf01549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.640+0000 7feaee59c700 1 -- 172.21.15.3:0/2143278339 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae8009b30 con 0x7feaf006cb90 2024-05-08T21:23:33.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.640+0000 7feaee59c700 1 --2- 172.21.15.3:0/2143278339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feaf006cb90 0x7feaf00989a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7feae000c450 tx=0x7feae000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:33.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.640+0000 7feadeffd700 1 -- 172.21.15.3:0/2143278339 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae00052d0 con 0x7feaf006cb90 2024-05-08T21:23:33.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.640+0000 7feadeffd700 1 -- 172.21.15.3:0/2143278339 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae0005430 con 0x7feaf006cb90 2024-05-08T21:23:33.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.640+0000 7feaf4a81700 1 -- 172.21.15.3:0/2143278339 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf0156df0 con 0x7feaf006cb90 2024-05-08T21:23:33.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.640+0000 7feadeffd700 1 -- 172.21.15.3:0/2143278339 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae00038c0 con 0x7feaf006cb90 2024-05-08T21:23:33.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.640+0000 7feaf4a81700 1 -- 172.21.15.3:0/2143278339 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf01573b0 con 0x7feaf006cb90 2024-05-08T21:23:33.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.642+0000 7feaf4a81700 1 -- 172.21.15.3:0/2143278339 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf00678f0 con 0x7feaf006cb90 2024-05-08T21:23:33.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.642+0000 7feadeffd700 1 -- 172.21.15.3:0/2143278339 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7feae0003a20 con 0x7feaf006cb90 2024-05-08T21:23:33.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.643+0000 7feadeffd700 1 --2- 172.21.15.3:0/2143278339 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fead006ac90 0x7fead006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:33.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.643+0000 7feadeffd700 1 -- 172.21.15.3:0/2143278339 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5810+0+0 (secure 0 0 0) 0x7feae00886b0 con 0x7feaf006cb90 2024-05-08T21:23:33.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.643+0000 7feaedd9b700 1 --2- 172.21.15.3:0/2143278339 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fead006ac90 0x7fead006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:33.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.643+0000 7feaedd9b700 1 --2- 172.21.15.3:0/2143278339 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fead006ac90 0x7fead006d150 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7feaf00b0360 tx=0x7feae4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:33.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.648+0000 7feadeffd700 1 -- 172.21.15.3:0/2143278339 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7feae0053820 con 0x7feaf006cb90 2024-05-08T21:23:33.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.869+0000 7feaf4a81700 1 -- 172.21.15.3:0/2143278339 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feaf015dc90 con 0x7feaf006cb90 2024-05-08T21:23:33.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.870+0000 7feadeffd700 1 -- 172.21.15.3:0/2143278339 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+16195 (secure 0 0 0) 0x7feae000ce60 con 0x7feaf006cb90 2024-05-08T21:23:33.872 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:33.872 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":43,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:23:29.839993+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-08T21:22:49.153759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"recovery_fs_metadata","create_time":"2024-05-08T21:23:03.921695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6834/3599481370":"2024-05-09T21:23:01.563392+0000","172.21.15.3:6832/3599481370":"2024-05-09T21:23:01.563392+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:23:33.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.872+0000 7feaf4a81700 1 -- 172.21.15.3:0/2143278339 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fead006ac90 msgr2=0x7fead006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:33.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.872+0000 7feaf4a81700 1 --2- 172.21.15.3:0/2143278339 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fead006ac90 0x7fead006d150 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7feaf00b0360 tx=0x7feae4008040 comp rx=0 tx=0).stop 2024-05-08T21:23:33.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.872+0000 7feaf4a81700 1 -- 172.21.15.3:0/2143278339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feaf006cb90 msgr2=0x7feaf00989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:33.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.872+0000 7feaf4a81700 1 --2- 172.21.15.3:0/2143278339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feaf006cb90 0x7feaf00989a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7feae000c450 tx=0x7feae000c760 comp rx=0 tx=0).stop 2024-05-08T21:23:33.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.873+0000 7feaf4a81700 1 -- 172.21.15.3:0/2143278339 shutdown_connections 2024-05-08T21:23:33.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.873+0000 7feaf4a81700 1 --2- 172.21.15.3:0/2143278339 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feaf0154400 0x7feaf006a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.873+0000 7feaf4a81700 1 --2- 172.21.15.3:0/2143278339 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fead006ac90 0x7fead006d150 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.873+0000 7feaf4a81700 1 --2- 172.21.15.3:0/2143278339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feaf006cb90 0x7feaf00989a0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.873+0000 7feaf4a81700 1 --2- 172.21.15.3:0/2143278339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feaf0071c90 0x7feaf01549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:33.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.873+0000 7feaf4a81700 1 -- 172.21.15.3:0/2143278339 >> 172.21.15.3:0/2143278339 conn(0x7feaf00cf270 msgr2=0x7feaf006cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:33.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.873+0000 7feaf4a81700 1 -- 172.21.15.3:0/2143278339 shutdown_connections 2024-05-08T21:23:33.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:33.873+0000 7feaf4a81700 1 -- 172.21.15.3:0/2143278339 wait complete. 2024-05-08T21:23:33.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail recovery_fs 2024-05-08T21:23:34.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.026+0000 7fe493602700 1 Processor -- start 2024-05-08T21:23:34.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.026+0000 7fe493602700 1 -- start start 2024-05-08T21:23:34.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.027+0000 7fe493602700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe48c0b1180 0x7fe48c0b1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:34.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.027+0000 7fe493602700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48c1622b0 con 0x7fe48c0c4430 2024-05-08T21:23:34.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.027+0000 7fe493602700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48c0b25c0 con 0x7fe48c09a430 2024-05-08T21:23:34.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.027+0000 7fe493602700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48c0b2c60 con 0x7fe48c0b1180 2024-05-08T21:23:34.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.027+0000 7fe491b9f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe48c0b1180 0x7fe48c0b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:34.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.027+0000 7fe49139e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe48c0c4430 0x7fe48c0c4810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:36392/0 (socket says 172.21.15.3:36392) 2024-05-08T21:23:34.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.027+0000 7fe49139e700 1 -- 172.21.15.3:0/1806956795 learned_addr learned my addr 172.21.15.3:0/1806956795 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:34.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.027+0000 7fe491b9f700 1 -- 172.21.15.3:0/1806956795 >> v1:172.21.15.186:6789/0 conn(0x7fe48c09a430 legacy=0x7fe48c0b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:34.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.027+0000 7fe491b9f700 1 -- 172.21.15.3:0/1806956795 >> v1:172.21.15.3:6789/0 conn(0x7fe48c0c4430 legacy=0x7fe48c0c4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:34.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.027+0000 7fe491b9f700 1 -- 172.21.15.3:0/1806956795 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c06bde0 con 0x7fe48c0b1180 2024-05-08T21:23:34.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.027+0000 7fe483fff700 1 -- 172.21.15.3:0/1806956795 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654285765 0 0) 0x7fe488001110 con 0x7fe48c0c4430 2024-05-08T21:23:34.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.028+0000 7fe491b9f700 1 --2- 172.21.15.3:0/1806956795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe48c0b1180 0x7fe48c0b1590 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe484009d90 tx=0x7fe48400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2aee07f8371de03 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:34.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.028+0000 7fe483fff700 1 -- 172.21.15.3:0/1806956795 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48400e070 con 0x7fe48c0b1180 2024-05-08T21:23:34.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.028+0000 7fe483fff700 1 -- 172.21.15.3:0/1806956795 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48400bb90 con 0x7fe48c0b1180 2024-05-08T21:23:34.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.028+0000 7fe483fff700 1 -- 172.21.15.3:0/1806956795 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484003a20 con 0x7fe48c0b1180 2024-05-08T21:23:34.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.028+0000 7fe493602700 1 -- 172.21.15.3:0/1806956795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe48c0b1180 msgr2=0x7fe48c0b1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:34.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.028+0000 7fe493602700 1 --2- 172.21.15.3:0/1806956795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe48c0b1180 0x7fe48c0b1590 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe484009d90 tx=0x7fe48400b330 comp rx=0 tx=0).stop 2024-05-08T21:23:34.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.028+0000 7fe493602700 1 -- 172.21.15.3:0/1806956795 shutdown_connections 2024-05-08T21:23:34.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.028+0000 7fe493602700 1 --2- 172.21.15.3:0/1806956795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe48c0b1180 0x7fe48c0b1590 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:34.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.028+0000 7fe493602700 1 -- 172.21.15.3:0/1806956795 >> 172.21.15.3:0/1806956795 conn(0x7fe48c0b2f50 msgr2=0x7fe48c0b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:34.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.029+0000 7fe493602700 1 -- 172.21.15.3:0/1806956795 shutdown_connections 2024-05-08T21:23:34.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.029+0000 7fe493602700 1 -- 172.21.15.3:0/1806956795 wait complete. 2024-05-08T21:23:34.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.030+0000 7fe493602700 1 Processor -- start 2024-05-08T21:23:34.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.030+0000 7fe493602700 1 -- start start 2024-05-08T21:23:34.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe493602700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe48c09a430 0x7fe48c153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:34.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe493602700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe48c0b1180 0x7fe48c150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:34.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe493602700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe48c0c4430 0x7fe48c150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:34.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe493602700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c153d60 con 0x7fe48c0b1180 2024-05-08T21:23:34.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe493602700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0c49b0 con 0x7fe48c0c4430 2024-05-08T21:23:34.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe493602700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0c4c90 con 0x7fe48c09a430 2024-05-08T21:23:34.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe490b9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe48c0b1180 0x7fe48c150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:34.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe490b9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe48c0b1180 0x7fe48c150d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35118/0 (socket says 172.21.15.3:35118) 2024-05-08T21:23:34.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe490b9d700 1 -- 172.21.15.3:0/1647697560 learned_addr learned my addr 172.21.15.3:0/1647697560 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:34.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe491b9f700 1 --2- 172.21.15.3:0/1647697560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe48c0c4430 0x7fe48c150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:34.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe49139e700 1 --2- 172.21.15.3:0/1647697560 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe48c09a430 0x7fe48c153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:34.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe490b9d700 1 -- 172.21.15.3:0/1647697560 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe48c09a430 msgr2=0x7fe48c153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:34.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe490b9d700 1 --2- 172.21.15.3:0/1647697560 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe48c09a430 0x7fe48c153820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:34.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe490b9d700 1 -- 172.21.15.3:0/1647697560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe48c0c4430 msgr2=0x7fe48c150480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:34.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe490b9d700 1 --2- 172.21.15.3:0/1647697560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe48c0c4430 0x7fe48c150480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:34.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe490b9d700 1 -- 172.21.15.3:0/1647697560 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe484009a70 con 0x7fe48c0b1180 2024-05-08T21:23:34.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe491b9f700 1 --2- 172.21.15.3:0/1647697560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe48c0c4430 0x7fe48c150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:34.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe490b9d700 1 --2- 172.21.15.3:0/1647697560 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe48c0b1180 0x7fe48c150d80 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe47c00b4a0 tx=0x7fe47c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:34.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe481ffb700 1 -- 172.21.15.3:0/1647697560 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c011830 con 0x7fe48c0b1180 2024-05-08T21:23:34.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe481ffb700 1 -- 172.21.15.3:0/1647697560 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c011990 con 0x7fe48c0b1180 2024-05-08T21:23:34.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.031+0000 7fe493602700 1 -- 172.21.15.3:0/1647697560 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48c0c66c0 con 0x7fe48c0b1180 2024-05-08T21:23:34.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.032+0000 7fe481ffb700 1 -- 172.21.15.3:0/1647697560 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c011d40 con 0x7fe48c0b1180 2024-05-08T21:23:34.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.032+0000 7fe493602700 1 -- 172.21.15.3:0/1647697560 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48c1512c0 con 0x7fe48c0b1180 2024-05-08T21:23:34.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.033+0000 7fe493602700 1 -- 172.21.15.3:0/1647697560 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48c068000 con 0x7fe48c0b1180 2024-05-08T21:23:34.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.033+0000 7fe481ffb700 1 -- 172.21.15.3:0/1647697560 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe47c01a400 con 0x7fe48c0b1180 2024-05-08T21:23:34.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.034+0000 7fe481ffb700 1 --2- 172.21.15.3:0/1647697560 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe47006ac90 0x7fe47006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:34.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.034+0000 7fe481ffb700 1 -- 172.21.15.3:0/1647697560 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5810+0+0 (secure 0 0 0) 0x7fe47c089000 con 0x7fe48c0b1180 2024-05-08T21:23:34.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.034+0000 7fe49139e700 1 --2- 172.21.15.3:0/1647697560 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe47006ac90 0x7fe47006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:34.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.034+0000 7fe49139e700 1 --2- 172.21.15.3:0/1647697560 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe47006ac90 0x7fe47006d150 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe488000f60 tx=0x7fe488007aa0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:34.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.038+0000 7fe481ffb700 1 -- 172.21.15.3:0/1647697560 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe47c0bc900 con 0x7fe48c0b1180 2024-05-08T21:23:34.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.305+0000 7fe493602700 1 -- 172.21.15.3:0/1647697560 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "recovery_fs"} v 0) v1 -- 0x7fe48c0cf910 con 0x7fe48c0b1180 2024-05-08T21:23:34.307 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-05-08T21:23:34.306+0000 7f74f7652700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-08T21:23:34.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.313+0000 7fe481ffb700 1 -- 172.21.15.3:0/1647697560 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "recovery_fs"}]=0 recovery_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) v1 ==== 171+0+0 (secure 0 0 0) 0x7fe47c054200 con 0x7fe48c0b1180 2024-05-08T21:23:34.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.316+0000 7fe493602700 1 -- 172.21.15.3:0/1647697560 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe47006ac90 msgr2=0x7fe47006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:34.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.316+0000 7fe493602700 1 --2- 172.21.15.3:0/1647697560 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe47006ac90 0x7fe47006d150 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe488000f60 tx=0x7fe488007aa0 comp rx=0 tx=0).stop 2024-05-08T21:23:34.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.316+0000 7fe493602700 1 -- 172.21.15.3:0/1647697560 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe48c0b1180 msgr2=0x7fe48c150d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:34.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.316+0000 7fe493602700 1 --2- 172.21.15.3:0/1647697560 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe48c0b1180 0x7fe48c150d80 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe47c00b4a0 tx=0x7fe47c00b860 comp rx=0 tx=0).stop 2024-05-08T21:23:34.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.316+0000 7fe493602700 1 -- 172.21.15.3:0/1647697560 shutdown_connections 2024-05-08T21:23:34.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.316+0000 7fe493602700 1 --2- 172.21.15.3:0/1647697560 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe48c09a430 0x7fe48c153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:34.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.316+0000 7fe493602700 1 --2- 172.21.15.3:0/1647697560 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe47006ac90 0x7fe47006d150 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:34.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.316+0000 7fe493602700 1 --2- 172.21.15.3:0/1647697560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe48c0c4430 0x7fe48c150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:34.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.316+0000 7fe493602700 1 --2- 172.21.15.3:0/1647697560 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe48c0b1180 0x7fe48c150d80 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:34.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.316+0000 7fe493602700 1 -- 172.21.15.3:0/1647697560 >> 172.21.15.3:0/1647697560 conn(0x7fe48c0b2f50 msgr2=0x7fe48c0b0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:34.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.316+0000 7fe493602700 1 -- 172.21.15.3:0/1647697560 shutdown_connections 2024-05-08T21:23:34.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.316+0000 7fe493602700 1 -- 172.21.15.3:0/1647697560 wait complete. 2024-05-08T21:23:34.319 INFO:tasks.ceph.mds.d.smithi186.stderr: -6090> 2024-05-08T21:23:12.662+0000 7f6cfa1b6700 -1 mds.0.openfiles _load_finish got (2) No such file or directory 2024-05-08T21:23:34.319 INFO:teuthology.orchestra.run.smithi003.stderr:recovery_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-08T21:23:34.320 INFO:tasks.ceph.mds.d.smithi186.stderr: -4630> 2024-05-08T21:23:17.615+0000 7f6cfd9bd700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-08T21:23:34.321 INFO:tasks.ceph.mds.d.smithi186.stderr: -4433> 2024-05-08T21:23:18.002+0000 7f6cfa1b6700 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x10000000000 2024-05-08T21:23:34.321 INFO:tasks.ceph.mds.d.smithi186.stderr: -4297> 2024-05-08T21:23:18.003+0000 7f6cfa1b6700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /subdir/ auth v1 pv4 ap=3 DIRTYPARENT f(v0 1=1+0) n() (inest lock dirty) (ifile lock->sync w=1 flushing) (iversion lock) | dirtyscattered=2 lock=1 dirfrag=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55d2cb12d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(6)0x10000000000:[<0x1/subdir v1>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 1=1+0)","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 1=1+0)","memory_value.rstat":"n()","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-61} 2024-05-08T21:23:34.321 INFO:tasks.ceph.mds.d.smithi186.stderr: -4243> 2024-05-08T21:23:18.004+0000 7f6cfa1b6700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /subdir/sixmegs auth v1 ap=1 DIRTYPARENT s=6291456 n() (iversion lock) | dirtyparent=1 authpin=1 scrubqueue=0 0x55d2cb173b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(5)0x10000000001:[<0x10000000000/sixmegs v10>,<0x1/subdir v12>]//[]","memoryvalue":"(5)0x10000000001:[<0x10000000000/sixmegs v1>,<0x1/subdir v1>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-08T21:23:34.321 INFO:tasks.ceph.mds.d.smithi186.stderr: -4003> 2024-05-08T21:23:18.008+0000 7f6cfa1b6700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1 [...2,head] / auth v4 ap=5 snaprealm=0x55d2cb128d80 DIRTYPARENT f(v1 1=0+1) n(v0 1=0+1)/n() (idft sync r=1) (inest sync r=1) (ifile sync->mix r=1) (iversion lock) | dirtyscattered=0 lock=3 dirfrag=1 openingsnapparents=0 dirtyparent=1 dirty=1 waiter=1 authpin=1 scrubqueue=0 0x55d2cb12cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(6)0x1:[]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 1=0+1)","ondisk_value.rstat":"n(v0 2=0+2)","memory_value.dirstat":"f(v1 1=0+1)","memory_value.rstat":"n(v0 1=0+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-61} 2024-05-08T21:23:34.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm recovery_fs --yes-i-really-mean-it 2024-05-08T21:23:34.351 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2024-05-08T21:23:34.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.469+0000 7f21ad2b9700 1 Processor -- start 2024-05-08T21:23:34.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.469+0000 7f21ad2b9700 1 -- start start 2024-05-08T21:23:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.469+0000 7f21ad2b9700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f21a80cd450 0x7f21a806c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.470+0000 7f21ad2b9700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f21a809a160 0x7f21a815d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.470+0000 7f21ad2b9700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a80cf770 con 0x7f21a806cbd0 2024-05-08T21:23:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.470+0000 7f21ad2b9700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a80ccb00 con 0x7f21a809a160 2024-05-08T21:23:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.470+0000 7f21ad2b9700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a80cf9a0 con 0x7f21a80cd450 2024-05-08T21:23:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.470+0000 7f21a77fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f21a809a160 0x7f21a815d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.470+0000 7f21a6ffd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f21a80cd450 0x7f21a806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.470+0000 7f21a77fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f21a809a160 0x7f21a815d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35128/0 (socket says 172.21.15.3:35128) 2024-05-08T21:23:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.470+0000 7f21a77fe700 1 -- 172.21.15.3:0/953650125 learned_addr learned my addr 172.21.15.3:0/953650125 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.470+0000 7f21a77fe700 1 -- 172.21.15.3:0/953650125 >> v2:172.21.15.186:3300/0 conn(0x7f21a80cd450 msgr2=0x7f21a806c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.470+0000 7f21a77fe700 1 --2- 172.21.15.3:0/953650125 >> v2:172.21.15.186:3300/0 conn(0x7f21a80cd450 0x7f21a806c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.470+0000 7f21a77fe700 1 -- 172.21.15.3:0/953650125 >> v1:172.21.15.186:6789/0 conn(0x7f21a806cbd0 legacy=0x7f21a8099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:34.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.470+0000 7f21a77fe700 1 -- 172.21.15.3:0/953650125 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a80cfc80 con 0x7f21a809a160 2024-05-08T21:23:34.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.470+0000 7f21a77fe700 1 --2- 172.21.15.3:0/953650125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a809a160 0x7f21a815d110 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f21a8067160 tx=0x7f21a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de1c931ed3a87d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:34.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.470+0000 7f21a5ffb700 1 -- 172.21.15.3:0/953650125 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a000e070 con 0x7f21a809a160 2024-05-08T21:23:34.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.470+0000 7f21a5ffb700 1 -- 172.21.15.3:0/953650125 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a000b910 con 0x7f21a809a160 2024-05-08T21:23:34.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.470+0000 7f21a5ffb700 1 -- 172.21.15.3:0/953650125 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a000bcc0 con 0x7f21a809a160 2024-05-08T21:23:34.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.471+0000 7f21ad2b9700 1 -- 172.21.15.3:0/953650125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a809a160 msgr2=0x7f21a815d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:34.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.471+0000 7f21ad2b9700 1 --2- 172.21.15.3:0/953650125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a809a160 0x7f21a815d110 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f21a8067160 tx=0x7f21a000b330 comp rx=0 tx=0).stop 2024-05-08T21:23:34.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.471+0000 7f21ad2b9700 1 -- 172.21.15.3:0/953650125 shutdown_connections 2024-05-08T21:23:34.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.471+0000 7f21ad2b9700 1 --2- 172.21.15.3:0/953650125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a809a160 0x7f21a815d110 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:34.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.471+0000 7f21ad2b9700 1 --2- 172.21.15.3:0/953650125 >> v2:172.21.15.186:3300/0 conn(0x7f21a80cd450 0x7f21a806c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:34.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.471+0000 7f21ad2b9700 1 -- 172.21.15.3:0/953650125 >> 172.21.15.3:0/953650125 conn(0x7f21a80b2f90 msgr2=0x7f21a80b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:34.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.471+0000 7f21ad2b9700 1 -- 172.21.15.3:0/953650125 shutdown_connections 2024-05-08T21:23:34.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.471+0000 7f21ad2b9700 1 -- 172.21.15.3:0/953650125 wait complete. 2024-05-08T21:23:34.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.473+0000 7f21ad2b9700 1 Processor -- start 2024-05-08T21:23:34.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.473+0000 7f21ad2b9700 1 -- start start 2024-05-08T21:23:34.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.473+0000 7f21ad2b9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a806cbd0 0x7f21a80c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:34.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.473+0000 7f21ad2b9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21a809a160 0x7f21a81f0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:34.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21ad2b9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21a80cd450 0x7f21a815cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21ad2b9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a80c9950 con 0x7f21a806cbd0 2024-05-08T21:23:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21ad2b9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a81f6c90 con 0x7f21a809a160 2024-05-08T21:23:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21ad2b9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a81f6f70 con 0x7f21a80cd450 2024-05-08T21:23:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21a6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a806cbd0 0x7f21a80c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21a6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a806cbd0 0x7f21a80c9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35140/0 (socket says 172.21.15.3:35140) 2024-05-08T21:23:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21a6ffd700 1 -- 172.21.15.3:0/3115457757 learned_addr learned my addr 172.21.15.3:0/3115457757 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21a67fc700 1 --2- 172.21.15.3:0/3115457757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21a809a160 0x7f21a81f0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21a77fe700 1 --2- 172.21.15.3:0/3115457757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21a80cd450 0x7f21a815cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21a6ffd700 1 -- 172.21.15.3:0/3115457757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21a80cd450 msgr2=0x7f21a815cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21a6ffd700 1 --2- 172.21.15.3:0/3115457757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21a80cd450 0x7f21a815cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21a6ffd700 1 -- 172.21.15.3:0/3115457757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21a809a160 msgr2=0x7f21a81f0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21a6ffd700 1 --2- 172.21.15.3:0/3115457757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21a809a160 0x7f21a81f0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21a6ffd700 1 -- 172.21.15.3:0/3115457757 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a0009b30 con 0x7f21a806cbd0 2024-05-08T21:23:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21a67fc700 1 --2- 172.21.15.3:0/3115457757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21a809a160 0x7f21a81f0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21a6ffd700 1 --2- 172.21.15.3:0/3115457757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a806cbd0 0x7f21a80c9410 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f219800e870 tx=0x7f219800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f218f7fe700 1 -- 172.21.15.3:0/3115457757 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198009590 con 0x7f21a806cbd0 2024-05-08T21:23:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f218f7fe700 1 -- 172.21.15.3:0/3115457757 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219800ee70 con 0x7f21a806cbd0 2024-05-08T21:23:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21ad2b9700 1 -- 172.21.15.3:0/3115457757 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a81f7250 con 0x7f21a806cbd0 2024-05-08T21:23:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f218f7fe700 1 -- 172.21.15.3:0/3115457757 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219800ca50 con 0x7f21a806cbd0 2024-05-08T21:23:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.474+0000 7f21ad2b9700 1 -- 172.21.15.3:0/3115457757 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a81f7810 con 0x7f21a806cbd0 2024-05-08T21:23:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.475+0000 7f21ad2b9700 1 -- 172.21.15.3:0/3115457757 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a81f3840 con 0x7f21a806cbd0 2024-05-08T21:23:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.476+0000 7f218f7fe700 1 -- 172.21.15.3:0/3115457757 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f219800cbb0 con 0x7f21a806cbd0 2024-05-08T21:23:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.476+0000 7f218f7fe700 1 --2- 172.21.15.3:0/3115457757 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f219406ac90 0x7f219406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.476+0000 7f218f7fe700 1 -- 172.21.15.3:0/3115457757 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5896+0+0 (secure 0 0 0) 0x7f21980893a0 con 0x7f21a806cbd0 2024-05-08T21:23:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.477+0000 7f21a67fc700 1 --2- 172.21.15.3:0/3115457757 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f219406ac90 0x7f219406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:34.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.477+0000 7f21a67fc700 1 --2- 172.21.15.3:0/3115457757 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f219406ac90 0x7f219406d150 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f21a80b04e0 tx=0x7f219c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:34.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.481+0000 7f218f7fe700 1 -- 172.21.15.3:0/3115457757 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f21980544c0 con 0x7f21a806cbd0 2024-05-08T21:23:34.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:34.751+0000 7f21ad2b9700 1 -- 172.21.15.3:0/3115457757 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "recovery_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f21a80cd830 con 0x7f21a806cbd0 2024-05-08T21:23:35.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.319+0000 7f218f7fe700 1 -- 172.21.15.3:0/3115457757 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "recovery_fs", "yes_i_really_mean_it": true}]=0 v30) v1 ==== 109+0+0 (secure 0 0 0) 0x7f219800ce60 con 0x7f21a806cbd0 2024-05-08T21:23:35.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.322+0000 7f21ad2b9700 1 -- 172.21.15.3:0/3115457757 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f219406ac90 msgr2=0x7f219406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:35.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.322+0000 7f21ad2b9700 1 --2- 172.21.15.3:0/3115457757 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f219406ac90 0x7f219406d150 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f21a80b04e0 tx=0x7f219c009040 comp rx=0 tx=0).stop 2024-05-08T21:23:35.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.322+0000 7f21ad2b9700 1 -- 172.21.15.3:0/3115457757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a806cbd0 msgr2=0x7f21a80c9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:35.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.322+0000 7f21ad2b9700 1 --2- 172.21.15.3:0/3115457757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a806cbd0 0x7f21a80c9410 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f219800e870 tx=0x7f219800ec30 comp rx=0 tx=0).stop 2024-05-08T21:23:35.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.322+0000 7f21ad2b9700 1 -- 172.21.15.3:0/3115457757 shutdown_connections 2024-05-08T21:23:35.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.322+0000 7f21ad2b9700 1 --2- 172.21.15.3:0/3115457757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21a80cd450 0x7f21a815cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:35.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.322+0000 7f21ad2b9700 1 --2- 172.21.15.3:0/3115457757 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f219406ac90 0x7f219406d150 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:35.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.322+0000 7f21ad2b9700 1 --2- 172.21.15.3:0/3115457757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a806cbd0 0x7f21a80c9410 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:35.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.322+0000 7f21ad2b9700 1 --2- 172.21.15.3:0/3115457757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21a809a160 0x7f21a81f0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:35.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.322+0000 7f21ad2b9700 1 -- 172.21.15.3:0/3115457757 >> 172.21.15.3:0/3115457757 conn(0x7f21a80b2f90 msgr2=0x7f21a815ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:35.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.322+0000 7f21ad2b9700 1 -- 172.21.15.3:0/3115457757 shutdown_connections 2024-05-08T21:23:35.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.322+0000 7f21ad2b9700 1 -- 172.21.15.3:0/3115457757 wait complete. 2024-05-08T21:23:35.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm recovery_fs_metadata recovery_fs_metadata --yes-i-really-really-mean-it 2024-05-08T21:23:35.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.482+0000 7ff6fd046700 1 Processor -- start 2024-05-08T21:23:35.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.482+0000 7ff6fd046700 1 -- start start 2024-05-08T21:23:35.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.483+0000 7ff6fd046700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff6f815d8c0 0x7ff6f80ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:35.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.483+0000 7ff6fd046700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff6f80b5550 0x7ff6f81618f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:35.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.483+0000 7ff6fd046700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f8099ce0 con 0x7ff6f80b50d0 2024-05-08T21:23:35.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.483+0000 7ff6fd046700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f806fde0 con 0x7ff6f80b5550 2024-05-08T21:23:35.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.483+0000 7ff6fd046700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f806f260 con 0x7ff6f815d8c0 2024-05-08T21:23:35.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.483+0000 7ff6f759e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff6f80b5550 0x7ff6f81618f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:35.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.483+0000 7ff6f759e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff6f80b5550 0x7ff6f81618f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35150/0 (socket says 172.21.15.3:35150) 2024-05-08T21:23:35.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.483+0000 7ff6f759e700 1 -- 172.21.15.3:0/1971760477 learned_addr learned my addr 172.21.15.3:0/1971760477 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:35.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.483+0000 7ff6f6d9d700 1 --2- 172.21.15.3:0/1971760477 >> v2:172.21.15.186:3300/0 conn(0x7ff6f815d8c0 0x7ff6f80ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:35.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.483+0000 7ff6f759e700 1 -- 172.21.15.3:0/1971760477 >> v2:172.21.15.186:3300/0 conn(0x7ff6f815d8c0 msgr2=0x7ff6f80ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:35.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.483+0000 7ff6f759e700 1 --2- 172.21.15.3:0/1971760477 >> v2:172.21.15.186:3300/0 conn(0x7ff6f815d8c0 0x7ff6f80ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:35.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.483+0000 7ff6f759e700 1 -- 172.21.15.3:0/1971760477 >> v1:172.21.15.186:6789/0 conn(0x7ff6f80b50d0 legacy=0x7ff6f806a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:35.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.484+0000 7ff6f759e700 1 -- 172.21.15.3:0/1971760477 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f806f540 con 0x7ff6f80b5550 2024-05-08T21:23:35.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.484+0000 7ff6f759e700 1 --2- 172.21.15.3:0/1971760477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f80b5550 0x7ff6f81618f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff6f806d730 tx=0x7ff6f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76f65a307399a0f server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:35.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.484+0000 7ff6f5d9b700 1 -- 172.21.15.3:0/1971760477 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f000e070 con 0x7ff6f80b5550 2024-05-08T21:23:35.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.484+0000 7ff6f5d9b700 1 -- 172.21.15.3:0/1971760477 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f0003a20 con 0x7ff6f80b5550 2024-05-08T21:23:35.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.484+0000 7ff6f5d9b700 1 -- 172.21.15.3:0/1971760477 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0003dd0 con 0x7ff6f80b5550 2024-05-08T21:23:35.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.484+0000 7ff6fd046700 1 -- 172.21.15.3:0/1971760477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f80b5550 msgr2=0x7ff6f81618f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:35.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.484+0000 7ff6fd046700 1 --2- 172.21.15.3:0/1971760477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f80b5550 0x7ff6f81618f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff6f806d730 tx=0x7ff6f000b330 comp rx=0 tx=0).stop 2024-05-08T21:23:35.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.484+0000 7ff6fd046700 1 -- 172.21.15.3:0/1971760477 shutdown_connections 2024-05-08T21:23:35.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.484+0000 7ff6fd046700 1 --2- 172.21.15.3:0/1971760477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f80b5550 0x7ff6f81618f0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:35.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.484+0000 7ff6fd046700 1 --2- 172.21.15.3:0/1971760477 >> v2:172.21.15.186:3300/0 conn(0x7ff6f815d8c0 0x7ff6f80ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:35.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.484+0000 7ff6fd046700 1 -- 172.21.15.3:0/1971760477 >> 172.21.15.3:0/1971760477 conn(0x7ff6f80b5a30 msgr2=0x7ff6f80b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:35.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.485+0000 7ff6fd046700 1 -- 172.21.15.3:0/1971760477 shutdown_connections 2024-05-08T21:23:35.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.485+0000 7ff6fd046700 1 -- 172.21.15.3:0/1971760477 wait complete. 2024-05-08T21:23:35.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.486+0000 7ff6fd046700 1 Processor -- start 2024-05-08T21:23:35.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.486+0000 7ff6fd046700 1 -- start start 2024-05-08T21:23:35.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.487+0000 7ff6fd046700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f80b50d0 0x7ff6f81f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:35.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.487+0000 7ff6fd046700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6f80b5550 0x7ff6f815f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:35.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.487+0000 7ff6fd046700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6f815d8c0 0x7ff6f81f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:35.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.487+0000 7ff6fd046700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f81f4ab0 con 0x7ff6f80b50d0 2024-05-08T21:23:35.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.487+0000 7ff6fd046700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f81f7fe0 con 0x7ff6f815d8c0 2024-05-08T21:23:35.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.487+0000 7ff6fd046700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f81f44f0 con 0x7ff6f80b5550 2024-05-08T21:23:35.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.487+0000 7ff6f6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f80b50d0 0x7ff6f81f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:35.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.487+0000 7ff6f6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f80b50d0 0x7ff6f81f7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35166/0 (socket says 172.21.15.3:35166) 2024-05-08T21:23:35.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.487+0000 7ff6f6d9d700 1 -- 172.21.15.3:0/2632342829 learned_addr learned my addr 172.21.15.3:0/2632342829 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:35.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.487+0000 7ff6f759e700 1 --2- 172.21.15.3:0/2632342829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6f815d8c0 0x7ff6f81f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:35.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.487+0000 7ff6f659c700 1 --2- 172.21.15.3:0/2632342829 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6f80b5550 0x7ff6f815f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:35.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.488+0000 7ff6f6d9d700 1 -- 172.21.15.3:0/2632342829 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6f80b5550 msgr2=0x7ff6f815f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:35.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.488+0000 7ff6f6d9d700 1 --2- 172.21.15.3:0/2632342829 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6f80b5550 0x7ff6f815f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:35.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.488+0000 7ff6f6d9d700 1 -- 172.21.15.3:0/2632342829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6f815d8c0 msgr2=0x7ff6f81f3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:35.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.488+0000 7ff6f6d9d700 1 --2- 172.21.15.3:0/2632342829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6f815d8c0 0x7ff6f81f3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:35.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.488+0000 7ff6f6d9d700 1 -- 172.21.15.3:0/2632342829 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f0009b30 con 0x7ff6f80b50d0 2024-05-08T21:23:35.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.488+0000 7ff6f6d9d700 1 --2- 172.21.15.3:0/2632342829 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f80b50d0 0x7ff6f81f7aa0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff6e800d800 tx=0x7ff6e800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:35.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.488+0000 7ff6df7fe700 1 -- 172.21.15.3:0/2632342829 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e80125f0 con 0x7ff6f80b50d0 2024-05-08T21:23:35.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.488+0000 7ff6df7fe700 1 -- 172.21.15.3:0/2632342829 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e8012750 con 0x7ff6f80b50d0 2024-05-08T21:23:35.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.488+0000 7ff6df7fe700 1 -- 172.21.15.3:0/2632342829 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8012b00 con 0x7ff6f80b50d0 2024-05-08T21:23:35.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.488+0000 7ff6fd046700 1 -- 172.21.15.3:0/2632342829 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f81f47d0 con 0x7ff6f80b50d0 2024-05-08T21:23:35.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.488+0000 7ff6fd046700 1 -- 172.21.15.3:0/2632342829 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f81f7580 con 0x7ff6f80b50d0 2024-05-08T21:23:35.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.489+0000 7ff6fd046700 1 -- 172.21.15.3:0/2632342829 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f8074bc0 con 0x7ff6f80b50d0 2024-05-08T21:23:35.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.490+0000 7ff6df7fe700 1 -- 172.21.15.3:0/2632342829 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff6e8012c60 con 0x7ff6f80b50d0 2024-05-08T21:23:35.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.490+0000 7ff6df7fe700 1 --2- 172.21.15.3:0/2632342829 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6e406ac90 0x7ff6e406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:35.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.490+0000 7ff6df7fe700 1 -- 172.21.15.3:0/2632342829 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5896+0+0 (secure 0 0 0) 0x7ff6e8089670 con 0x7ff6f80b50d0 2024-05-08T21:23:35.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.490+0000 7ff6f659c700 1 --2- 172.21.15.3:0/2632342829 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6e406ac90 0x7ff6e406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:35.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.491+0000 7ff6f659c700 1 --2- 172.21.15.3:0/2632342829 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6e406ac90 0x7ff6e406d150 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80b2dc0 tx=0x7ff6ec009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:35.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.495+0000 7ff6df7fe700 1 -- 172.21.15.3:0/2632342829 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff6e8054820 con 0x7ff6f80b50d0 2024-05-08T21:23:35.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:35.713+0000 7ff6fd046700 1 -- 172.21.15.3:0/2632342829 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "recovery_fs_metadata", "pool2": "recovery_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff6f806b640 con 0x7ff6f80b50d0 2024-05-08T21:23:36.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.321+0000 7ff6df7fe700 1 -- 172.21.15.3:0/2632342829 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "recovery_fs_metadata", "pool2": "recovery_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'recovery_fs_metadata' removed v45) v1 ==== 196+0+0 (secure 0 0 0) 0x7ff6e8057e40 con 0x7ff6f80b50d0 2024-05-08T21:23:36.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.325+0000 7ff6fd046700 1 -- 172.21.15.3:0/2632342829 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6e406ac90 msgr2=0x7ff6e406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:36.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.325+0000 7ff6fd046700 1 --2- 172.21.15.3:0/2632342829 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6e406ac90 0x7ff6e406d150 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80b2dc0 tx=0x7ff6ec009040 comp rx=0 tx=0).stop 2024-05-08T21:23:36.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.325+0000 7ff6fd046700 1 -- 172.21.15.3:0/2632342829 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f80b50d0 msgr2=0x7ff6f81f7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:36.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.325+0000 7ff6fd046700 1 --2- 172.21.15.3:0/2632342829 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f80b50d0 0x7ff6f81f7aa0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff6e800d800 tx=0x7ff6e800db10 comp rx=0 tx=0).stop 2024-05-08T21:23:36.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.325+0000 7ff6fd046700 1 -- 172.21.15.3:0/2632342829 shutdown_connections 2024-05-08T21:23:36.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.325+0000 7ff6fd046700 1 --2- 172.21.15.3:0/2632342829 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7ff6e406ac90 0x7ff6e406d150 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:36.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.325+0000 7ff6fd046700 1 --2- 172.21.15.3:0/2632342829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6f815d8c0 0x7ff6f81f3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:36.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.325+0000 7ff6fd046700 1 --2- 172.21.15.3:0/2632342829 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f80b50d0 0x7ff6f81f7aa0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:36.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.325+0000 7ff6fd046700 1 --2- 172.21.15.3:0/2632342829 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6f80b5550 0x7ff6f815f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:36.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.325+0000 7ff6fd046700 1 -- 172.21.15.3:0/2632342829 >> 172.21.15.3:0/2632342829 conn(0x7ff6f80b5a30 msgr2=0x7ff6f806dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:36.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.325+0000 7ff6fd046700 1 -- 172.21.15.3:0/2632342829 shutdown_connections 2024-05-08T21:23:36.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.325+0000 7ff6fd046700 1 -- 172.21.15.3:0/2632342829 wait complete. 2024-05-08T21:23:36.328 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'recovery_fs_metadata' removed 2024-05-08T21:23:36.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-08T21:23:36.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.486+0000 7f12cd143700 1 Processor -- start 2024-05-08T21:23:36.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.486+0000 7f12cd143700 1 -- start start 2024-05-08T21:23:36.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.487+0000 7f12cd143700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f12c815b4e0 0x7f12c80b1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:36.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.487+0000 7f12cd143700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12c80d1910 0x7f12c80cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:36.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.487+0000 7f12cd143700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c80cf560 con 0x7f12c815b8c0 2024-05-08T21:23:36.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.487+0000 7f12cd143700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c80cefe0 con 0x7f12c815b4e0 2024-05-08T21:23:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.487+0000 7f12cd143700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c80ccff0 con 0x7f12c80d1910 2024-05-08T21:23:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.487+0000 7f12c659c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f12c815b8c0 0x7f12c80d1200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:36396/0 (socket says 172.21.15.3:36396) 2024-05-08T21:23:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.487+0000 7f12c6d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f12c815b4e0 0x7f12c80b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.487+0000 7f12c659c700 1 -- 172.21.15.3:0/3192325464 learned_addr learned my addr 172.21.15.3:0/3192325464 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.488+0000 7f12c5d9b700 1 -- 172.21.15.3:0/3192325464 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301458087 0 0) 0x7f12c80cf560 con 0x7f12c815b8c0 2024-05-08T21:23:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.488+0000 7f12c759e700 1 --2- 172.21.15.3:0/3192325464 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12c80d1910 0x7f12c80cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.488+0000 7f12c5d9b700 1 -- 172.21.15.3:0/3192325464 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b00035b0 con 0x7f12c815b8c0 2024-05-08T21:23:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.488+0000 7f12c6d9d700 1 -- 172.21.15.3:0/3192325464 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12c80d1910 msgr2=0x7f12c80cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.488+0000 7f12c6d9d700 1 --2- 172.21.15.3:0/3192325464 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12c80d1910 0x7f12c80cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.488+0000 7f12c6d9d700 1 -- 172.21.15.3:0/3192325464 >> v1:172.21.15.3:6789/0 conn(0x7f12c815b8c0 legacy=0x7f12c80d1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.488+0000 7f12c6d9d700 1 -- 172.21.15.3:0/3192325464 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c80cd2d0 con 0x7f12c815b4e0 2024-05-08T21:23:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.488+0000 7f12c6d9d700 1 --2- 172.21.15.3:0/3192325464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c815b4e0 0x7f12c80b1200 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009840 tx=0x7f12b8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554f19b3230564d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.488+0000 7f12c5d9b700 1 -- 172.21.15.3:0/3192325464 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b8011830 con 0x7f12c815b4e0 2024-05-08T21:23:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.488+0000 7f12c5d9b700 1 -- 172.21.15.3:0/3192325464 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b8011990 con 0x7f12c815b4e0 2024-05-08T21:23:36.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.488+0000 7f12c5d9b700 1 -- 172.21.15.3:0/3192325464 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b8011d20 con 0x7f12c815b4e0 2024-05-08T21:23:36.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.489+0000 7f12cd143700 1 -- 172.21.15.3:0/3192325464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c815b4e0 msgr2=0x7f12c80b1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:36.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.489+0000 7f12cd143700 1 --2- 172.21.15.3:0/3192325464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c815b4e0 0x7f12c80b1200 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009840 tx=0x7f12b8009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:36.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.489+0000 7f12cd143700 1 -- 172.21.15.3:0/3192325464 shutdown_connections 2024-05-08T21:23:36.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.489+0000 7f12cd143700 1 --2- 172.21.15.3:0/3192325464 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12c80d1910 0x7f12c80cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:36.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.489+0000 7f12cd143700 1 --2- 172.21.15.3:0/3192325464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c815b4e0 0x7f12c80b1200 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:36.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.489+0000 7f12cd143700 1 -- 172.21.15.3:0/3192325464 >> 172.21.15.3:0/3192325464 conn(0x7f12c808afe0 msgr2=0x7f12c808b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:36.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.489+0000 7f12cd143700 1 -- 172.21.15.3:0/3192325464 shutdown_connections 2024-05-08T21:23:36.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.489+0000 7f12cd143700 1 -- 172.21.15.3:0/3192325464 wait complete. 2024-05-08T21:23:36.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.490+0000 7f12cd143700 1 Processor -- start 2024-05-08T21:23:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.490+0000 7f12cd143700 1 -- start start 2024-05-08T21:23:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.491+0000 7f12cd143700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c80d1910 0x7f12c806c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.491+0000 7f12cd143700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12c815b4e0 0x7f12c81f2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.491+0000 7f12cd143700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12c815b8c0 0x7f12c80b5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.491+0000 7f12cd143700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c806c930 con 0x7f12c80d1910 2024-05-08T21:23:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.491+0000 7f12cd143700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c806cb90 con 0x7f12c815b8c0 2024-05-08T21:23:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.491+0000 7f12cd143700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c806ce70 con 0x7f12c815b4e0 2024-05-08T21:23:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.491+0000 7f12c6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c80d1910 0x7f12c806c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.491+0000 7f12c6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c80d1910 0x7f12c806c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35192/0 (socket says 172.21.15.3:35192) 2024-05-08T21:23:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.491+0000 7f12c6d9d700 1 -- 172.21.15.3:0/3623757956 learned_addr learned my addr 172.21.15.3:0/3623757956 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.492+0000 7f12c659c700 1 --2- 172.21.15.3:0/3623757956 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12c815b4e0 0x7f12c81f2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.492+0000 7f12c759e700 1 --2- 172.21.15.3:0/3623757956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12c815b8c0 0x7f12c80b5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.492+0000 7f12c6d9d700 1 -- 172.21.15.3:0/3623757956 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12c815b4e0 msgr2=0x7f12c81f2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.492+0000 7f12c6d9d700 1 --2- 172.21.15.3:0/3623757956 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12c815b4e0 0x7f12c81f2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.492+0000 7f12c6d9d700 1 -- 172.21.15.3:0/3623757956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12c815b8c0 msgr2=0x7f12c80b5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.492+0000 7f12c6d9d700 1 --2- 172.21.15.3:0/3623757956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12c815b8c0 0x7f12c80b5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.492+0000 7f12c6d9d700 1 -- 172.21.15.3:0/3623757956 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b80094a0 con 0x7f12c80d1910 2024-05-08T21:23:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.492+0000 7f12c759e700 1 --2- 172.21.15.3:0/3623757956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12c815b8c0 0x7f12c80b5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.492+0000 7f12c6d9d700 1 --2- 172.21.15.3:0/3623757956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c80d1910 0x7f12c806c3f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009fd0 tx=0x7f12b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.492+0000 7f12af7fe700 1 -- 172.21.15.3:0/3623757956 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b80036a0 con 0x7f12c80d1910 2024-05-08T21:23:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.492+0000 7f12af7fe700 1 -- 172.21.15.3:0/3623757956 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b801f070 con 0x7f12c80d1910 2024-05-08T21:23:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.492+0000 7f12cd143700 1 -- 172.21.15.3:0/3623757956 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c80c9660 con 0x7f12c80d1910 2024-05-08T21:23:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.492+0000 7f12af7fe700 1 -- 172.21.15.3:0/3623757956 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b8003a60 con 0x7f12c80d1910 2024-05-08T21:23:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.492+0000 7f12cd143700 1 -- 172.21.15.3:0/3623757956 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c806d150 con 0x7f12c80d1910 2024-05-08T21:23:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.493+0000 7f12cd143700 1 -- 172.21.15.3:0/3623757956 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c8074bc0 con 0x7f12c80d1910 2024-05-08T21:23:36.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.494+0000 7f12af7fe700 1 -- 172.21.15.3:0/3623757956 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f12b8003bc0 con 0x7f12c80d1910 2024-05-08T21:23:36.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.494+0000 7f12af7fe700 1 --2- 172.21.15.3:0/3623757956 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f12b406ac90 0x7f12b406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:36.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.494+0000 7f12af7fe700 1 -- 172.21.15.3:0/3623757956 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5448+0+0 (secure 0 0 0) 0x7f12b8021070 con 0x7f12c80d1910 2024-05-08T21:23:36.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.494+0000 7f12c659c700 1 --2- 172.21.15.3:0/3623757956 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f12b406ac90 0x7f12b406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:36.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.495+0000 7f12c659c700 1 --2- 172.21.15.3:0/3623757956 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f12b406ac90 0x7f12b406d150 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f12c8157060 tx=0x7f12bc007dc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.499+0000 7f12af7fe700 1 -- 172.21.15.3:0/3623757956 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f12b8055e30 con 0x7f12c80d1910 2024-05-08T21:23:36.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:36.713+0000 7f12cd143700 1 -- 172.21.15.3:0/3623757956 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f12c80b0fe0 con 0x7f12c80d1910 2024-05-08T21:23:37.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.327+0000 7f12af7fe700 1 -- 172.21.15.3:0/3623757956 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v46) v1 ==== 169+0+0 (secure 0 0 0) 0x7f12b802b0d0 con 0x7f12c80d1910 2024-05-08T21:23:37.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.331+0000 7f12cd143700 1 -- 172.21.15.3:0/3623757956 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f12b406ac90 msgr2=0x7f12b406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:37.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.332+0000 7f12cd143700 1 --2- 172.21.15.3:0/3623757956 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f12b406ac90 0x7f12b406d150 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f12c8157060 tx=0x7f12bc007dc0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.332+0000 7f12cd143700 1 -- 172.21.15.3:0/3623757956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c80d1910 msgr2=0x7f12c806c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:37.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.332+0000 7f12cd143700 1 --2- 172.21.15.3:0/3623757956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c80d1910 0x7f12c806c3f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009fd0 tx=0x7f12b8004750 comp rx=0 tx=0).stop 2024-05-08T21:23:37.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.332+0000 7f12cd143700 1 -- 172.21.15.3:0/3623757956 shutdown_connections 2024-05-08T21:23:37.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.332+0000 7f12cd143700 1 --2- 172.21.15.3:0/3623757956 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f12b406ac90 0x7f12b406d150 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.332+0000 7f12cd143700 1 --2- 172.21.15.3:0/3623757956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12c815b8c0 0x7f12c80b5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.332+0000 7f12cd143700 1 --2- 172.21.15.3:0/3623757956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c80d1910 0x7f12c806c3f0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.332+0000 7f12cd143700 1 --2- 172.21.15.3:0/3623757956 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12c815b4e0 0x7f12c81f2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.332+0000 7f12cd143700 1 -- 172.21.15.3:0/3623757956 >> 172.21.15.3:0/3623757956 conn(0x7f12c808afe0 msgr2=0x7f12c815fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:37.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.332+0000 7f12cd143700 1 -- 172.21.15.3:0/3623757956 shutdown_connections 2024-05-08T21:23:37.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.332+0000 7f12cd143700 1 -- 172.21.15.3:0/3623757956 wait complete. 2024-05-08T21:23:37.335 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2024-05-08T21:23:37.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds debug_mds 2024-05-08T21:23:37.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.496+0000 7f8784707700 1 Processor -- start 2024-05-08T21:23:37.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.496+0000 7f8784707700 1 -- start start 2024-05-08T21:23:37.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.497+0000 7f8784707700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f877c15d880 0x7f877c0af990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:37.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.497+0000 7f8784707700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f877c0b3870 con 0x7f877c06f8c0 2024-05-08T21:23:37.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.497+0000 7f8784707700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f877c0b4ee0 con 0x7f877c06e7e0 2024-05-08T21:23:37.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.497+0000 7f8784707700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f877c0b32f0 con 0x7f877c15d880 2024-05-08T21:23:37.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.497+0000 7f87824a3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f877c15d880 0x7f877c0af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:37.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.497+0000 7f8781ca2700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f877c06f8c0 0x7f877c0ad260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:36398/0 (socket says 172.21.15.3:36398) 2024-05-08T21:23:37.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.497+0000 7f8781ca2700 1 -- 172.21.15.3:0/1300448641 learned_addr learned my addr 172.21.15.3:0/1300448641 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:37.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.497+0000 7f87824a3700 1 -- 172.21.15.3:0/1300448641 >> v1:172.21.15.186:6789/0 conn(0x7f877c06e7e0 legacy=0x7f877c06a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:37.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.497+0000 7f87824a3700 1 -- 172.21.15.3:0/1300448641 >> v1:172.21.15.3:6789/0 conn(0x7f877c06f8c0 legacy=0x7f877c0ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:37.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.497+0000 7f87814a1700 1 -- 172.21.15.3:0/1300448641 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383111256 0 0) 0x7f877c0b3870 con 0x7f877c06f8c0 2024-05-08T21:23:37.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.497+0000 7f87824a3700 1 -- 172.21.15.3:0/1300448641 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877c06fd50 con 0x7f877c15d880 2024-05-08T21:23:37.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.498+0000 7f87824a3700 1 --2- 172.21.15.3:0/1300448641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c15d880 0x7f877c0af990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8778009840 tx=0x7f8778009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37e8475a060854 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:37.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.498+0000 7f87814a1700 1 -- 172.21.15.3:0/1300448641 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8778004230 con 0x7f877c15d880 2024-05-08T21:23:37.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.498+0000 7f87814a1700 1 -- 172.21.15.3:0/1300448641 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8778004390 con 0x7f877c15d880 2024-05-08T21:23:37.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.498+0000 7f87814a1700 1 -- 172.21.15.3:0/1300448641 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87780036a0 con 0x7f877c15d880 2024-05-08T21:23:37.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.498+0000 7f8784707700 1 -- 172.21.15.3:0/1300448641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c15d880 msgr2=0x7f877c0af990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:37.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.498+0000 7f8784707700 1 --2- 172.21.15.3:0/1300448641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c15d880 0x7f877c0af990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8778009840 tx=0x7f8778009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:37.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.498+0000 7f8784707700 1 -- 172.21.15.3:0/1300448641 shutdown_connections 2024-05-08T21:23:37.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.498+0000 7f8784707700 1 --2- 172.21.15.3:0/1300448641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c15d880 0x7f877c0af990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.498+0000 7f8784707700 1 -- 172.21.15.3:0/1300448641 >> 172.21.15.3:0/1300448641 conn(0x7f877c0b5810 msgr2=0x7f877c0b5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:37.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.499+0000 7f8784707700 1 -- 172.21.15.3:0/1300448641 shutdown_connections 2024-05-08T21:23:37.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.499+0000 7f8784707700 1 -- 172.21.15.3:0/1300448641 wait complete. 2024-05-08T21:23:37.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.500+0000 7f8784707700 1 Processor -- start 2024-05-08T21:23:37.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.500+0000 7f8784707700 1 -- start start 2024-05-08T21:23:37.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.501+0000 7f8784707700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f877c06e7e0 0x7f877c09bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:37.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.501+0000 7f8784707700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c06f8c0 0x7f877c14fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:37.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.501+0000 7f8784707700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877c15d880 0x7f877c0ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:37.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.501+0000 7f8784707700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c09c5e0 con 0x7f877c06f8c0 2024-05-08T21:23:37.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.501+0000 7f8784707700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c0c9dd0 con 0x7f877c15d880 2024-05-08T21:23:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.501+0000 7f8784707700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f877c09d590 con 0x7f877c06e7e0 2024-05-08T21:23:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.501+0000 7f8781ca2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c06f8c0 0x7f877c14fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.501+0000 7f8781ca2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c06f8c0 0x7f877c14fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35208/0 (socket says 172.21.15.3:35208) 2024-05-08T21:23:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.501+0000 7f8781ca2700 1 -- 172.21.15.3:0/1771340884 learned_addr learned my addr 172.21.15.3:0/1771340884 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.502+0000 7f87824a3700 1 --2- 172.21.15.3:0/1771340884 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f877c06e7e0 0x7f877c09bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.502+0000 7f8782ca4700 1 --2- 172.21.15.3:0/1771340884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877c15d880 0x7f877c0ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.502+0000 7f8781ca2700 1 -- 172.21.15.3:0/1771340884 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f877c06e7e0 msgr2=0x7f877c09bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.502+0000 7f8781ca2700 1 --2- 172.21.15.3:0/1771340884 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f877c06e7e0 0x7f877c09bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.502+0000 7f8781ca2700 1 -- 172.21.15.3:0/1771340884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877c15d880 msgr2=0x7f877c0ca060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.502+0000 7f8781ca2700 1 --2- 172.21.15.3:0/1771340884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877c15d880 0x7f877c0ca060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.502+0000 7f8781ca2700 1 -- 172.21.15.3:0/1771340884 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87780094a0 con 0x7f877c06f8c0 2024-05-08T21:23:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.502+0000 7f8781ca2700 1 --2- 172.21.15.3:0/1771340884 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c06f8c0 0x7f877c14fed0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f876c00c520 tx=0x7f876c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.502+0000 7f8772ffd700 1 -- 172.21.15.3:0/1771340884 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876c004c80 con 0x7f877c06f8c0 2024-05-08T21:23:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.502+0000 7f8772ffd700 1 -- 172.21.15.3:0/1771340884 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f876c004e00 con 0x7f877c06f8c0 2024-05-08T21:23:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.502+0000 7f8784707700 1 -- 172.21.15.3:0/1771340884 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877c0ca5a0 con 0x7f877c06f8c0 2024-05-08T21:23:37.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.502+0000 7f8772ffd700 1 -- 172.21.15.3:0/1771340884 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876c007310 con 0x7f877c06f8c0 2024-05-08T21:23:37.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.502+0000 7f8784707700 1 -- 172.21.15.3:0/1771340884 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877c0c9af0 con 0x7f877c06f8c0 2024-05-08T21:23:37.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.503+0000 7f8784707700 1 -- 172.21.15.3:0/1771340884 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f877c09d360 con 0x7f877c06f8c0 2024-05-08T21:23:37.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.504+0000 7f8772ffd700 1 -- 172.21.15.3:0/1771340884 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f876c007470 con 0x7f877c06f8c0 2024-05-08T21:23:37.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.504+0000 7f8772ffd700 1 --2- 172.21.15.3:0/1771340884 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f876006ac90 0x7f876006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:37.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.504+0000 7f8772ffd700 1 -- 172.21.15.3:0/1771340884 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f876c088d40 con 0x7f877c06f8c0 2024-05-08T21:23:37.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.504+0000 7f87824a3700 1 --2- 172.21.15.3:0/1771340884 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f876006ac90 0x7f876006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:37.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.505+0000 7f87824a3700 1 --2- 172.21.15.3:0/1771340884 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f876006ac90 0x7f876006d150 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f87780090b0 tx=0x7f8778008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:37.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.509+0000 7f8772ffd700 1 -- 172.21.15.3:0/1771340884 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f876c054250 con 0x7f877c06f8c0 2024-05-08T21:23:37.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.728+0000 7f8784707700 1 -- 172.21.15.3:0/1771340884 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "debug_mds"} v 0) v1 -- 0x7f877c074bc0 con 0x7f877c06f8c0 2024-05-08T21:23:37.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.732+0000 7f8772ffd700 1 -- 172.21.15.3:0/1771340884 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "debug_mds"}]=0 v12) v1 ==== 92+0+0 (secure 0 0 0) 0x7f876c057870 con 0x7f877c06f8c0 2024-05-08T21:23:37.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.735+0000 7f8784707700 1 -- 172.21.15.3:0/1771340884 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f876006ac90 msgr2=0x7f876006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:37.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.735+0000 7f8784707700 1 --2- 172.21.15.3:0/1771340884 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f876006ac90 0x7f876006d150 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f87780090b0 tx=0x7f8778008030 comp rx=0 tx=0).stop 2024-05-08T21:23:37.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.735+0000 7f8784707700 1 -- 172.21.15.3:0/1771340884 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c06f8c0 msgr2=0x7f877c14fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:37.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.735+0000 7f8784707700 1 --2- 172.21.15.3:0/1771340884 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c06f8c0 0x7f877c14fed0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f876c00c520 tx=0x7f876c00c8e0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.735+0000 7f8784707700 1 -- 172.21.15.3:0/1771340884 shutdown_connections 2024-05-08T21:23:37.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.735+0000 7f8784707700 1 --2- 172.21.15.3:0/1771340884 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f877c06e7e0 0x7f877c09bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.735+0000 7f8784707700 1 --2- 172.21.15.3:0/1771340884 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f876006ac90 0x7f876006d150 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.735+0000 7f8784707700 1 --2- 172.21.15.3:0/1771340884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877c15d880 0x7f877c0ca060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.735+0000 7f8784707700 1 --2- 172.21.15.3:0/1771340884 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c06f8c0 0x7f877c14fed0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.735+0000 7f8784707700 1 -- 172.21.15.3:0/1771340884 >> 172.21.15.3:0/1771340884 conn(0x7f877c0b5810 msgr2=0x7f877c15ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:37.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.735+0000 7f8784707700 1 -- 172.21.15.3:0/1771340884 shutdown_connections 2024-05-08T21:23:37.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.735+0000 7f8784707700 1 -- 172.21.15.3:0/1771340884 wait complete. 2024-05-08T21:23:37.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-08T21:23:37.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.888+0000 7f3eeb124700 1 Processor -- start 2024-05-08T21:23:37.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.888+0000 7f3eeb124700 1 -- start start 2024-05-08T21:23:37.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.889+0000 7f3eeb124700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ee40af990 0x7f3ee406f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:37.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.889+0000 7f3eeb124700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3ee406e7e0 0x7f3ee406ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.889+0000 7f3eeb124700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee40b3870 con 0x7f3ee415d880 2024-05-08T21:23:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.889+0000 7f3eeb124700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee406fe00 con 0x7f3ee406e7e0 2024-05-08T21:23:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.889+0000 7f3eeb124700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee406f130 con 0x7f3ee40af990 2024-05-08T21:23:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.889+0000 7f3ee8ec0700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3ee415d880 0x7f3ee40ad260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:36400/0 (socket says 172.21.15.3:36400) 2024-05-08T21:23:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.889+0000 7f3ee8ec0700 1 -- 172.21.15.3:0/2568622445 learned_addr learned my addr 172.21.15.3:0/2568622445 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.889+0000 7f3ee96c1700 1 --2- 172.21.15.3:0/2568622445 >> v2:172.21.15.186:3300/0 conn(0x7f3ee406e7e0 0x7f3ee406ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.889+0000 7f3ee3fff700 1 --2- 172.21.15.3:0/2568622445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ee40af990 0x7f3ee406f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.889+0000 7f3ee37fe700 1 -- 172.21.15.3:0/2568622445 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465248098 0 0) 0x7f3ee40b3870 con 0x7f3ee415d880 2024-05-08T21:23:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.889+0000 7f3ee37fe700 1 -- 172.21.15.3:0/2568622445 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed40035b0 con 0x7f3ee415d880 2024-05-08T21:23:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee37fe700 1 -- 172.21.15.3:0/2568622445 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3633551641 0 0) 0x7f3ed40035b0 con 0x7f3ee415d880 2024-05-08T21:23:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee37fe700 1 -- 172.21.15.3:0/2568622445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ee40af990 msgr2=0x7f3ee406f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee37fe700 1 --2- 172.21.15.3:0/2568622445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ee40af990 0x7f3ee406f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee37fe700 1 -- 172.21.15.3:0/2568622445 >> v2:172.21.15.186:3300/0 conn(0x7f3ee406e7e0 msgr2=0x7f3ee406ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee37fe700 1 --2- 172.21.15.3:0/2568622445 >> v2:172.21.15.186:3300/0 conn(0x7f3ee406e7e0 0x7f3ee406ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee37fe700 1 -- 172.21.15.3:0/2568622445 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee40b4ee0 con 0x7f3ee415d880 2024-05-08T21:23:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee37fe700 1 -- 172.21.15.3:0/2568622445 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f3ed8001670 con 0x7f3ee415d880 2024-05-08T21:23:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee96c1700 1 --2- 172.21.15.3:0/2568622445 >> v2:172.21.15.186:3300/0 conn(0x7f3ee406e7e0 0x7f3ee406ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee37fe700 1 -- 172.21.15.3:0/2568622445 >> v1:172.21.15.3:6789/0 conn(0x7f3ee415d880 legacy=0x7f3ee40ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee37fe700 1 --2- 172.21.15.3:0/2568622445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ed4003a00 0x7f3ed4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee37fe700 1 --2- 172.21.15.3:0/2568622445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ed4004850 0x7f3ed4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee37fe700 1 --2- 172.21.15.3:0/2568622445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ed4005210 0x7f3ed400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee37fe700 1 -- 172.21.15.3:0/2568622445 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee406f130 con 0x7f3ed4005210 2024-05-08T21:23:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee37fe700 1 -- 172.21.15.3:0/2568622445 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed400bf50 con 0x7f3ed4003a00 2024-05-08T21:23:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee37fe700 1 -- 172.21.15.3:0/2568622445 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed400c090 con 0x7f3ed4004850 2024-05-08T21:23:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee3fff700 1 --2- 172.21.15.3:0/2568622445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ee40af990 0x7f3ee406f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee96c1700 1 --2- 172.21.15.3:0/2568622445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ed4005210 0x7f3ed400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee8ec0700 1 --2- 172.21.15.3:0/2568622445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ed4003a00 0x7f3ed4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee3fff700 1 --2- 172.21.15.3:0/2568622445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ed4004850 0x7f3ed4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee96c1700 1 -- 172.21.15.3:0/2568622445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ed4004850 msgr2=0x7f3ed4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee96c1700 1 --2- 172.21.15.3:0/2568622445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ed4004850 0x7f3ed4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee96c1700 1 -- 172.21.15.3:0/2568622445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ed4003a00 msgr2=0x7f3ed4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee96c1700 1 --2- 172.21.15.3:0/2568622445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ed4003a00 0x7f3ed4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.890+0000 7f3ee96c1700 1 -- 172.21.15.3:0/2568622445 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed40035b0 con 0x7f3ed4005210 2024-05-08T21:23:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.891+0000 7f3ee96c1700 1 --2- 172.21.15.3:0/2568622445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ed4005210 0x7f3ed400ba10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00e380 tx=0x7f3ed4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b743d844b3ca2c server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.891+0000 7f3ee37fe700 1 -- 172.21.15.3:0/2568622445 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc00ee20 con 0x7f3ed4005210 2024-05-08T21:23:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.891+0000 7f3ee37fe700 1 -- 172.21.15.3:0/2568622445 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3edc00f040 con 0x7f3ed4005210 2024-05-08T21:23:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.891+0000 7f3ee37fe700 1 -- 172.21.15.3:0/2568622445 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc0146a0 con 0x7f3ed4005210 2024-05-08T21:23:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.891+0000 7f3eeb124700 1 -- 172.21.15.3:0/2568622445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ed4005210 msgr2=0x7f3ed400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.891+0000 7f3eeb124700 1 --2- 172.21.15.3:0/2568622445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ed4005210 0x7f3ed400ba10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00e380 tx=0x7f3ed4005880 comp rx=0 tx=0).stop 2024-05-08T21:23:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.891+0000 7f3ee96c1700 1 -- 172.21.15.3:0/2568622445 reap_dead start 2024-05-08T21:23:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.891+0000 7f3eeb124700 1 -- 172.21.15.3:0/2568622445 shutdown_connections 2024-05-08T21:23:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.891+0000 7f3eeb124700 1 -- 172.21.15.3:0/2568622445 >> 172.21.15.3:0/2568622445 conn(0x7f3ee40b5810 msgr2=0x7f3ee40b5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:37.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.891+0000 7f3eeb124700 1 -- 172.21.15.3:0/2568622445 shutdown_connections 2024-05-08T21:23:37.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.891+0000 7f3eeb124700 1 -- 172.21.15.3:0/2568622445 wait complete. 2024-05-08T21:23:37.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.893+0000 7f3eeb124700 1 Processor -- start 2024-05-08T21:23:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.893+0000 7f3eeb124700 1 -- start start 2024-05-08T21:23:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3eeb124700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ed4005210 0x7f3ee41548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3eeb124700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ee40b1800 0x7f3ee409c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3eeb124700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee41574a0 0x7f3ee409cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3eeb124700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee40b1c90 con 0x7f3ee41574a0 2024-05-08T21:23:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3eeb124700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee40b1f70 con 0x7f3ed4005210 2024-05-08T21:23:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3eeb124700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee40b2250 con 0x7f3ee40b1800 2024-05-08T21:23:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3ee96c1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee41574a0 0x7f3ee409cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3ee96c1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee41574a0 0x7f3ee409cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35236/0 (socket says 172.21.15.3:35236) 2024-05-08T21:23:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3ee96c1700 1 -- 172.21.15.3:0/1755652035 learned_addr learned my addr 172.21.15.3:0/1755652035 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3ee3fff700 1 --2- 172.21.15.3:0/1755652035 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ee40b1800 0x7f3ee409c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3ee8ec0700 1 --2- 172.21.15.3:0/1755652035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ed4005210 0x7f3ee41548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3ee96c1700 1 -- 172.21.15.3:0/1755652035 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ee40b1800 msgr2=0x7f3ee409c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3ee96c1700 1 --2- 172.21.15.3:0/1755652035 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ee40b1800 0x7f3ee409c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3ee96c1700 1 -- 172.21.15.3:0/1755652035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ed4005210 msgr2=0x7f3ee41548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3ee96c1700 1 --2- 172.21.15.3:0/1755652035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ed4005210 0x7f3ee41548a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3ee96c1700 1 -- 172.21.15.3:0/1755652035 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee40b2530 con 0x7f3ee41574a0 2024-05-08T21:23:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3ee96c1700 1 --2- 172.21.15.3:0/1755652035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee41574a0 0x7f3ee409cb00 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f3edc002230 tx=0x7f3edc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.894+0000 7f3ee17fa700 1 -- 172.21.15.3:0/1755652035 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc004450 con 0x7f3ee41574a0 2024-05-08T21:23:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.895+0000 7f3ee17fa700 1 -- 172.21.15.3:0/1755652035 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3edc0045d0 con 0x7f3ee41574a0 2024-05-08T21:23:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.895+0000 7f3ee17fa700 1 -- 172.21.15.3:0/1755652035 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc014480 con 0x7f3ee41574a0 2024-05-08T21:23:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.895+0000 7f3eeb124700 1 -- 172.21.15.3:0/1755652035 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee4160040 con 0x7f3ee41574a0 2024-05-08T21:23:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.895+0000 7f3eeb124700 1 -- 172.21.15.3:0/1755652035 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee41f6240 con 0x7f3ee41574a0 2024-05-08T21:23:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.896+0000 7f3eeb124700 1 -- 172.21.15.3:0/1755652035 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee40b26c0 con 0x7f3ee41574a0 2024-05-08T21:23:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.896+0000 7f3ee17fa700 1 -- 172.21.15.3:0/1755652035 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3edc024070 con 0x7f3ee41574a0 2024-05-08T21:23:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.897+0000 7f3ee17fa700 1 --2- 172.21.15.3:0/1755652035 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3ec806ac90 0x7f3ec806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.897+0000 7f3ee17fa700 1 -- 172.21.15.3:0/1755652035 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f3edc08a660 con 0x7f3ee41574a0 2024-05-08T21:23:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.897+0000 7f3ee8ec0700 1 --2- 172.21.15.3:0/1755652035 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3ec806ac90 0x7f3ec806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.897+0000 7f3ee8ec0700 1 --2- 172.21.15.3:0/1755652035 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3ec806ac90 0x7f3ec806d150 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8000c00 tx=0x7f3ed8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:37.901+0000 7f3ee17fa700 1 -- 172.21.15.3:0/1755652035 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f3edc055c90 con 0x7f3ee41574a0 2024-05-08T21:23:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.120+0000 7f3eeb124700 1 -- 172.21.15.3:0/1755652035 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3ee4074bc0 con 0x7f3ee41574a0 2024-05-08T21:23:38.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.124+0000 7f3ee17fa700 1 -- 172.21.15.3:0/1755652035 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3edc025370 con 0x7f3ee41574a0 2024-05-08T21:23:38.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.126+0000 7f3eeb124700 1 -- 172.21.15.3:0/1755652035 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3ec806ac90 msgr2=0x7f3ec806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:38.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.126+0000 7f3eeb124700 1 --2- 172.21.15.3:0/1755652035 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3ec806ac90 0x7f3ec806d150 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8000c00 tx=0x7f3ed8009040 comp rx=0 tx=0).stop 2024-05-08T21:23:38.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.126+0000 7f3eeb124700 1 -- 172.21.15.3:0/1755652035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee41574a0 msgr2=0x7f3ee409cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:38.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.126+0000 7f3eeb124700 1 --2- 172.21.15.3:0/1755652035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee41574a0 0x7f3ee409cb00 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f3edc002230 tx=0x7f3edc009750 comp rx=0 tx=0).stop 2024-05-08T21:23:38.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.126+0000 7f3eeb124700 1 -- 172.21.15.3:0/1755652035 shutdown_connections 2024-05-08T21:23:38.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.126+0000 7f3eeb124700 1 --2- 172.21.15.3:0/1755652035 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3ec806ac90 0x7f3ec806d150 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:38.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.126+0000 7f3eeb124700 1 --2- 172.21.15.3:0/1755652035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee41574a0 0x7f3ee409cb00 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:38.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.126+0000 7f3eeb124700 1 --2- 172.21.15.3:0/1755652035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ed4005210 0x7f3ee41548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:38.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.126+0000 7f3eeb124700 1 --2- 172.21.15.3:0/1755652035 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ee40b1800 0x7f3ee409c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:38.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.126+0000 7f3eeb124700 1 -- 172.21.15.3:0/1755652035 >> 172.21.15.3:0/1755652035 conn(0x7f3ee40b5810 msgr2=0x7f3ee406a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:38.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.126+0000 7f3eeb124700 1 -- 172.21.15.3:0/1755652035 shutdown_connections 2024-05-08T21:23:38.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.126+0000 7f3eeb124700 1 -- 172.21.15.3:0/1755652035 wait complete. 2024-05-08T21:23:38.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple' 2024-05-08T21:23:38.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.280+0000 7fdb4df10700 1 Processor -- start 2024-05-08T21:23:38.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.280+0000 7fdb4df10700 1 -- start start 2024-05-08T21:23:38.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb4df10700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb4806f010 0x7fdb48164b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:38.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb4df10700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4806d860 con 0x7fdb48143200 2024-05-08T21:23:38.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb4df10700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4806db40 con 0x7fdb48142c30 2024-05-08T21:23:38.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb4df10700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4815e7e0 con 0x7fdb4806f010 2024-05-08T21:23:38.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb3effd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fdb48143200 0x7fdb4806e900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:36416/0 (socket says 172.21.15.3:36416) 2024-05-08T21:23:38.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb3effd700 1 -- 172.21.15.3:0/1074930494 learned_addr learned my addr 172.21.15.3:0/1074930494 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:38.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb47fff700 1 --2- 172.21.15.3:0/1074930494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb4806f010 0x7fdb48164b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:38.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb46ffd700 1 -- 172.21.15.3:0/1074930494 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629121785 0 0) 0x7fdb4806d860 con 0x7fdb48143200 2024-05-08T21:23:38.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb46ffd700 1 -- 172.21.15.3:0/1074930494 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb340035b0 con 0x7fdb48143200 2024-05-08T21:23:38.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb46ffd700 1 -- 172.21.15.3:0/1074930494 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 16150164 0 0) 0x7fdb340035b0 con 0x7fdb48143200 2024-05-08T21:23:38.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb46ffd700 1 -- 172.21.15.3:0/1074930494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb4806f010 msgr2=0x7fdb48164b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:38.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb46ffd700 1 --2- 172.21.15.3:0/1074930494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb4806f010 0x7fdb48164b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:38.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb46ffd700 1 -- 172.21.15.3:0/1074930494 >> v1:172.21.15.186:6789/0 conn(0x7fdb48142c30 legacy=0x7fdb480cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:38.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb46ffd700 1 -- 172.21.15.3:0/1074930494 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4806de20 con 0x7fdb48143200 2024-05-08T21:23:38.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb46ffd700 1 -- 172.21.15.3:0/1074930494 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7fdb30001670 con 0x7fdb48143200 2024-05-08T21:23:38.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb46ffd700 1 -- 172.21.15.3:0/1074930494 >> v1:172.21.15.3:6789/0 conn(0x7fdb48143200 legacy=0x7fdb4806e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:38.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb46ffd700 1 --2- 172.21.15.3:0/1074930494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb34003a30 0x7fdb34003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:38.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb46ffd700 1 --2- 172.21.15.3:0/1074930494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb4806f010 0x7fdb34004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:38.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb46ffd700 1 --2- 172.21.15.3:0/1074930494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb34004d90 0x7fdb34005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:38.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb46ffd700 1 -- 172.21.15.3:0/1074930494 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4815e7e0 con 0x7fdb34003a30 2024-05-08T21:23:38.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb46ffd700 1 -- 172.21.15.3:0/1074930494 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3400bb20 con 0x7fdb34004d90 2024-05-08T21:23:38.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.281+0000 7fdb46ffd700 1 -- 172.21.15.3:0/1074930494 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3400bce0 con 0x7fdb4806f010 2024-05-08T21:23:38.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.282+0000 7fdb477fe700 1 --2- 172.21.15.3:0/1074930494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb34003a30 0x7fdb34003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:38.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.282+0000 7fdb3effd700 1 --2- 172.21.15.3:0/1074930494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb4806f010 0x7fdb34004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:38.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.282+0000 7fdb47fff700 1 --2- 172.21.15.3:0/1074930494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb34004d90 0x7fdb34005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:38.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.282+0000 7fdb477fe700 1 -- 172.21.15.3:0/1074930494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb4806f010 msgr2=0x7fdb34004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.282+0000 7fdb477fe700 1 --2- 172.21.15.3:0/1074930494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb4806f010 0x7fdb34004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.282+0000 7fdb477fe700 1 -- 172.21.15.3:0/1074930494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb34004d90 msgr2=0x7fdb34005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.282+0000 7fdb477fe700 1 --2- 172.21.15.3:0/1074930494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb34004d90 0x7fdb34005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.282+0000 7fdb477fe700 1 -- 172.21.15.3:0/1074930494 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4806de20 con 0x7fdb34003a30 2024-05-08T21:23:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.282+0000 7fdb477fe700 1 --2- 172.21.15.3:0/1074930494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb34003a30 0x7fdb34003e10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fdb380018d0 tx=0x7fdb34005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7cbc9904910b386 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.282+0000 7fdb46ffd700 1 -- 172.21.15.3:0/1074930494 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3800c690 con 0x7fdb34003a30 2024-05-08T21:23:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.282+0000 7fdb46ffd700 1 -- 172.21.15.3:0/1074930494 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3800c810 con 0x7fdb34003a30 2024-05-08T21:23:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.282+0000 7fdb46ffd700 1 -- 172.21.15.3:0/1074930494 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3800cbc0 con 0x7fdb34003a30 2024-05-08T21:23:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.283+0000 7fdb4df10700 1 -- 172.21.15.3:0/1074930494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb34003a30 msgr2=0x7fdb34003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.283+0000 7fdb4df10700 1 --2- 172.21.15.3:0/1074930494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb34003a30 0x7fdb34003e10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fdb380018d0 tx=0x7fdb34005880 comp rx=0 tx=0).stop 2024-05-08T21:23:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.283+0000 7fdb47fff700 1 -- 172.21.15.3:0/1074930494 reap_dead start 2024-05-08T21:23:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.283+0000 7fdb4df10700 1 -- 172.21.15.3:0/1074930494 shutdown_connections 2024-05-08T21:23:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.283+0000 7fdb4df10700 1 -- 172.21.15.3:0/1074930494 >> 172.21.15.3:0/1074930494 conn(0x7fdb480cfce0 msgr2=0x7fdb480d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.283+0000 7fdb4df10700 1 -- 172.21.15.3:0/1074930494 shutdown_connections 2024-05-08T21:23:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.283+0000 7fdb4df10700 1 -- 172.21.15.3:0/1074930494 wait complete. 2024-05-08T21:23:38.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.284+0000 7fdb4df10700 1 Processor -- start 2024-05-08T21:23:38.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.284+0000 7fdb4df10700 1 -- start start 2024-05-08T21:23:38.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.285+0000 7fdb4df10700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb34003a30 0x7fdb48132670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:38.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.285+0000 7fdb4df10700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb481f41c0 0x7fdb4813eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.285+0000 7fdb4df10700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb480cb510 0x7fdb480cdf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.285+0000 7fdb4df10700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb48132bb0 con 0x7fdb481f41c0 2024-05-08T21:23:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.285+0000 7fdb4df10700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb48131bd0 con 0x7fdb480cb510 2024-05-08T21:23:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.285+0000 7fdb4df10700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4813f7e0 con 0x7fdb34003a30 2024-05-08T21:23:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.285+0000 7fdb3effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb481f41c0 0x7fdb4813eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.285+0000 7fdb3effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb481f41c0 0x7fdb4813eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35252/0 (socket says 172.21.15.3:35252) 2024-05-08T21:23:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.285+0000 7fdb3effd700 1 -- 172.21.15.3:0/1980101048 learned_addr learned my addr 172.21.15.3:0/1980101048 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.286+0000 7fdb477fe700 1 --2- 172.21.15.3:0/1980101048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb34003a30 0x7fdb48132670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.286+0000 7fdb47fff700 1 --2- 172.21.15.3:0/1980101048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb480cb510 0x7fdb480cdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.286+0000 7fdb3effd700 1 -- 172.21.15.3:0/1980101048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb34003a30 msgr2=0x7fdb48132670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.286+0000 7fdb3effd700 1 --2- 172.21.15.3:0/1980101048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb34003a30 0x7fdb48132670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.286+0000 7fdb3effd700 1 -- 172.21.15.3:0/1980101048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb480cb510 msgr2=0x7fdb480cdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.286+0000 7fdb3effd700 1 --2- 172.21.15.3:0/1980101048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb480cb510 0x7fdb480cdf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.286+0000 7fdb3effd700 1 -- 172.21.15.3:0/1980101048 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4813fac0 con 0x7fdb481f41c0 2024-05-08T21:23:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.286+0000 7fdb47fff700 1 --2- 172.21.15.3:0/1980101048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb480cb510 0x7fdb480cdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.286+0000 7fdb3effd700 1 --2- 172.21.15.3:0/1980101048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb481f41c0 0x7fdb4813eee0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fdb300072e0 tx=0x7fdb300076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.286+0000 7fdb44ff9700 1 -- 172.21.15.3:0/1980101048 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb300077c0 con 0x7fdb481f41c0 2024-05-08T21:23:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.286+0000 7fdb44ff9700 1 -- 172.21.15.3:0/1980101048 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3000f8d0 con 0x7fdb481f41c0 2024-05-08T21:23:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.286+0000 7fdb4df10700 1 -- 172.21.15.3:0/1980101048 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb48131e70 con 0x7fdb481f41c0 2024-05-08T21:23:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.286+0000 7fdb44ff9700 1 -- 172.21.15.3:0/1980101048 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3000fc80 con 0x7fdb481f41c0 2024-05-08T21:23:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.286+0000 7fdb4df10700 1 -- 172.21.15.3:0/1980101048 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4813f420 con 0x7fdb481f41c0 2024-05-08T21:23:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.287+0000 7fdb4df10700 1 -- 172.21.15.3:0/1980101048 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb4813fce0 con 0x7fdb481f41c0 2024-05-08T21:23:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.288+0000 7fdb44ff9700 1 -- 172.21.15.3:0/1980101048 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdb3001d460 con 0x7fdb481f41c0 2024-05-08T21:23:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.288+0000 7fdb44ff9700 1 --2- 172.21.15.3:0/1980101048 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdb2806ac90 0x7fdb2806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.288+0000 7fdb44ff9700 1 -- 172.21.15.3:0/1980101048 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fdb30089150 con 0x7fdb481f41c0 2024-05-08T21:23:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.288+0000 7fdb477fe700 1 --2- 172.21.15.3:0/1980101048 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdb2806ac90 0x7fdb2806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.289+0000 7fdb477fe700 1 --2- 172.21.15.3:0/1980101048 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdb2806ac90 0x7fdb2806d150 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fdb38014040 tx=0x7fdb38005040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:38.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.293+0000 7fdb44ff9700 1 -- 172.21.15.3:0/1980101048 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fdb4813fce0 con 0x7fdb481f41c0 2024-05-08T21:23:38.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:38.604+0000 7fdb4df10700 1 -- 172.21.15.3:0/1980101048 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple"]} v 0) v1 -- 0x7fdb48132390 con 0x7fdb481f41c0 2024-05-08T21:23:39.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:39.335+0000 7fdb44ff9700 1 -- 172.21.15.3:0/1980101048 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple"]}]=0 v124) v1 ==== 147+0+0 (secure 0 0 0) 0x7fdb48132390 con 0x7fdb481f41c0 2024-05-08T21:23:39.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:39.338+0000 7fdb4df10700 1 -- 172.21.15.3:0/1980101048 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdb2806ac90 msgr2=0x7fdb2806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:39.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:39.338+0000 7fdb4df10700 1 --2- 172.21.15.3:0/1980101048 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdb2806ac90 0x7fdb2806d150 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fdb38014040 tx=0x7fdb38005040 comp rx=0 tx=0).stop 2024-05-08T21:23:39.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:39.338+0000 7fdb4df10700 1 -- 172.21.15.3:0/1980101048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb481f41c0 msgr2=0x7fdb4813eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:39.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:39.338+0000 7fdb4df10700 1 --2- 172.21.15.3:0/1980101048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb481f41c0 0x7fdb4813eee0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fdb300072e0 tx=0x7fdb300076a0 comp rx=0 tx=0).stop 2024-05-08T21:23:39.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:39.338+0000 7fdb4df10700 1 -- 172.21.15.3:0/1980101048 shutdown_connections 2024-05-08T21:23:39.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:39.338+0000 7fdb4df10700 1 --2- 172.21.15.3:0/1980101048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb34003a30 0x7fdb48132670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:39.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:39.338+0000 7fdb4df10700 1 --2- 172.21.15.3:0/1980101048 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fdb2806ac90 0x7fdb2806d150 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:39.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:39.338+0000 7fdb4df10700 1 --2- 172.21.15.3:0/1980101048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb480cb510 0x7fdb480cdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:39.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:39.338+0000 7fdb4df10700 1 --2- 172.21.15.3:0/1980101048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb481f41c0 0x7fdb4813eee0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:39.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:39.338+0000 7fdb4df10700 1 -- 172.21.15.3:0/1980101048 >> 172.21.15.3:0/1980101048 conn(0x7fdb480cfce0 msgr2=0x7fdb480af900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:39.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:39.338+0000 7fdb4df10700 1 -- 172.21.15.3:0/1980101048 shutdown_connections 2024-05-08T21:23:39.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:39.338+0000 7fdb4df10700 1 -- 172.21.15.3:0/1980101048 wait complete. 2024-05-08T21:23:39.355 INFO:tasks.cephfs_test_runner:test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) ... ok 2024-05-08T21:23:39.355 INFO:tasks.cephfs_test_runner: 2024-05-08T21:23:39.355 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-05-08T21:23:39.355 INFO:tasks.cephfs_test_runner:Ran 1 test in 76.996s 2024-05-08T21:23:39.356 INFO:tasks.cephfs_test_runner: 2024-05-08T21:23:39.356 INFO:tasks.cephfs_test_runner:OK 2024-05-08T21:23:39.356 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-05-08T21:23:39.366 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-08T21:23:39.388 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-08T21:23:39.388 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:23:39.389 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:23:39.411 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16247272k,nr_inodes=4061818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=4629 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253140k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-05-08T21:23:39.411 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-05-08T21:23:39.411 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:23:39.473 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16247272k,nr_inodes=4061818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=4629 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253140k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-05-08T21:23:39.473 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:23:39.474 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:23:39.495 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244180k,nr_inodes=4061045,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15987 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-05-08T21:23:39.495 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-08T21:23:39.495 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T21:23:39.556 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244180k,nr_inodes=4061045,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15987 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-05-08T21:23:39.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:39.557 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2024-05-08T21:23:39.578 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-08T21:23:39.579 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-08T21:23:39.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:39.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-08T21:23:39.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:39.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-08T21:23:39.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:39.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link delete ceph-brx 2024-05-08T21:23:39.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:39.830 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2024-05-08T21:23:39.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:39.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link delete ceph-brx 2024-05-08T21:23:39.933 INFO:teuthology.orchestra.run.smithi003.stderr:Cannot find device "ceph-brx" 2024-05-08T21:23:39.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T21:23:39.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:39.936 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2024-05-08T21:23:39.957 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-05-08T21:23:39.957 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-05-08T21:23:39.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:39.957 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-08T21:23:40.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:40.036 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-08T21:23:40.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:40.115 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link delete ceph-brx 2024-05-08T21:23:40.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:40.211 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2024-05-08T21:23:40.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T21:23:40.229 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link delete ceph-brx 2024-05-08T21:23:40.301 INFO:teuthology.orchestra.run.smithi186.stderr:Cannot find device "ceph-brx" 2024-05-08T21:23:40.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T21:23:40.302 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-08T21:23:40.312 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-08T21:23:40.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T21:23:40.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.457+0000 7f406043c700 1 Processor -- start 2024-05-08T21:23:40.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.457+0000 7f406043c700 1 -- start start 2024-05-08T21:23:40.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.458+0000 7f406043c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f405806e530 0x7f405806e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:40.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.458+0000 7f406043c700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40580ae2a0 con 0x7f40580acfb0 2024-05-08T21:23:40.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.458+0000 7f406043c700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40580ad430 con 0x7f405815a670 2024-05-08T21:23:40.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.458+0000 7f406043c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f405806ef20 con 0x7f405806e530 2024-05-08T21:23:40.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d9d7700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f40580acfb0 0x7f405815ea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:36430/0 (socket says 172.21.15.3:36430) 2024-05-08T21:23:40.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d9d7700 1 -- 172.21.15.3:0/3090631354 learned_addr learned my addr 172.21.15.3:0/3090631354 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:40.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405e9d9700 1 --2- 172.21.15.3:0/3090631354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f405806e530 0x7f405806e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:40.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d1d6700 1 -- 172.21.15.3:0/3090631354 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689978962 0 0) 0x7f40580ae2a0 con 0x7f40580acfb0 2024-05-08T21:23:40.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d1d6700 1 -- 172.21.15.3:0/3090631354 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40440035b0 con 0x7f40580acfb0 2024-05-08T21:23:40.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d1d6700 1 -- 172.21.15.3:0/3090631354 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3656724827 0 0) 0x7f40440035b0 con 0x7f40580acfb0 2024-05-08T21:23:40.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d1d6700 1 -- 172.21.15.3:0/3090631354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f405806e530 msgr2=0x7f405806e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:40.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d1d6700 1 --2- 172.21.15.3:0/3090631354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f405806e530 0x7f405806e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:40.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d1d6700 1 -- 172.21.15.3:0/3090631354 >> v1:172.21.15.186:6789/0 conn(0x7f405815a670 legacy=0x7f40580ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:40.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d1d6700 1 -- 172.21.15.3:0/3090631354 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40580ad690 con 0x7f40580acfb0 2024-05-08T21:23:40.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d1d6700 1 -- 172.21.15.3:0/3090631354 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f4048001670 con 0x7f40580acfb0 2024-05-08T21:23:40.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d1d6700 1 -- 172.21.15.3:0/3090631354 >> v1:172.21.15.3:6789/0 conn(0x7f40580acfb0 legacy=0x7f405815ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:40.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d1d6700 1 --2- 172.21.15.3:0/3090631354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4044003a30 0x7f4044003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:40.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d1d6700 1 --2- 172.21.15.3:0/3090631354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f405806e530 0x7f4044004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:40.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d1d6700 1 --2- 172.21.15.3:0/3090631354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4044004d90 0x7f4044005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:40.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d1d6700 1 -- 172.21.15.3:0/3090631354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405806ef20 con 0x7f4044004d90 2024-05-08T21:23:40.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d1d6700 1 -- 172.21.15.3:0/3090631354 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404400ba40 con 0x7f4044003a30 2024-05-08T21:23:40.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d1d6700 1 -- 172.21.15.3:0/3090631354 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404400bc00 con 0x7f405806e530 2024-05-08T21:23:40.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405e1d8700 1 --2- 172.21.15.3:0/3090631354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4044003a30 0x7f4044003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:40.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405e9d9700 1 --2- 172.21.15.3:0/3090631354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4044004d90 0x7f4044005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:40.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.459+0000 7f405d9d7700 1 --2- 172.21.15.3:0/3090631354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f405806e530 0x7f4044004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:40.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.460+0000 7f405e9d9700 1 -- 172.21.15.3:0/3090631354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f405806e530 msgr2=0x7f4044004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:40.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.460+0000 7f405e9d9700 1 --2- 172.21.15.3:0/3090631354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f405806e530 0x7f4044004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:40.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.460+0000 7f405e9d9700 1 -- 172.21.15.3:0/3090631354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4044003a30 msgr2=0x7f4044003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:40.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.460+0000 7f405e9d9700 1 --2- 172.21.15.3:0/3090631354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4044003a30 0x7f4044003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:40.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.460+0000 7f405e9d9700 1 -- 172.21.15.3:0/3090631354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40440035b0 con 0x7f4044004d90 2024-05-08T21:23:40.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.460+0000 7f405e9d9700 1 --2- 172.21.15.3:0/3090631354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4044004d90 0x7f4044005210 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f4050009fc0 tx=0x7f4044005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d15ae48807e9f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:40.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.460+0000 7f405d1d6700 1 -- 172.21.15.3:0/3090631354 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405000ec30 con 0x7f4044004d90 2024-05-08T21:23:40.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.460+0000 7f405d1d6700 1 -- 172.21.15.3:0/3090631354 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405000f040 con 0x7f4044004d90 2024-05-08T21:23:40.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.460+0000 7f405d1d6700 1 -- 172.21.15.3:0/3090631354 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4050014480 con 0x7f4044004d90 2024-05-08T21:23:40.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.461+0000 7f406043c700 1 -- 172.21.15.3:0/3090631354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4044004d90 msgr2=0x7f4044005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:40.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.461+0000 7f406043c700 1 --2- 172.21.15.3:0/3090631354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4044004d90 0x7f4044005210 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f4050009fc0 tx=0x7f4044005880 comp rx=0 tx=0).stop 2024-05-08T21:23:40.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.461+0000 7f405e9d9700 1 -- 172.21.15.3:0/3090631354 reap_dead start 2024-05-08T21:23:40.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.461+0000 7f406043c700 1 -- 172.21.15.3:0/3090631354 shutdown_connections 2024-05-08T21:23:40.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.461+0000 7f406043c700 1 -- 172.21.15.3:0/3090631354 >> 172.21.15.3:0/3090631354 conn(0x7f40580cf270 msgr2=0x7f40580cf650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:40.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.461+0000 7f406043c700 1 -- 172.21.15.3:0/3090631354 shutdown_connections 2024-05-08T21:23:40.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.461+0000 7f406043c700 1 -- 172.21.15.3:0/3090631354 wait complete. 2024-05-08T21:23:40.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.462+0000 7f406043c700 1 Processor -- start 2024-05-08T21:23:40.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.462+0000 7f406043c700 1 -- start start 2024-05-08T21:23:40.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.463+0000 7f406043c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4044004d90 0x7f4058159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:40.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.463+0000 7f406043c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40580c9180 0x7f4058149180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:40.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.463+0000 7f406043c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40580cbdb0 0x7f40580cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:40.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.463+0000 7f406043c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40580c9940 con 0x7f40580cbdb0 2024-05-08T21:23:40.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.463+0000 7f406043c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40580cf080 con 0x7f40580c9180 2024-05-08T21:23:40.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.463+0000 7f406043c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40580ce790 con 0x7f4044004d90 2024-05-08T21:23:40.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.463+0000 7f405e9d9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40580cbdb0 0x7f40580cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:40.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.463+0000 7f405e9d9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40580cbdb0 0x7f40580cc770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35282/0 (socket says 172.21.15.3:35282) 2024-05-08T21:23:40.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.463+0000 7f405e9d9700 1 -- 172.21.15.3:0/3776190718 learned_addr learned my addr 172.21.15.3:0/3776190718 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:40.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.463+0000 7f405e1d8700 1 --2- 172.21.15.3:0/3776190718 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4044004d90 0x7f4058159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:40.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.463+0000 7f405d9d7700 1 --2- 172.21.15.3:0/3776190718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40580c9180 0x7f4058149180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:40.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.464+0000 7f405e9d9700 1 -- 172.21.15.3:0/3776190718 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4044004d90 msgr2=0x7f4058159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:40.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.464+0000 7f405e9d9700 1 --2- 172.21.15.3:0/3776190718 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4044004d90 0x7f4058159820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:40.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.464+0000 7f405e9d9700 1 -- 172.21.15.3:0/3776190718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40580c9180 msgr2=0x7f4058149180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:40.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.464+0000 7f405e9d9700 1 --2- 172.21.15.3:0/3776190718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40580c9180 0x7f4058149180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:40.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.464+0000 7f405e9d9700 1 -- 172.21.15.3:0/3776190718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40580c9660 con 0x7f40580cbdb0 2024-05-08T21:23:40.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.464+0000 7f405e9d9700 1 --2- 172.21.15.3:0/3776190718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40580cbdb0 0x7f40580cc770 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4050017040 tx=0x7f4050009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:40.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.464+0000 7f404effd700 1 -- 172.21.15.3:0/3776190718 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4050007000 con 0x7f40580cbdb0 2024-05-08T21:23:40.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.464+0000 7f404effd700 1 -- 172.21.15.3:0/3776190718 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4050003e50 con 0x7f40580cbdb0 2024-05-08T21:23:40.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.464+0000 7f406043c700 1 -- 172.21.15.3:0/3776190718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40580cea30 con 0x7f40580cbdb0 2024-05-08T21:23:40.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.464+0000 7f404effd700 1 -- 172.21.15.3:0/3776190718 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40500149f0 con 0x7f40580cbdb0 2024-05-08T21:23:40.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.464+0000 7f406043c700 1 -- 172.21.15.3:0/3776190718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40580cc260 con 0x7f40580cbdb0 2024-05-08T21:23:40.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.465+0000 7f406043c700 1 -- 172.21.15.3:0/3776190718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40580cc3f0 con 0x7f40580cbdb0 2024-05-08T21:23:40.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.466+0000 7f404effd700 1 -- 172.21.15.3:0/3776190718 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4050023020 con 0x7f40580cbdb0 2024-05-08T21:23:40.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.466+0000 7f404effd700 1 --2- 172.21.15.3:0/3776190718 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f403c06ac90 0x7f403c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:40.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.466+0000 7f404effd700 1 -- 172.21.15.3:0/3776190718 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f405008a520 con 0x7f40580cbdb0 2024-05-08T21:23:40.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.467+0000 7f405e1d8700 1 --2- 172.21.15.3:0/3776190718 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f403c06ac90 0x7f403c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:40.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.467+0000 7f405e1d8700 1 --2- 172.21.15.3:0/3776190718 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f403c06ac90 0x7f403c06d150 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4054000c00 tx=0x7f4054009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:40.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.471+0000 7f404effd700 1 -- 172.21.15.3:0/3776190718 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f4050055ab0 con 0x7f40580cbdb0 2024-05-08T21:23:40.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.682+0000 7f406043c700 1 -- 172.21.15.3:0/3776190718 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40580adf30 con 0x7f403c06ac90 2024-05-08T21:23:40.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.685+0000 7f404effd700 1 -- 172.21.15.3:0/3776190718 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50862 (secure 0 0 0) 0x7f4054012700 con 0x7f403c06ac90 2024-05-08T21:23:40.687 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:40.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.687+0000 7f406043c700 1 -- 172.21.15.3:0/3776190718 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f403c06ac90 msgr2=0x7f403c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:40.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.687+0000 7f406043c700 1 --2- 172.21.15.3:0/3776190718 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f403c06ac90 0x7f403c06d150 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4054000c00 tx=0x7f4054009040 comp rx=0 tx=0).stop 2024-05-08T21:23:40.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.687+0000 7f406043c700 1 -- 172.21.15.3:0/3776190718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40580cbdb0 msgr2=0x7f40580cc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:40.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.687+0000 7f406043c700 1 --2- 172.21.15.3:0/3776190718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40580cbdb0 0x7f40580cc770 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4050017040 tx=0x7f4050009750 comp rx=0 tx=0).stop 2024-05-08T21:23:40.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.687+0000 7f406043c700 1 -- 172.21.15.3:0/3776190718 shutdown_connections 2024-05-08T21:23:40.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.687+0000 7f406043c700 1 --2- 172.21.15.3:0/3776190718 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4044004d90 0x7f4058159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:40.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.687+0000 7f406043c700 1 --2- 172.21.15.3:0/3776190718 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f403c06ac90 0x7f403c06d150 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:40.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.687+0000 7f406043c700 1 --2- 172.21.15.3:0/3776190718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40580cbdb0 0x7f40580cc770 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:40.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.687+0000 7f406043c700 1 --2- 172.21.15.3:0/3776190718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40580c9180 0x7f4058149180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:40.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.687+0000 7f406043c700 1 -- 172.21.15.3:0/3776190718 >> 172.21.15.3:0/3776190718 conn(0x7f40580cf270 msgr2=0x7f40580b03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:40.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.687+0000 7f406043c700 1 -- 172.21.15.3:0/3776190718 shutdown_connections 2024-05-08T21:23:40.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.687+0000 7f406043c700 1 -- 172.21.15.3:0/3776190718 wait complete. 2024-05-08T21:23:40.690 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-08T21:23:40.702 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-05-08T21:23:40.557484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233716,"kb_used_data":17556,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754741004,"statfs":{"total":773094113280,"available":772854788096,"internally_reserved":0,"allocated":17977344,"data_stored":15543186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6322160,"num_objects":-15,"num_object_clones":0,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-15,"num_whiteouts":0,"num_read":-53,"num_read_kb":-39,"num_write":-50,"num_write_kb":-6193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.248664"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":186,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-08T21:23:29.908714+0000","last_change":"2024-05-08T21:21:42.333097+0000","last_active":"2024-05-08T21:23:29.908714+0000","last_peered":"2024-05-08T21:23:29.908714+0000","last_clean":"2024-05-08T21:23:29.908714+0000","last_became_active":"2024-05-08T21:21:42.332033+0000","last_became_peered":"2024-05-08T21:21:42.332033+0000","last_unstale":"2024-05-08T21:23:29.908714+0000","last_undegraded":"2024-05-08T21:23:29.908714+0000","last_fullsized":"2024-05-08T21:23:29.908714+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_clean_scrub_stamp":"2024-05-08T21:21:40.862622+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:39:24.772551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30152,"kb_used_data":3132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341688,"statfs":{"total":96636764160,"available":96605888512,"internally_reserved":0,"allocated":3207168,"data_stored":2903149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.54600000000000004}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.53600000000000003}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.57499999999999996}]},{"osd":3,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.45700000000000002}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.433},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.56000000000000005}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.49299999999999999}]},{"osd":6,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.47399999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29576,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342264,"statfs":{"total":96636764160,"available":96606478336,"internally_reserved":0,"allocated":2617344,"data_stored":2313363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.48199999999999998}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.39200000000000002}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.50800000000000001}]},{"osd":3,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.56200000000000006}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.44500000000000001}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.58599999999999997}]},{"osd":7,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.52100000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31608,"kb_used_data":4588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340232,"statfs":{"total":96636764160,"available":96604397568,"internally_reserved":0,"allocated":4698112,"data_stored":4403731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.314},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.36899999999999999}]},{"osd":2,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.33100000000000002}]},{"osd":4,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52300000000000002}]},{"osd":5,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.51100000000000001}]},{"osd":6,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.57099999999999995}]},{"osd":7,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.622}]}]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28104,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343736,"statfs":{"total":96636764160,"available":96607985664,"internally_reserved":0,"allocated":1110016,"data_stored":805941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.435},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.54800000000000004}]},{"osd":2,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.59899999999999998}]},{"osd":3,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.68400000000000005}]},{"osd":4,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.80000000000000004}]},{"osd":5,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.82799999999999996}]},{"osd":6,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.76000000000000001}]},{"osd":7,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.874},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.89700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":245467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.379},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.443}]},{"osd":1,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38900000000000001}]},{"osd":3,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.40699999999999997}]},{"osd":4,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.67200000000000004}]},{"osd":5,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.755}]},{"osd":6,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.74399999999999999}]},{"osd":7,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.64800000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":524288,"data_stored":216177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":2.7919999999999998,"5min":2.7919999999999998,"15min":2.7919999999999998},"last":0.375}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.433},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.40300000000000002}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":2.9369999999999998,"5min":2.9369999999999998,"15min":2.9369999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.38600000000000001}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.8380000000000001,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.9060000000000001,"5min":2.9060000000000001,"15min":2.9060000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.46700000000000003}]},{"osd":6,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":2.8730000000000002,"5min":2.8730000000000002,"15min":2.8730000000000002},"last":0.50600000000000001}]},{"osd":7,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.495},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.63300000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27560,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344280,"statfs":{"total":96636764160,"available":96608542720,"internally_reserved":0,"allocated":552960,"data_stored":245445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.39700000000000002}]},{"osd":1,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.58999999999999997}]},{"osd":2,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.56999999999999995}]},{"osd":3,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.433},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.41899999999999998}]},{"osd":5,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52400000000000002}]},{"osd":6,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.54800000000000004}]},{"osd":7,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.49399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31620,"kb_used_data":4600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340220,"statfs":{"total":96636764160,"available":96604385280,"internally_reserved":0,"allocated":4710400,"data_stored":4409913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.38900000000000001}]},{"osd":1,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.57599999999999996}]},{"osd":2,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.48999999999999999}]},{"osd":3,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.504}]},{"osd":4,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.375}]},{"osd":6,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.53600000000000003}]},{"osd":7,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T21:23:40.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T21:23:40.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.839+0000 7f36d13dc700 1 Processor -- start 2024-05-08T21:23:40.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.839+0000 7f36d13dc700 1 -- start start 2024-05-08T21:23:40.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.839+0000 7f36d13dc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36cc099a50 0x7f36cc099ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:40.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.839+0000 7f36d13dc700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f36cc0ccdd0 0x7f36cc0cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:40.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.839+0000 7f36d13dc700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36cc09aca0 con 0x7f36cc15b810 2024-05-08T21:23:40.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.839+0000 7f36d13dc700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36cc09a410 con 0x7f36cc0ccdd0 2024-05-08T21:23:40.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.840+0000 7f36d13dc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc0cd7a0 con 0x7f36cc099a50 2024-05-08T21:23:40.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.840+0000 7f36caffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f36cc15b810 0x7f36cc0cffe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:36432/0 (socket says 172.21.15.3:36432) 2024-05-08T21:23:40.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.840+0000 7f36caffd700 1 -- 172.21.15.3:0/1186843051 learned_addr learned my addr 172.21.15.3:0/1186843051 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:40.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.840+0000 7f36cb7fe700 1 --2- 172.21.15.3:0/1186843051 >> v2:172.21.15.186:3300/0 conn(0x7f36cc0ccdd0 0x7f36cc0cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:40.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.840+0000 7f36ca7fc700 1 --2- 172.21.15.3:0/1186843051 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36cc099a50 0x7f36cc099ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:40.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.840+0000 7f36c9ffb700 1 -- 172.21.15.3:0/1186843051 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349853508 0 0) 0x7f36cc09aca0 con 0x7f36cc15b810 2024-05-08T21:23:40.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.840+0000 7f36c9ffb700 1 -- 172.21.15.3:0/1186843051 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b40035b0 con 0x7f36cc15b810 2024-05-08T21:23:40.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.840+0000 7f36cb7fe700 1 -- 172.21.15.3:0/1186843051 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36cc099a50 msgr2=0x7f36cc099ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:40.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.840+0000 7f36c9ffb700 1 -- 172.21.15.3:0/1186843051 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 891341191 0 0) 0x7f36b40035b0 con 0x7f36cc15b810 2024-05-08T21:23:40.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.840+0000 7f36cb7fe700 1 --2- 172.21.15.3:0/1186843051 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36cc099a50 0x7f36cc099ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:40.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.840+0000 7f36cb7fe700 1 -- 172.21.15.3:0/1186843051 >> v1:172.21.15.3:6789/0 conn(0x7f36cc15b810 legacy=0x7f36cc0cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:40.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.840+0000 7f36cb7fe700 1 -- 172.21.15.3:0/1186843051 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36cc0cd9e0 con 0x7f36cc0ccdd0 2024-05-08T21:23:40.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.840+0000 7f36ca7fc700 1 --2- 172.21.15.3:0/1186843051 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36cc099a50 0x7f36cc099ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:40.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.841+0000 7f36cb7fe700 1 --2- 172.21.15.3:0/1186843051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36cc0ccdd0 0x7f36cc0cd260 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f36c4009d90 tx=0x7f36c400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8a18a68e8897e1c server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:40.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.841+0000 7f36c9ffb700 1 -- 172.21.15.3:0/1186843051 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c400e070 con 0x7f36cc0ccdd0 2024-05-08T21:23:40.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.841+0000 7f36c9ffb700 1 -- 172.21.15.3:0/1186843051 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c4003a20 con 0x7f36cc0ccdd0 2024-05-08T21:23:40.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.841+0000 7f36c9ffb700 1 -- 172.21.15.3:0/1186843051 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc001670 con 0x7f36cc0ccdd0 2024-05-08T21:23:40.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.841+0000 7f36d13dc700 1 -- 172.21.15.3:0/1186843051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36cc0ccdd0 msgr2=0x7f36cc0cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:40.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.841+0000 7f36d13dc700 1 --2- 172.21.15.3:0/1186843051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36cc0ccdd0 0x7f36cc0cd260 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f36c4009d90 tx=0x7f36c400b330 comp rx=0 tx=0).stop 2024-05-08T21:23:40.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.841+0000 7f36d13dc700 1 -- 172.21.15.3:0/1186843051 shutdown_connections 2024-05-08T21:23:40.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.841+0000 7f36d13dc700 1 --2- 172.21.15.3:0/1186843051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36cc0ccdd0 0x7f36cc0cd260 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:40.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.841+0000 7f36d13dc700 1 --2- 172.21.15.3:0/1186843051 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36cc099a50 0x7f36cc099ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:40.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.841+0000 7f36d13dc700 1 -- 172.21.15.3:0/1186843051 >> 172.21.15.3:0/1186843051 conn(0x7f36cc0b3010 msgr2=0x7f36cc0b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:40.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.841+0000 7f36d13dc700 1 -- 172.21.15.3:0/1186843051 shutdown_connections 2024-05-08T21:23:40.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.841+0000 7f36d13dc700 1 -- 172.21.15.3:0/1186843051 wait complete. 2024-05-08T21:23:40.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.843+0000 7f36d13dc700 1 Processor -- start 2024-05-08T21:23:40.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.843+0000 7f36d13dc700 1 -- start start 2024-05-08T21:23:40.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36d13dc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36cc099a50 0x7f36cc0b2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:40.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36d13dc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36cc0ccdd0 0x7f36cc1599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:40.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36d13dc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36cc15b810 0x7f36cc0b4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:40.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36d13dc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc0b70a0 con 0x7f36cc0ccdd0 2024-05-08T21:23:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36d13dc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc0b6dc0 con 0x7f36cc15b810 2024-05-08T21:23:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36d13dc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc15c710 con 0x7f36cc099a50 2024-05-08T21:23:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36ca7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36cc0ccdd0 0x7f36cc1599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36ca7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36cc0ccdd0 0x7f36cc1599b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35286/0 (socket says 172.21.15.3:35286) 2024-05-08T21:23:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36ca7fc700 1 -- 172.21.15.3:0/1409938983 learned_addr learned my addr 172.21.15.3:0/1409938983 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36caffd700 1 --2- 172.21.15.3:0/1409938983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36cc099a50 0x7f36cc0b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36ca7fc700 1 -- 172.21.15.3:0/1409938983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36cc099a50 msgr2=0x7f36cc0b2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36cb7fe700 1 --2- 172.21.15.3:0/1409938983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36cc15b810 0x7f36cc0b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36ca7fc700 1 --2- 172.21.15.3:0/1409938983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36cc099a50 0x7f36cc0b2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36ca7fc700 1 -- 172.21.15.3:0/1409938983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36cc15b810 msgr2=0x7f36cc0b4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36ca7fc700 1 --2- 172.21.15.3:0/1409938983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36cc15b810 0x7f36cc0b4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36ca7fc700 1 -- 172.21.15.3:0/1409938983 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c00094a0 con 0x7f36cc0ccdd0 2024-05-08T21:23:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36cb7fe700 1 --2- 172.21.15.3:0/1409938983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36cc15b810 0x7f36cc0b4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36caffd700 1 --2- 172.21.15.3:0/1409938983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36cc099a50 0x7f36cc0b2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.844+0000 7f36ca7fc700 1 --2- 172.21.15.3:0/1409938983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36cc0ccdd0 0x7f36cc1599b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f36c000ebe0 tx=0x7f36c0009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:40.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.845+0000 7f36b37fe700 1 -- 172.21.15.3:0/1409938983 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c000c8b0 con 0x7f36cc0ccdd0 2024-05-08T21:23:40.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.845+0000 7f36b37fe700 1 -- 172.21.15.3:0/1409938983 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c000ca10 con 0x7f36cc0ccdd0 2024-05-08T21:23:40.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.845+0000 7f36d13dc700 1 -- 172.21.15.3:0/1409938983 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c4009a70 con 0x7f36cc0ccdd0 2024-05-08T21:23:40.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.845+0000 7f36b37fe700 1 -- 172.21.15.3:0/1409938983 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c000cda0 con 0x7f36cc0ccdd0 2024-05-08T21:23:40.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.845+0000 7f36d13dc700 1 -- 172.21.15.3:0/1409938983 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36cc15ccd0 con 0x7f36cc0ccdd0 2024-05-08T21:23:40.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.846+0000 7f36d13dc700 1 -- 172.21.15.3:0/1409938983 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36cc068000 con 0x7f36cc0ccdd0 2024-05-08T21:23:40.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.847+0000 7f36b37fe700 1 -- 172.21.15.3:0/1409938983 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f36c0013470 con 0x7f36cc0ccdd0 2024-05-08T21:23:40.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.847+0000 7f36b37fe700 1 --2- 172.21.15.3:0/1409938983 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f36b806ac90 0x7f36b806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:40.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.847+0000 7f36b37fe700 1 -- 172.21.15.3:0/1409938983 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f36c0089880 con 0x7f36cc0ccdd0 2024-05-08T21:23:40.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.848+0000 7f36caffd700 1 --2- 172.21.15.3:0/1409938983 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f36b806ac90 0x7f36b806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:40.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.848+0000 7f36caffd700 1 --2- 172.21.15.3:0/1409938983 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f36b806ac90 0x7f36b806d150 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f36bc000f60 tx=0x7f36bc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:40.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:40.851+0000 7f36b37fe700 1 -- 172.21.15.3:0/1409938983 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f36c0054e70 con 0x7f36cc0ccdd0 2024-05-08T21:23:41.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.066+0000 7f36d13dc700 1 -- 172.21.15.3:0/1409938983 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f36cc0cf7e0 con 0x7f36b806ac90 2024-05-08T21:23:41.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.069+0000 7f36b37fe700 1 -- 172.21.15.3:0/1409938983 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50862 (secure 0 0 0) 0x7f36bc0015a0 con 0x7f36b806ac90 2024-05-08T21:23:41.071 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.071+0000 7f36d13dc700 1 -- 172.21.15.3:0/1409938983 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f36b806ac90 msgr2=0x7f36b806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.071+0000 7f36d13dc700 1 --2- 172.21.15.3:0/1409938983 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f36b806ac90 0x7f36b806d150 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f36bc000f60 tx=0x7f36bc009040 comp rx=0 tx=0).stop 2024-05-08T21:23:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.071+0000 7f36d13dc700 1 -- 172.21.15.3:0/1409938983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36cc0ccdd0 msgr2=0x7f36cc1599b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.071+0000 7f36d13dc700 1 --2- 172.21.15.3:0/1409938983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36cc0ccdd0 0x7f36cc1599b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f36c000ebe0 tx=0x7f36c0009e20 comp rx=0 tx=0).stop 2024-05-08T21:23:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.071+0000 7f36d13dc700 1 -- 172.21.15.3:0/1409938983 shutdown_connections 2024-05-08T21:23:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.071+0000 7f36d13dc700 1 --2- 172.21.15.3:0/1409938983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36cc099a50 0x7f36cc0b2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.071+0000 7f36d13dc700 1 --2- 172.21.15.3:0/1409938983 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f36b806ac90 0x7f36b806d150 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.071+0000 7f36d13dc700 1 --2- 172.21.15.3:0/1409938983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36cc15b810 0x7f36cc0b4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.071+0000 7f36d13dc700 1 --2- 172.21.15.3:0/1409938983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36cc0ccdd0 0x7f36cc1599b0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.071+0000 7f36d13dc700 1 -- 172.21.15.3:0/1409938983 >> 172.21.15.3:0/1409938983 conn(0x7f36cc0b3010 msgr2=0x7f36cc0cffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.071+0000 7f36d13dc700 1 -- 172.21.15.3:0/1409938983 shutdown_connections 2024-05-08T21:23:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.071+0000 7f36d13dc700 1 -- 172.21.15.3:0/1409938983 wait complete. 2024-05-08T21:23:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-08T21:23:41.086 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-05-08T21:23:40.557484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233716,"kb_used_data":17556,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754741004,"statfs":{"total":773094113280,"available":772854788096,"internally_reserved":0,"allocated":17977344,"data_stored":15543186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6322160,"num_objects":-15,"num_object_clones":0,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-15,"num_whiteouts":0,"num_read":-53,"num_read_kb":-39,"num_write":-50,"num_write_kb":-6193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.248664"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":186,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-08T21:23:29.908714+0000","last_change":"2024-05-08T21:21:42.333097+0000","last_active":"2024-05-08T21:23:29.908714+0000","last_peered":"2024-05-08T21:23:29.908714+0000","last_clean":"2024-05-08T21:23:29.908714+0000","last_became_active":"2024-05-08T21:21:42.332033+0000","last_became_peered":"2024-05-08T21:21:42.332033+0000","last_unstale":"2024-05-08T21:23:29.908714+0000","last_undegraded":"2024-05-08T21:23:29.908714+0000","last_fullsized":"2024-05-08T21:23:29.908714+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_clean_scrub_stamp":"2024-05-08T21:21:40.862622+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:39:24.772551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30152,"kb_used_data":3132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341688,"statfs":{"total":96636764160,"available":96605888512,"internally_reserved":0,"allocated":3207168,"data_stored":2903149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.54600000000000004}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.53600000000000003}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.57499999999999996}]},{"osd":3,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.45700000000000002}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.433},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.56000000000000005}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.49299999999999999}]},{"osd":6,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.47399999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29576,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342264,"statfs":{"total":96636764160,"available":96606478336,"internally_reserved":0,"allocated":2617344,"data_stored":2313363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.48199999999999998}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.39200000000000002}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.50800000000000001}]},{"osd":3,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.56200000000000006}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.44500000000000001}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.58599999999999997}]},{"osd":7,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.52100000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31608,"kb_used_data":4588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340232,"statfs":{"total":96636764160,"available":96604397568,"internally_reserved":0,"allocated":4698112,"data_stored":4403731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.314},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.36899999999999999}]},{"osd":2,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.33100000000000002}]},{"osd":4,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52300000000000002}]},{"osd":5,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.51100000000000001}]},{"osd":6,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.57099999999999995}]},{"osd":7,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.622}]}]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28104,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343736,"statfs":{"total":96636764160,"available":96607985664,"internally_reserved":0,"allocated":1110016,"data_stored":805941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.435},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.54800000000000004}]},{"osd":2,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.59899999999999998}]},{"osd":3,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.68400000000000005}]},{"osd":4,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.80000000000000004}]},{"osd":5,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.82799999999999996}]},{"osd":6,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.76000000000000001}]},{"osd":7,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.874},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.89700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":245467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.379},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.443}]},{"osd":1,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38900000000000001}]},{"osd":3,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.40699999999999997}]},{"osd":4,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.67200000000000004}]},{"osd":5,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.755}]},{"osd":6,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.74399999999999999}]},{"osd":7,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.64800000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":524288,"data_stored":216177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":2.7919999999999998,"5min":2.7919999999999998,"15min":2.7919999999999998},"last":0.375}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.433},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.40300000000000002}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":2.9369999999999998,"5min":2.9369999999999998,"15min":2.9369999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.38600000000000001}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.8380000000000001,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.9060000000000001,"5min":2.9060000000000001,"15min":2.9060000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.46700000000000003}]},{"osd":6,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":2.8730000000000002,"5min":2.8730000000000002,"15min":2.8730000000000002},"last":0.50600000000000001}]},{"osd":7,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.495},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.63300000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27560,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344280,"statfs":{"total":96636764160,"available":96608542720,"internally_reserved":0,"allocated":552960,"data_stored":245445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.39700000000000002}]},{"osd":1,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.58999999999999997}]},{"osd":2,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.56999999999999995}]},{"osd":3,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.433},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.41899999999999998}]},{"osd":5,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52400000000000002}]},{"osd":6,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.54800000000000004}]},{"osd":7,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.49399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31620,"kb_used_data":4600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340220,"statfs":{"total":96636764160,"available":96604385280,"internally_reserved":0,"allocated":4710400,"data_stored":4409913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.38900000000000001}]},{"osd":1,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.57599999999999996}]},{"osd":2,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.48999999999999999}]},{"osd":3,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.504}]},{"osd":4,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.375}]},{"osd":6,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.53600000000000003}]},{"osd":7,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T21:23:41.088 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-08T21:23:41.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T21:23:41.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.223+0000 7f9a5acd6700 1 Processor -- start 2024-05-08T21:23:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.223+0000 7f9a5acd6700 1 -- start start 2024-05-08T21:23:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.224+0000 7f9a5acd6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a5415ed40 0x7f9a5409a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.224+0000 7f9a5acd6700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9a540b03e0 0x7f9a540b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.224+0000 7f9a5acd6700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a540cc6b0 con 0x7f9a54099a50 2024-05-08T21:23:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.224+0000 7f9a5acd6700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a540d0030 con 0x7f9a540b03e0 2024-05-08T21:23:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.224+0000 7f9a5acd6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a540d0310 con 0x7f9a5415ed40 2024-05-08T21:23:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.224+0000 7f9a53fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9a540b03e0 0x7f9a540b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.224+0000 7f9a58a72700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a5415ed40 0x7f9a5409a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.224+0000 7f9a59273700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f9a54099a50 0x7f9a540ccb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:42422/0 (socket says 172.21.15.3:42422) 2024-05-08T21:23:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.224+0000 7f9a59273700 1 -- 172.21.15.3:0/2162641064 learned_addr learned my addr 172.21.15.3:0/2162641064 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.225+0000 7f9a53fff700 1 -- 172.21.15.3:0/2162641064 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a5415ed40 msgr2=0x7f9a5409a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.225+0000 7f9a53fff700 1 --2- 172.21.15.3:0/2162641064 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a5415ed40 0x7f9a5409a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.225+0000 7f9a53fff700 1 -- 172.21.15.3:0/2162641064 >> v1:172.21.15.186:6789/0 conn(0x7f9a54099a50 legacy=0x7f9a540ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.225+0000 7f9a53fff700 1 -- 172.21.15.3:0/2162641064 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a540cc340 con 0x7f9a540b03e0 2024-05-08T21:23:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.225+0000 7f9a537fe700 1 -- 172.21.15.3:0/2162641064 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033808294 0 0) 0x7f9a540cc6b0 con 0x7f9a54099a50 2024-05-08T21:23:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.225+0000 7f9a58a72700 1 --2- 172.21.15.3:0/2162641064 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a5415ed40 0x7f9a5409a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.225+0000 7f9a53fff700 1 --2- 172.21.15.3:0/2162641064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a540b03e0 0x7f9a540b0da0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9a40009840 tx=0x7f9a40009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76cafbec35991d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.225+0000 7f9a537fe700 1 -- 172.21.15.3:0/2162641064 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a40011830 con 0x7f9a540b03e0 2024-05-08T21:23:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.225+0000 7f9a537fe700 1 -- 172.21.15.3:0/2162641064 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a40011990 con 0x7f9a540b03e0 2024-05-08T21:23:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.225+0000 7f9a537fe700 1 -- 172.21.15.3:0/2162641064 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a40011d20 con 0x7f9a540b03e0 2024-05-08T21:23:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.226+0000 7f9a5acd6700 1 -- 172.21.15.3:0/2162641064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a540b03e0 msgr2=0x7f9a540b0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.226+0000 7f9a5acd6700 1 --2- 172.21.15.3:0/2162641064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a540b03e0 0x7f9a540b0da0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9a40009840 tx=0x7f9a40009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.226+0000 7f9a5acd6700 1 -- 172.21.15.3:0/2162641064 shutdown_connections 2024-05-08T21:23:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.226+0000 7f9a5acd6700 1 --2- 172.21.15.3:0/2162641064 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a5415ed40 0x7f9a5409a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.226+0000 7f9a5acd6700 1 --2- 172.21.15.3:0/2162641064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a540b03e0 0x7f9a540b0da0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.226+0000 7f9a5acd6700 1 -- 172.21.15.3:0/2162641064 >> 172.21.15.3:0/2162641064 conn(0x7f9a540b2f30 msgr2=0x7f9a540b3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.226+0000 7f9a5acd6700 1 -- 172.21.15.3:0/2162641064 shutdown_connections 2024-05-08T21:23:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.226+0000 7f9a5acd6700 1 -- 172.21.15.3:0/2162641064 wait complete. 2024-05-08T21:23:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.227+0000 7f9a5acd6700 1 Processor -- start 2024-05-08T21:23:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.227+0000 7f9a5acd6700 1 -- start start 2024-05-08T21:23:41.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.228+0000 7f9a5acd6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a54099a50 0x7f9a5406d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.228+0000 7f9a5acd6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a540b03e0 0x7f9a540b5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.228+0000 7f9a5acd6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a5415ed40 0x7f9a54153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.228+0000 7f9a5acd6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a54153f60 con 0x7f9a540b03e0 2024-05-08T21:23:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.228+0000 7f9a5acd6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5406bcc0 con 0x7f9a5415ed40 2024-05-08T21:23:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.228+0000 7f9a5acd6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a540b6600 con 0x7f9a54099a50 2024-05-08T21:23:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.228+0000 7f9a53fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a540b03e0 0x7f9a540b5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.228+0000 7f9a53fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a540b03e0 0x7f9a540b5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35298/0 (socket says 172.21.15.3:35298) 2024-05-08T21:23:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.228+0000 7f9a53fff700 1 -- 172.21.15.3:0/3102510063 learned_addr learned my addr 172.21.15.3:0/3102510063 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.228+0000 7f9a58a72700 1 --2- 172.21.15.3:0/3102510063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a54099a50 0x7f9a5406d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.228+0000 7f9a59273700 1 --2- 172.21.15.3:0/3102510063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a5415ed40 0x7f9a54153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.229+0000 7f9a53fff700 1 -- 172.21.15.3:0/3102510063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a54099a50 msgr2=0x7f9a5406d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.229+0000 7f9a53fff700 1 --2- 172.21.15.3:0/3102510063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a54099a50 0x7f9a5406d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.229+0000 7f9a53fff700 1 -- 172.21.15.3:0/3102510063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a5415ed40 msgr2=0x7f9a54153a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.229+0000 7f9a53fff700 1 --2- 172.21.15.3:0/3102510063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a5415ed40 0x7f9a54153a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.229+0000 7f9a53fff700 1 -- 172.21.15.3:0/3102510063 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a480094a0 con 0x7f9a540b03e0 2024-05-08T21:23:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.229+0000 7f9a53fff700 1 --2- 172.21.15.3:0/3102510063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a540b03e0 0x7f9a540b5660 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9a40009b90 tx=0x7f9a40004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.229+0000 7f9a517fa700 1 -- 172.21.15.3:0/3102510063 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a400038a0 con 0x7f9a540b03e0 2024-05-08T21:23:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.229+0000 7f9a517fa700 1 -- 172.21.15.3:0/3102510063 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4001e070 con 0x7f9a540b03e0 2024-05-08T21:23:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.229+0000 7f9a5acd6700 1 -- 172.21.15.3:0/3102510063 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a400094a0 con 0x7f9a540b03e0 2024-05-08T21:23:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.229+0000 7f9a517fa700 1 -- 172.21.15.3:0/3102510063 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a40003c40 con 0x7f9a540b03e0 2024-05-08T21:23:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.229+0000 7f9a5acd6700 1 -- 172.21.15.3:0/3102510063 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a540b6040 con 0x7f9a540b03e0 2024-05-08T21:23:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.230+0000 7f9a5acd6700 1 -- 172.21.15.3:0/3102510063 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a5406ba70 con 0x7f9a540b03e0 2024-05-08T21:23:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.231+0000 7f9a517fa700 1 -- 172.21.15.3:0/3102510063 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9a4002d450 con 0x7f9a540b03e0 2024-05-08T21:23:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.231+0000 7f9a517fa700 1 --2- 172.21.15.3:0/3102510063 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9a3806ac90 0x7f9a3806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.231+0000 7f9a517fa700 1 -- 172.21.15.3:0/3102510063 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f9a4008a000 con 0x7f9a540b03e0 2024-05-08T21:23:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.231+0000 7f9a58a72700 1 --2- 172.21.15.3:0/3102510063 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9a3806ac90 0x7f9a3806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:41.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.232+0000 7f9a58a72700 1 --2- 172.21.15.3:0/3102510063 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9a3806ac90 0x7f9a3806d150 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9a48009c60 tx=0x7f9a48008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:41.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.236+0000 7f9a517fa700 1 -- 172.21.15.3:0/3102510063 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9a40055510 con 0x7f9a540b03e0 2024-05-08T21:23:41.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.448+0000 7f9a5acd6700 1 -- 172.21.15.3:0/3102510063 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9a5415f120 con 0x7f9a3806ac90 2024-05-08T21:23:41.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.450+0000 7f9a517fa700 1 -- 172.21.15.3:0/3102510063 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50862 (secure 0 0 0) 0x7f9a48009da0 con 0x7f9a3806ac90 2024-05-08T21:23:41.452 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:41.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.452+0000 7f9a5acd6700 1 -- 172.21.15.3:0/3102510063 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9a3806ac90 msgr2=0x7f9a3806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:41.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.452+0000 7f9a5acd6700 1 --2- 172.21.15.3:0/3102510063 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9a3806ac90 0x7f9a3806d150 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9a48009c60 tx=0x7f9a48008040 comp rx=0 tx=0).stop 2024-05-08T21:23:41.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.452+0000 7f9a5acd6700 1 -- 172.21.15.3:0/3102510063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a540b03e0 msgr2=0x7f9a540b5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:41.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.452+0000 7f9a5acd6700 1 --2- 172.21.15.3:0/3102510063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a540b03e0 0x7f9a540b5660 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9a40009b90 tx=0x7f9a40004750 comp rx=0 tx=0).stop 2024-05-08T21:23:41.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.453+0000 7f9a5acd6700 1 -- 172.21.15.3:0/3102510063 shutdown_connections 2024-05-08T21:23:41.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.453+0000 7f9a5acd6700 1 --2- 172.21.15.3:0/3102510063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a54099a50 0x7f9a5406d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.453+0000 7f9a5acd6700 1 --2- 172.21.15.3:0/3102510063 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f9a3806ac90 0x7f9a3806d150 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.453+0000 7f9a5acd6700 1 --2- 172.21.15.3:0/3102510063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a5415ed40 0x7f9a54153a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.453+0000 7f9a5acd6700 1 --2- 172.21.15.3:0/3102510063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a540b03e0 0x7f9a540b5660 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.453+0000 7f9a5acd6700 1 -- 172.21.15.3:0/3102510063 >> 172.21.15.3:0/3102510063 conn(0x7f9a540b2f30 msgr2=0x7f9a540af290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:41.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.453+0000 7f9a5acd6700 1 -- 172.21.15.3:0/3102510063 shutdown_connections 2024-05-08T21:23:41.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.453+0000 7f9a5acd6700 1 -- 172.21.15.3:0/3102510063 wait complete. 2024-05-08T21:23:41.456 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-08T21:23:41.467 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-05-08T21:23:40.557484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233716,"kb_used_data":17556,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754741004,"statfs":{"total":773094113280,"available":772854788096,"internally_reserved":0,"allocated":17977344,"data_stored":15543186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6322160,"num_objects":-15,"num_object_clones":0,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-15,"num_whiteouts":0,"num_read":-53,"num_read_kb":-39,"num_write":-50,"num_write_kb":-6193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.248664"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":186,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-08T21:23:29.908714+0000","last_change":"2024-05-08T21:21:42.333097+0000","last_active":"2024-05-08T21:23:29.908714+0000","last_peered":"2024-05-08T21:23:29.908714+0000","last_clean":"2024-05-08T21:23:29.908714+0000","last_became_active":"2024-05-08T21:21:42.332033+0000","last_became_peered":"2024-05-08T21:21:42.332033+0000","last_unstale":"2024-05-08T21:23:29.908714+0000","last_undegraded":"2024-05-08T21:23:29.908714+0000","last_fullsized":"2024-05-08T21:23:29.908714+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_clean_scrub_stamp":"2024-05-08T21:21:40.862622+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:39:24.772551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30152,"kb_used_data":3132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341688,"statfs":{"total":96636764160,"available":96605888512,"internally_reserved":0,"allocated":3207168,"data_stored":2903149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.54600000000000004}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.53600000000000003}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.57499999999999996}]},{"osd":3,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.45700000000000002}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.433},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.56000000000000005}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.49299999999999999}]},{"osd":6,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.47399999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29576,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342264,"statfs":{"total":96636764160,"available":96606478336,"internally_reserved":0,"allocated":2617344,"data_stored":2313363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.48199999999999998}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.39200000000000002}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.50800000000000001}]},{"osd":3,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.56200000000000006}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.44500000000000001}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.58599999999999997}]},{"osd":7,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.52100000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31608,"kb_used_data":4588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340232,"statfs":{"total":96636764160,"available":96604397568,"internally_reserved":0,"allocated":4698112,"data_stored":4403731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.314},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.36899999999999999}]},{"osd":2,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.33100000000000002}]},{"osd":4,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52300000000000002}]},{"osd":5,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.51100000000000001}]},{"osd":6,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.57099999999999995}]},{"osd":7,"last update":"Wed May 8 21:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.622}]}]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28104,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343736,"statfs":{"total":96636764160,"available":96607985664,"internally_reserved":0,"allocated":1110016,"data_stored":805941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.435},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.54800000000000004}]},{"osd":2,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.59899999999999998}]},{"osd":3,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.68400000000000005}]},{"osd":4,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.80000000000000004}]},{"osd":5,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.82799999999999996}]},{"osd":6,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.76000000000000001}]},{"osd":7,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.874},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.89700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":245467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.379},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.443}]},{"osd":1,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38900000000000001}]},{"osd":3,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.40699999999999997}]},{"osd":4,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.67200000000000004}]},{"osd":5,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.755}]},{"osd":6,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.74399999999999999}]},{"osd":7,"last update":"Wed May 8 21:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.64800000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":524288,"data_stored":216177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":2.7919999999999998,"5min":2.7919999999999998,"15min":2.7919999999999998},"last":0.375}]},{"osd":1,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.433},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.40300000000000002}]},{"osd":2,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":2.9369999999999998,"5min":2.9369999999999998,"15min":2.9369999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.38600000000000001}]},{"osd":4,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.8380000000000001,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.9060000000000001,"5min":2.9060000000000001,"15min":2.9060000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.46700000000000003}]},{"osd":6,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":2.8730000000000002,"5min":2.8730000000000002,"15min":2.8730000000000002},"last":0.50600000000000001}]},{"osd":7,"last update":"Wed May 8 21:22:40 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.495},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.63300000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27560,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344280,"statfs":{"total":96636764160,"available":96608542720,"internally_reserved":0,"allocated":552960,"data_stored":245445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.39700000000000002}]},{"osd":1,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.58999999999999997}]},{"osd":2,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.56999999999999995}]},{"osd":3,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.433},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.41899999999999998}]},{"osd":5,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52400000000000002}]},{"osd":6,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.54800000000000004}]},{"osd":7,"last update":"Wed May 8 21:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.49399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31620,"kb_used_data":4600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340220,"statfs":{"total":96636764160,"available":96604385280,"internally_reserved":0,"allocated":4710400,"data_stored":4409913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.38900000000000001}]},{"osd":1,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.57599999999999996}]},{"osd":2,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.48999999999999999}]},{"osd":3,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.504}]},{"osd":4,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.375}]},{"osd":6,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.53600000000000003}]},{"osd":7,"last update":"Wed May 8 21:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T21:23:41.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T21:23:41.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.604+0000 7f4a6da42700 1 Processor -- start 2024-05-08T21:23:41.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.604+0000 7f4a6da42700 1 -- start start 2024-05-08T21:23:41.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.605+0000 7f4a6da42700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a6815a670 0x7f4a680acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:41.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.605+0000 7f4a6da42700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4a6806e530 0x7f4a6806e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:41.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.605+0000 7f4a6da42700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a680ae2a0 con 0x7f4a680ae730 2024-05-08T21:23:41.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.605+0000 7f4a6da42700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a6806ef20 con 0x7f4a6806e530 2024-05-08T21:23:41.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.605+0000 7f4a6da42700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a680ad690 con 0x7f4a6815a670 2024-05-08T21:23:41.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.605+0000 7f4a677fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4a6806e530 0x7f4a6806e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:41.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.605+0000 7f4a667fc700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f4a680ae730 0x7f4a6815ea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.3:51218/0 (socket says 172.21.15.3:51218) 2024-05-08T21:23:41.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.605+0000 7f4a667fc700 1 -- 172.21.15.3:0/1463590027 learned_addr learned my addr 172.21.15.3:0/1463590027 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:41.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.605+0000 7f4a66ffd700 1 --2- 172.21.15.3:0/1463590027 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a6815a670 0x7f4a680acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:41.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.605+0000 7f4a65ffb700 1 -- 172.21.15.3:0/1463590027 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187529831 0 0) 0x7f4a680ae2a0 con 0x7f4a680ae730 2024-05-08T21:23:41.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.605+0000 7f4a65ffb700 1 -- 172.21.15.3:0/1463590027 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a500035b0 con 0x7f4a680ae730 2024-05-08T21:23:41.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.606+0000 7f4a66ffd700 1 -- 172.21.15.3:0/1463590027 >> v2:172.21.15.186:3300/0 conn(0x7f4a6806e530 msgr2=0x7f4a6806e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:41.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.606+0000 7f4a66ffd700 1 --2- 172.21.15.3:0/1463590027 >> v2:172.21.15.186:3300/0 conn(0x7f4a6806e530 0x7f4a6806e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.606+0000 7f4a66ffd700 1 -- 172.21.15.3:0/1463590027 >> v1:172.21.15.186:6789/0 conn(0x7f4a680ae730 legacy=0x7f4a6815ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:41.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.606+0000 7f4a66ffd700 1 -- 172.21.15.3:0/1463590027 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a680ad970 con 0x7f4a6815a670 2024-05-08T21:23:41.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.606+0000 7f4a677fe700 1 --2- 172.21.15.3:0/1463590027 >> v2:172.21.15.186:3300/0 conn(0x7f4a6806e530 0x7f4a6806e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:41.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.606+0000 7f4a66ffd700 1 --2- 172.21.15.3:0/1463590027 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a6815a670 0x7f4a680acfb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009840 tx=0x7f4a58009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e7d0b234cdad0e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:41.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.606+0000 7f4a65ffb700 1 -- 172.21.15.3:0/1463590027 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a58011830 con 0x7f4a6815a670 2024-05-08T21:23:41.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.606+0000 7f4a65ffb700 1 -- 172.21.15.3:0/1463590027 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a58011990 con 0x7f4a6815a670 2024-05-08T21:23:41.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.606+0000 7f4a65ffb700 1 -- 172.21.15.3:0/1463590027 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a58011d20 con 0x7f4a6815a670 2024-05-08T21:23:41.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.607+0000 7f4a6da42700 1 -- 172.21.15.3:0/1463590027 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a6815a670 msgr2=0x7f4a680acfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:41.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.607+0000 7f4a6da42700 1 --2- 172.21.15.3:0/1463590027 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a6815a670 0x7f4a680acfb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009840 tx=0x7f4a58009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:41.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.607+0000 7f4a6da42700 1 -- 172.21.15.3:0/1463590027 shutdown_connections 2024-05-08T21:23:41.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.607+0000 7f4a6da42700 1 --2- 172.21.15.3:0/1463590027 >> v2:172.21.15.186:3300/0 conn(0x7f4a6806e530 0x7f4a6806e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.607+0000 7f4a6da42700 1 --2- 172.21.15.3:0/1463590027 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a6815a670 0x7f4a680acfb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.607+0000 7f4a6da42700 1 -- 172.21.15.3:0/1463590027 >> 172.21.15.3:0/1463590027 conn(0x7f4a680cf270 msgr2=0x7f4a680cf650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:41.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.607+0000 7f4a6da42700 1 -- 172.21.15.3:0/1463590027 shutdown_connections 2024-05-08T21:23:41.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.607+0000 7f4a6da42700 1 -- 172.21.15.3:0/1463590027 wait complete. 2024-05-08T21:23:41.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.608+0000 7f4a6da42700 1 Processor -- start 2024-05-08T21:23:41.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.608+0000 7f4a6da42700 1 -- start start 2024-05-08T21:23:41.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.609+0000 7f4a6da42700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a6806e530 0x7f4a68150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:41.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.609+0000 7f4a6da42700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a680ae730 0x7f4a68145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:41.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.609+0000 7f4a6da42700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a680ce360 0x7f4a68154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:41.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.609+0000 7f4a6da42700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a68156ef0 con 0x7f4a6806e530 2024-05-08T21:23:41.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.609+0000 7f4a6da42700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a681516e0 con 0x7f4a680ce360 2024-05-08T21:23:41.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.609+0000 7f4a6da42700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a681507c0 con 0x7f4a680ae730 2024-05-08T21:23:41.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.609+0000 7f4a66ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a6806e530 0x7f4a68150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:41.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.609+0000 7f4a66ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a6806e530 0x7f4a68150280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34482/0 (socket says 172.21.15.3:34482) 2024-05-08T21:23:41.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.609+0000 7f4a66ffd700 1 -- 172.21.15.3:0/2913532210 learned_addr learned my addr 172.21.15.3:0/2913532210 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:41.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.609+0000 7f4a677fe700 1 --2- 172.21.15.3:0/2913532210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a680ce360 0x7f4a68154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:41.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.609+0000 7f4a667fc700 1 --2- 172.21.15.3:0/2913532210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a680ae730 0x7f4a68145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:41.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.610+0000 7f4a66ffd700 1 -- 172.21.15.3:0/2913532210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a680ae730 msgr2=0x7f4a68145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:41.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.610+0000 7f4a66ffd700 1 --2- 172.21.15.3:0/2913532210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a680ae730 0x7f4a68145f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.610+0000 7f4a66ffd700 1 -- 172.21.15.3:0/2913532210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a680ce360 msgr2=0x7f4a68154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:41.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.610+0000 7f4a66ffd700 1 --2- 172.21.15.3:0/2913532210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a680ce360 0x7f4a68154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.610+0000 7f4a66ffd700 1 -- 172.21.15.3:0/2913532210 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a60009a70 con 0x7f4a6806e530 2024-05-08T21:23:41.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.610+0000 7f4a66ffd700 1 --2- 172.21.15.3:0/2913532210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a6806e530 0x7f4a68150280 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009fd0 tx=0x7f4a58004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:41.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.610+0000 7f4a4ffff700 1 -- 172.21.15.3:0/2913532210 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a58004590 con 0x7f4a6806e530 2024-05-08T21:23:41.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.610+0000 7f4a6da42700 1 -- 172.21.15.3:0/2913532210 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a580094a0 con 0x7f4a6806e530 2024-05-08T21:23:41.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.610+0000 7f4a4ffff700 1 -- 172.21.15.3:0/2913532210 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5801e070 con 0x7f4a6806e530 2024-05-08T21:23:41.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.610+0000 7f4a4ffff700 1 -- 172.21.15.3:0/2913532210 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a580038c0 con 0x7f4a6806e530 2024-05-08T21:23:41.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.610+0000 7f4a6da42700 1 -- 172.21.15.3:0/2913532210 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a68150d40 con 0x7f4a6806e530 2024-05-08T21:23:41.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.611+0000 7f4a6da42700 1 -- 172.21.15.3:0/2913532210 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a680678f0 con 0x7f4a6806e530 2024-05-08T21:23:41.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.612+0000 7f4a4ffff700 1 -- 172.21.15.3:0/2913532210 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4a58003a20 con 0x7f4a6806e530 2024-05-08T21:23:41.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.612+0000 7f4a4ffff700 1 --2- 172.21.15.3:0/2913532210 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a5406ac90 0x7f4a5406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:41.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.612+0000 7f4a4ffff700 1 -- 172.21.15.3:0/2913532210 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f4a58089e30 con 0x7f4a6806e530 2024-05-08T21:23:41.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.612+0000 7f4a667fc700 1 --2- 172.21.15.3:0/2913532210 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a5406ac90 0x7f4a5406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:41.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.613+0000 7f4a667fc700 1 --2- 172.21.15.3:0/2913532210 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a5406ac90 0x7f4a5406d150 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4a68159d60 tx=0x7f4a5c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:41.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.617+0000 7f4a4ffff700 1 -- 172.21.15.3:0/2913532210 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f4a580552c0 con 0x7f4a6806e530 2024-05-08T21:23:41.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.836+0000 7f4a6da42700 1 -- 172.21.15.3:0/2913532210 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a6806cf10 con 0x7f4a6806e530 2024-05-08T21:23:41.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.837+0000 7f4a4ffff700 1 -- 172.21.15.3:0/2913532210 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+12600 (secure 0 0 0) 0x7f4a58020090 con 0x7f4a6806e530 2024-05-08T21:23:41.839 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:41.839 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":46,"fsid":"d0f58275-758f-4470-b003-84d6abe25489","created":"2024-05-08T21:21:28.545722+0000","modified":"2024-05-08T21:23:37.320175+0000","last_up_change":"2024-05-08T21:21:39.643574+0000","last_in_change":"2024-05-08T21:21:31.108524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T21:21:40.595836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4e85793-ae1b-4a72-a1a8-2f1e299373ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":47120},{"type":"v1","addr":"172.21.15.3:6825","nonce":47120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":47120},{"type":"v1","addr":"172.21.15.3:6827","nonce":47120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":47120},{"type":"v1","addr":"172.21.15.3:6831","nonce":47120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":47120},{"type":"v1","addr":"172.21.15.3:6829","nonce":47120}]},"public_addr":"172.21.15.3:6825/47120","cluster_addr":"172.21.15.3:6827/47120","heartbeat_back_addr":"172.21.15.3:6831/47120","heartbeat_front_addr":"172.21.15.3:6829/47120","state":["exists","up"]},{"osd":1,"uuid":"05adc715-3b7c-4b51-966c-b3c4c4269ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":47119},{"type":"v1","addr":"172.21.15.3:6817","nonce":47119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":47119},{"type":"v1","addr":"172.21.15.3:6819","nonce":47119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":47119},{"type":"v1","addr":"172.21.15.3:6823","nonce":47119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":47119},{"type":"v1","addr":"172.21.15.3:6821","nonce":47119}]},"public_addr":"172.21.15.3:6817/47119","cluster_addr":"172.21.15.3:6819/47119","heartbeat_back_addr":"172.21.15.3:6823/47119","heartbeat_front_addr":"172.21.15.3:6821/47119","state":["exists","up"]},{"osd":2,"uuid":"d9717d7b-dfd4-4826-b6db-6bb3e61511fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":47117},{"type":"v1","addr":"172.21.15.3:6809","nonce":47117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":47117},{"type":"v1","addr":"172.21.15.3:6811","nonce":47117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":47117},{"type":"v1","addr":"172.21.15.3:6815","nonce":47117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":47117},{"type":"v1","addr":"172.21.15.3:6813","nonce":47117}]},"public_addr":"172.21.15.3:6809/47117","cluster_addr":"172.21.15.3:6811/47117","heartbeat_back_addr":"172.21.15.3:6815/47117","heartbeat_front_addr":"172.21.15.3:6813/47117","state":["exists","up"]},{"osd":3,"uuid":"1a904ea7-5acf-4af7-b73d-c4f0070ad767","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":47118},{"type":"v1","addr":"172.21.15.3:6801","nonce":47118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":47118},{"type":"v1","addr":"172.21.15.3:6803","nonce":47118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":47118},{"type":"v1","addr":"172.21.15.3:6807","nonce":47118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":47118},{"type":"v1","addr":"172.21.15.3:6805","nonce":47118}]},"public_addr":"172.21.15.3:6801/47118","cluster_addr":"172.21.15.3:6803/47118","heartbeat_back_addr":"172.21.15.3:6807/47118","heartbeat_front_addr":"172.21.15.3:6805/47118","state":["exists","up"]},{"osd":4,"uuid":"3f01306b-2f27-4185-b978-bff25e857c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":47344},{"type":"v1","addr":"172.21.15.186:6813","nonce":47344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":47344},{"type":"v1","addr":"172.21.15.186:6819","nonce":47344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6827","nonce":47344},{"type":"v1","addr":"172.21.15.186:6830","nonce":47344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":47344},{"type":"v1","addr":"172.21.15.186:6824","nonce":47344}]},"public_addr":"172.21.15.186:6813/47344","cluster_addr":"172.21.15.186:6819/47344","heartbeat_back_addr":"172.21.15.186:6830/47344","heartbeat_front_addr":"172.21.15.186:6824/47344","state":["exists","up"]},{"osd":5,"uuid":"36f775b1-4bc3-4889-958b-5d13b5855b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":47340},{"type":"v1","addr":"172.21.15.186:6801","nonce":47340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":47340},{"type":"v1","addr":"172.21.15.186:6803","nonce":47340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":47340},{"type":"v1","addr":"172.21.15.186:6807","nonce":47340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":47340},{"type":"v1","addr":"172.21.15.186:6805","nonce":47340}]},"public_addr":"172.21.15.186:6801/47340","cluster_addr":"172.21.15.186:6803/47340","heartbeat_back_addr":"172.21.15.186:6807/47340","heartbeat_front_addr":"172.21.15.186:6805/47340","state":["exists","up"]},{"osd":6,"uuid":"2869e118-04fe-4577-91f4-80c935cd259e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":47351},{"type":"v1","addr":"172.21.15.186:6811","nonce":47351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":47351},{"type":"v1","addr":"172.21.15.186:6817","nonce":47351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":47351},{"type":"v1","addr":"172.21.15.186:6831","nonce":47351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":47351},{"type":"v1","addr":"172.21.15.186:6825","nonce":47351}]},"public_addr":"172.21.15.186:6811/47351","cluster_addr":"172.21.15.186:6817/47351","heartbeat_back_addr":"172.21.15.186:6831/47351","heartbeat_front_addr":"172.21.15.186:6825/47351","state":["exists","up"]},{"osd":7,"uuid":"89b4d094-f7dd-4110-8601-0263df2c6cfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":47341},{"type":"v1","addr":"172.21.15.186:6812","nonce":47341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6815","nonce":47341},{"type":"v1","addr":"172.21.15.186:6818","nonce":47341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":47341},{"type":"v1","addr":"172.21.15.186:6829","nonce":47341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":47341},{"type":"v1","addr":"172.21.15.186:6823","nonce":47341}]},"public_addr":"172.21.15.186:6812/47341","cluster_addr":"172.21.15.186:6818/47341","heartbeat_back_addr":"172.21.15.186:6829/47341","heartbeat_front_addr":"172.21.15.186:6823/47341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.342241+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.338629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.324054+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.300108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.323504+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.251661+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.298629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T21:21:36.264629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.186:6837/2000072997":"2024-05-09T21:23:34.306550+0000","172.21.15.186:6835/2000072997":"2024-05-09T21:23:34.306550+0000","172.21.15.3:6832/3599481370":"2024-05-09T21:23:01.563392+0000","172.21.15.3:6834/3599481370":"2024-05-09T21:23:01.563392+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T21:23:41.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.840+0000 7f4a6da42700 1 -- 172.21.15.3:0/2913532210 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a5406ac90 msgr2=0x7f4a5406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:41.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.840+0000 7f4a6da42700 1 --2- 172.21.15.3:0/2913532210 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a5406ac90 0x7f4a5406d150 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4a68159d60 tx=0x7f4a5c009040 comp rx=0 tx=0).stop 2024-05-08T21:23:41.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.840+0000 7f4a6da42700 1 -- 172.21.15.3:0/2913532210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a6806e530 msgr2=0x7f4a68150280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:41.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.840+0000 7f4a6da42700 1 --2- 172.21.15.3:0/2913532210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a6806e530 0x7f4a68150280 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009fd0 tx=0x7f4a58004750 comp rx=0 tx=0).stop 2024-05-08T21:23:41.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.840+0000 7f4a6da42700 1 -- 172.21.15.3:0/2913532210 shutdown_connections 2024-05-08T21:23:41.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.840+0000 7f4a6da42700 1 --2- 172.21.15.3:0/2913532210 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a5406ac90 0x7f4a5406d150 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.840+0000 7f4a6da42700 1 --2- 172.21.15.3:0/2913532210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a680ce360 0x7f4a68154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.840+0000 7f4a6da42700 1 --2- 172.21.15.3:0/2913532210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a6806e530 0x7f4a68150280 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.840+0000 7f4a6da42700 1 --2- 172.21.15.3:0/2913532210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a680ae730 0x7f4a68145f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:41.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.840+0000 7f4a6da42700 1 -- 172.21.15.3:0/2913532210 >> 172.21.15.3:0/2913532210 conn(0x7f4a680cf270 msgr2=0x7f4a68070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:41.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.840+0000 7f4a6da42700 1 -- 172.21.15.3:0/2913532210 shutdown_connections 2024-05-08T21:23:41.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:41.840+0000 7f4a6da42700 1 -- 172.21.15.3:0/2913532210 wait complete. 2024-05-08T21:23:42.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-08T21:23:42.154 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-08T21:23:42.857 INFO:tasks.ceph:Scrubbing osd.0 2024-05-08T21:23:42.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-08T21:23:43.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.005+0000 7f4a9dfca700 1 Processor -- start 2024-05-08T21:23:43.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.005+0000 7f4a9dfca700 1 -- start start 2024-05-08T21:23:43.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.006+0000 7f4a9dfca700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4a980cff30 0x7f4a980d0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.006+0000 7f4a9dfca700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4a980d3d20 0x7f4a980ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.006+0000 7f4a9dfca700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a980ce7a0 0x7f4a980cec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.006+0000 7f4a9dfca700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a980d0850 con 0x7f4a980cff30 2024-05-08T21:23:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.006+0000 7f4a9dfca700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a980cf610 con 0x7f4a980d3d20 2024-05-08T21:23:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.006+0000 7f4a9dfca700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a980d1fc0 con 0x7f4a980ce7a0 2024-05-08T21:23:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.006+0000 7f4a977fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4a980cff30 0x7f4a980d0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.006+0000 7f4a977fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4a980cff30 0x7f4a980d0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34488/0 (socket says 172.21.15.3:34488) 2024-05-08T21:23:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.006+0000 7f4a977fe700 1 -- 172.21.15.3:0/4084923418 learned_addr learned my addr 172.21.15.3:0/4084923418 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.006+0000 7f4a97fff700 1 --2- 172.21.15.3:0/4084923418 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a980ce7a0 0x7f4a980cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.006+0000 7f4a96ffd700 1 --2- 172.21.15.3:0/4084923418 >> v2:172.21.15.186:3300/0 conn(0x7f4a980d3d20 0x7f4a980ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.006+0000 7f4a977fe700 1 -- 172.21.15.3:0/4084923418 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a980ce7a0 msgr2=0x7f4a980cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.006+0000 7f4a977fe700 1 --2- 172.21.15.3:0/4084923418 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a980ce7a0 0x7f4a980cec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.006+0000 7f4a977fe700 1 -- 172.21.15.3:0/4084923418 >> v2:172.21.15.186:3300/0 conn(0x7f4a980d3d20 msgr2=0x7f4a980ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.006+0000 7f4a977fe700 1 --2- 172.21.15.3:0/4084923418 >> v2:172.21.15.186:3300/0 conn(0x7f4a980d3d20 0x7f4a980ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.006+0000 7f4a977fe700 1 -- 172.21.15.3:0/4084923418 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a980d22a0 con 0x7f4a980cff30 2024-05-08T21:23:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.007+0000 7f4a977fe700 1 --2- 172.21.15.3:0/4084923418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a980cff30 0x7f4a980d0310 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4a9808b9e0 tx=0x7f4a88009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d90a9d592e1b96 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.007+0000 7f4a967fc700 1 -- 172.21.15.3:0/4084923418 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8800b870 con 0x7f4a980cff30 2024-05-08T21:23:43.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.007+0000 7f4a967fc700 1 -- 172.21.15.3:0/4084923418 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8800b9d0 con 0x7f4a980cff30 2024-05-08T21:23:43.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.007+0000 7f4a967fc700 1 -- 172.21.15.3:0/4084923418 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8800bd60 con 0x7f4a980cff30 2024-05-08T21:23:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.007+0000 7f4a9dfca700 1 -- 172.21.15.3:0/4084923418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a980cff30 msgr2=0x7f4a980d0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.007+0000 7f4a9dfca700 1 --2- 172.21.15.3:0/4084923418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a980cff30 0x7f4a980d0310 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4a9808b9e0 tx=0x7f4a88009bd0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.007+0000 7f4a9dfca700 1 -- 172.21.15.3:0/4084923418 shutdown_connections 2024-05-08T21:23:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.007+0000 7f4a9dfca700 1 --2- 172.21.15.3:0/4084923418 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a980ce7a0 0x7f4a980cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.007+0000 7f4a9dfca700 1 --2- 172.21.15.3:0/4084923418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a980cff30 0x7f4a980d0310 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.007+0000 7f4a9dfca700 1 --2- 172.21.15.3:0/4084923418 >> v2:172.21.15.186:3300/0 conn(0x7f4a980d3d20 0x7f4a980ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.007+0000 7f4a9dfca700 1 -- 172.21.15.3:0/4084923418 >> 172.21.15.3:0/4084923418 conn(0x7f4a9808afe0 msgr2=0x7f4a980d3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.007+0000 7f4a9dfca700 1 -- 172.21.15.3:0/4084923418 shutdown_connections 2024-05-08T21:23:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.008+0000 7f4a9dfca700 1 -- 172.21.15.3:0/4084923418 wait complete. 2024-05-08T21:23:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.009+0000 7f4a9dfca700 1 Processor -- start 2024-05-08T21:23:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.009+0000 7f4a9dfca700 1 -- start start 2024-05-08T21:23:43.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a9dfca700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a980ce7a0 0x7f4a980b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a9dfca700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a980cff30 0x7f4a9814baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a9dfca700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a980d3d20 0x7f4a9815b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a9dfca700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a980b73a0 con 0x7f4a980cff30 2024-05-08T21:23:43.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a9dfca700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9814bfe0 con 0x7f4a980ce7a0 2024-05-08T21:23:43.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a9dfca700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a980b9260 con 0x7f4a980d3d20 2024-05-08T21:23:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a96ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a980cff30 0x7f4a9814baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a96ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a980cff30 0x7f4a9814baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34490/0 (socket says 172.21.15.3:34490) 2024-05-08T21:23:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a96ffd700 1 -- 172.21.15.3:0/762536696 learned_addr learned my addr 172.21.15.3:0/762536696 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a977fe700 1 --2- 172.21.15.3:0/762536696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a980ce7a0 0x7f4a980b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a97fff700 1 --2- 172.21.15.3:0/762536696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a980d3d20 0x7f4a9815b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a96ffd700 1 -- 172.21.15.3:0/762536696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a980d3d20 msgr2=0x7f4a9815b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a96ffd700 1 --2- 172.21.15.3:0/762536696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a980d3d20 0x7f4a9815b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a96ffd700 1 -- 172.21.15.3:0/762536696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a980ce7a0 msgr2=0x7f4a980b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a96ffd700 1 --2- 172.21.15.3:0/762536696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a980ce7a0 0x7f4a980b77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a96ffd700 1 -- 172.21.15.3:0/762536696 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a88009550 con 0x7f4a980cff30 2024-05-08T21:23:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.010+0000 7f4a97fff700 1 --2- 172.21.15.3:0/762536696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a980d3d20 0x7f4a9815b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.011+0000 7f4a96ffd700 1 --2- 172.21.15.3:0/762536696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a980cff30 0x7f4a9814baa0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00e810 tx=0x7f4a8c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.011+0000 7f4a7ffff700 1 -- 172.21.15.3:0/762536696 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c013610 con 0x7f4a980cff30 2024-05-08T21:23:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.011+0000 7f4a7ffff700 1 -- 172.21.15.3:0/762536696 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8c013770 con 0x7f4a980cff30 2024-05-08T21:23:43.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.011+0000 7f4a9dfca700 1 -- 172.21.15.3:0/762536696 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a980b8f80 con 0x7f4a980cff30 2024-05-08T21:23:43.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.011+0000 7f4a7ffff700 1 -- 172.21.15.3:0/762536696 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c013b20 con 0x7f4a980cff30 2024-05-08T21:23:43.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.011+0000 7f4a9dfca700 1 -- 172.21.15.3:0/762536696 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a980b6fc0 con 0x7f4a980cff30 2024-05-08T21:23:43.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.012+0000 7f4a9dfca700 1 -- 172.21.15.3:0/762536696 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4a980ca150 con 0x7f4a980cff30 2024-05-08T21:23:43.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.012+0000 7f4a7ffff700 1 -- 172.21.15.3:0/762536696 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4a8c013c80 con 0x7f4a980cff30 2024-05-08T21:23:43.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.013+0000 7f4a7ffff700 1 --2- 172.21.15.3:0/762536696 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a8406ac90 0x7f4a8406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.013+0000 7f4a7ffff700 1 -- 172.21.15.3:0/762536696 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f4a8c088d00 con 0x7f4a980cff30 2024-05-08T21:23:43.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.013+0000 7f4a7ffff700 1 --2- 172.21.15.3:0/762536696 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x7f4a84070b90 0x7f4a84072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.013+0000 7f4a7ffff700 1 -- 172.21.15.3:0/762536696 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a84073660 con 0x7f4a84070b90 2024-05-08T21:23:43.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.013+0000 7f4a7ffff700 1 -- 172.21.15.3:0/762536696 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a8c08b3c0 con 0x7f4a980cff30 2024-05-08T21:23:43.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.013+0000 7f4a977fe700 1 --2- 172.21.15.3:0/762536696 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a8406ac90 0x7f4a8406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.013+0000 7f4a97fff700 1 --2- 172.21.15.3:0/762536696 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x7f4a84070b90 0x7f4a84072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.014+0000 7f4a97fff700 1 --2- 172.21.15.3:0/762536696 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x7f4a84070b90 0x7f4a84072fb0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:43.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.014+0000 7f4a977fe700 1 --2- 172.21.15.3:0/762536696 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a8406ac90 0x7f4a8406d150 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4a88000c00 tx=0x7f4a88008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:43.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.014+0000 7f4a7ffff700 1 -- 172.21.15.3:0/762536696 <== osd.0 v2:172.21.15.3:6824/47120 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f4a84073660 con 0x7f4a84070b90 2024-05-08T21:23:43.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.030+0000 7f4a9dfca700 1 -- 172.21.15.3:0/762536696 --> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4a980cf8f0 con 0x7f4a84070b90 2024-05-08T21:23:43.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.030+0000 7f4a7ffff700 1 -- 172.21.15.3:0/762536696 <== osd.0 v2:172.21.15.3:6824/47120 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4a90009a60 con 0x7f4a84070b90 2024-05-08T21:23:43.032 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:23:43.032 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T21:23:43.032 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:23:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.030+0000 7f4a9dfca700 1 -- 172.21.15.3:0/762536696 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x7f4a84070b90 msgr2=0x7f4a84072fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.030+0000 7f4a9dfca700 1 --2- 172.21.15.3:0/762536696 >> [v2:172.21.15.3:6824/47120,v1:172.21.15.3:6825/47120] conn(0x7f4a84070b90 0x7f4a84072fb0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.031+0000 7f4a9dfca700 1 -- 172.21.15.3:0/762536696 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a8406ac90 msgr2=0x7f4a8406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.031+0000 7f4a9dfca700 1 --2- 172.21.15.3:0/762536696 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4a8406ac90 0x7f4a8406d150 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4a88000c00 tx=0x7f4a88008030 comp rx=0 tx=0).stop 2024-05-08T21:23:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.031+0000 7f4a9dfca700 1 -- 172.21.15.3:0/762536696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a980cff30 msgr2=0x7f4a9814baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.031+0000 7f4a9dfca700 1 --2- 172.21.15.3:0/762536696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a980cff30 0x7f4a9814baa0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00e810 tx=0x7f4a8c00ebd0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.031+0000 7f4a97fff700 1 -- 172.21.15.3:0/762536696 reap_dead start 2024-05-08T21:23:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.031+0000 7f4a9dfca700 1 -- 172.21.15.3:0/762536696 shutdown_connections 2024-05-08T21:23:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.031+0000 7f4a9dfca700 1 -- 172.21.15.3:0/762536696 >> 172.21.15.3:0/762536696 conn(0x7f4a9808afe0 msgr2=0x7f4a98160410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:43.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.031+0000 7f4a9dfca700 1 -- 172.21.15.3:0/762536696 shutdown_connections 2024-05-08T21:23:43.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.031+0000 7f4a9dfca700 1 -- 172.21.15.3:0/762536696 wait complete. 2024-05-08T21:23:43.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-05-08T21:23:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.181+0000 7f90576cf700 1 Processor -- start 2024-05-08T21:23:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.181+0000 7f90576cf700 1 -- start start 2024-05-08T21:23:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.182+0000 7f90576cf700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f905009a160 0x7f90500b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.182+0000 7f90576cf700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90500b4280 con 0x7f905015ba00 2024-05-08T21:23:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.182+0000 7f90576cf700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90500b4560 con 0x7f90500b2510 2024-05-08T21:23:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.182+0000 7f90576cf700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90500cccf0 con 0x7f905009a160 2024-05-08T21:23:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.182+0000 7f905546b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f905015ba00 0x7f9050099a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34136/0 (socket says 172.21.15.3:34136) 2024-05-08T21:23:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.182+0000 7f905546b700 1 -- 172.21.15.3:0/637913565 learned_addr learned my addr 172.21.15.3:0/637913565 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:43.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9054c6a700 1 --2- 172.21.15.3:0/637913565 >> v2:172.21.15.186:3300/0 conn(0x7f905009a160 0x7f90500b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 -- 172.21.15.3:0/637913565 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618465713 0 0) 0x7f90500b4280 con 0x7f905015ba00 2024-05-08T21:23:43.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 -- 172.21.15.3:0/637913565 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f903c0035b0 con 0x7f905015ba00 2024-05-08T21:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 -- 172.21.15.3:0/637913565 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119311510 0 0) 0x7f90500b4560 con 0x7f90500b2510 2024-05-08T21:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 -- 172.21.15.3:0/637913565 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90500b4280 con 0x7f90500b2510 2024-05-08T21:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 -- 172.21.15.3:0/637913565 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2660289100 0 0) 0x7f903c0035b0 con 0x7f905015ba00 2024-05-08T21:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 -- 172.21.15.3:0/637913565 >> v2:172.21.15.186:3300/0 conn(0x7f905009a160 msgr2=0x7f90500b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 --2- 172.21.15.3:0/637913565 >> v2:172.21.15.186:3300/0 conn(0x7f905009a160 0x7f90500b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 -- 172.21.15.3:0/637913565 >> v1:172.21.15.186:6789/0 conn(0x7f90500b2510 legacy=0x7f90500b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 -- 172.21.15.3:0/637913565 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90500ccfd0 con 0x7f905015ba00 2024-05-08T21:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 -- 172.21.15.3:0/637913565 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f904c001670 con 0x7f905015ba00 2024-05-08T21:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 -- 172.21.15.3:0/637913565 >> v1:172.21.15.3:6789/0 conn(0x7f905015ba00 legacy=0x7f9050099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 --2- 172.21.15.3:0/637913565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f903c003be0 0x7f903c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 --2- 172.21.15.3:0/637913565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f903c004850 0x7f903c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 --2- 172.21.15.3:0/637913565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f903c005210 0x7f903c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 -- 172.21.15.3:0/637913565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90500cccf0 con 0x7f903c005210 2024-05-08T21:23:43.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 -- 172.21.15.3:0/637913565 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c00c0e0 con 0x7f903c003be0 2024-05-08T21:23:43.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9047fff700 1 -- 172.21.15.3:0/637913565 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903c00c2a0 con 0x7f903c004850 2024-05-08T21:23:43.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9055c6c700 1 --2- 172.21.15.3:0/637913565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f903c005210 0x7f903c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f905546b700 1 --2- 172.21.15.3:0/637913565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f903c003be0 0x7f903c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.183+0000 7f9054c6a700 1 --2- 172.21.15.3:0/637913565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f903c004850 0x7f903c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.184+0000 7f9055c6c700 1 -- 172.21.15.3:0/637913565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f903c004850 msgr2=0x7f903c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:43.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.184+0000 7f9055c6c700 1 --2- 172.21.15.3:0/637913565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f903c004850 0x7f903c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.184+0000 7f9055c6c700 1 -- 172.21.15.3:0/637913565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f903c003be0 msgr2=0x7f903c003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:43.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.184+0000 7f9055c6c700 1 --2- 172.21.15.3:0/637913565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f903c003be0 0x7f903c003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.184+0000 7f9055c6c700 1 -- 172.21.15.3:0/637913565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903c0035b0 con 0x7f903c005210 2024-05-08T21:23:43.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.184+0000 7f9055c6c700 1 -- 172.21.15.3:0/637913565 reap_dead start 2024-05-08T21:23:43.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.184+0000 7f9055c6c700 1 --2- 172.21.15.3:0/637913565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f903c005210 0x7f903c00ba10 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f904800c2f0 tx=0x7f903c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aef3375b07a9f64 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:43.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.184+0000 7f9047fff700 1 -- 172.21.15.3:0/637913565 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9048001f30 con 0x7f903c005210 2024-05-08T21:23:43.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.184+0000 7f9047fff700 1 -- 172.21.15.3:0/637913565 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9048003a00 con 0x7f903c005210 2024-05-08T21:23:43.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.184+0000 7f9047fff700 1 -- 172.21.15.3:0/637913565 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9048003db0 con 0x7f903c005210 2024-05-08T21:23:43.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.184+0000 7f90576cf700 1 -- 172.21.15.3:0/637913565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f903c005210 msgr2=0x7f903c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.184+0000 7f90576cf700 1 --2- 172.21.15.3:0/637913565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f903c005210 0x7f903c00ba10 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f904800c2f0 tx=0x7f903c005880 comp rx=0 tx=0).stop 2024-05-08T21:23:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.185+0000 7f90576cf700 1 -- 172.21.15.3:0/637913565 shutdown_connections 2024-05-08T21:23:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.185+0000 7f90576cf700 1 --2- 172.21.15.3:0/637913565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f903c005210 0x7f903c00ba10 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.185+0000 7f90576cf700 1 -- 172.21.15.3:0/637913565 >> 172.21.15.3:0/637913565 conn(0x7f90500b2f20 msgr2=0x7f90500b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.185+0000 7f90576cf700 1 -- 172.21.15.3:0/637913565 shutdown_connections 2024-05-08T21:23:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.185+0000 7f90576cf700 1 -- 172.21.15.3:0/637913565 wait complete. 2024-05-08T21:23:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.186+0000 7f90576cf700 1 Processor -- start 2024-05-08T21:23:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.187+0000 7f90576cf700 1 -- start start 2024-05-08T21:23:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.187+0000 7f90576cf700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f903c005210 0x7f905009edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f90576cf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90500cb3f0 0x7f905009cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f90576cf700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f90500b48f0 0x7f90500b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f90576cf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90500b52b0 con 0x7f90500cb3f0 2024-05-08T21:23:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f90576cf700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90500cb870 con 0x7f903c005210 2024-05-08T21:23:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f90576cf700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f905009d790 con 0x7f90500b48f0 2024-05-08T21:23:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f9054c6a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90500cb3f0 0x7f905009cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f905546b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f903c005210 0x7f905009edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f9054c6a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90500cb3f0 0x7f905009cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34498/0 (socket says 172.21.15.3:34498) 2024-05-08T21:23:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f9054c6a700 1 -- 172.21.15.3:0/2129311824 learned_addr learned my addr 172.21.15.3:0/2129311824 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f9055c6c700 1 --2- 172.21.15.3:0/2129311824 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f90500b48f0 0x7f90500b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f9054c6a700 1 -- 172.21.15.3:0/2129311824 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f90500b48f0 msgr2=0x7f90500b4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f9054c6a700 1 --2- 172.21.15.3:0/2129311824 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f90500b48f0 0x7f90500b4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f9054c6a700 1 -- 172.21.15.3:0/2129311824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f903c005210 msgr2=0x7f905009edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f9054c6a700 1 --2- 172.21.15.3:0/2129311824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f903c005210 0x7f905009edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f9054c6a700 1 -- 172.21.15.3:0/2129311824 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905009da70 con 0x7f90500cb3f0 2024-05-08T21:23:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f9054c6a700 1 --2- 172.21.15.3:0/2129311824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90500cb3f0 0x7f905009cf60 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f90400119a0 tx=0x7f9040011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f9045ffb700 1 -- 172.21.15.3:0/2129311824 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904000f780 con 0x7f90500cb3f0 2024-05-08T21:23:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f9045ffb700 1 -- 172.21.15.3:0/2129311824 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904000f8e0 con 0x7f90500cb3f0 2024-05-08T21:23:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.188+0000 7f90576cf700 1 -- 172.21.15.3:0/2129311824 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f905009dd50 con 0x7f90500cb3f0 2024-05-08T21:23:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.189+0000 7f9045ffb700 1 -- 172.21.15.3:0/2129311824 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904000fc70 con 0x7f90500cb3f0 2024-05-08T21:23:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.189+0000 7f90576cf700 1 -- 172.21.15.3:0/2129311824 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f905009e310 con 0x7f90500cb3f0 2024-05-08T21:23:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.190+0000 7f90576cf700 1 -- 172.21.15.3:0/2129311824 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90500b44d0 con 0x7f90500cb3f0 2024-05-08T21:23:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.190+0000 7f9045ffb700 1 -- 172.21.15.3:0/2129311824 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9040014440 con 0x7f90500cb3f0 2024-05-08T21:23:43.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.191+0000 7f9045ffb700 1 --2- 172.21.15.3:0/2129311824 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f903406acc0 0x7f903406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.191+0000 7f9045ffb700 1 -- 172.21.15.3:0/2129311824 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f9040089650 con 0x7f90500cb3f0 2024-05-08T21:23:43.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.191+0000 7f905546b700 1 --2- 172.21.15.3:0/2129311824 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f903406acc0 0x7f903406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.192+0000 7f905546b700 1 --2- 172.21.15.3:0/2129311824 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f903406acc0 0x7f903406d180 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f904c001050 tx=0x7f904c001b00 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:43.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.195+0000 7f9045ffb700 1 -- 172.21.15.3:0/2129311824 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9040054a30 con 0x7f90500cb3f0 2024-05-08T21:23:43.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.420+0000 7f90576cf700 1 -- 172.21.15.3:0/2129311824 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f90500b3a40 con 0x7f903406acc0 2024-05-08T21:23:43.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.421+0000 7f9045ffb700 1 -- 172.21.15.3:0/2129311824 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f904c017040 con 0x7f903406acc0 2024-05-08T21:23:43.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.423+0000 7f90576cf700 1 -- 172.21.15.3:0/2129311824 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f903406acc0 msgr2=0x7f903406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.423+0000 7f90576cf700 1 --2- 172.21.15.3:0/2129311824 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f903406acc0 0x7f903406d180 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f904c001050 tx=0x7f904c001b00 comp rx=0 tx=0).stop 2024-05-08T21:23:43.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.423+0000 7f90576cf700 1 -- 172.21.15.3:0/2129311824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90500cb3f0 msgr2=0x7f905009cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.423+0000 7f90576cf700 1 --2- 172.21.15.3:0/2129311824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90500cb3f0 0x7f905009cf60 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f90400119a0 tx=0x7f9040011cb0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.423+0000 7f90576cf700 1 -- 172.21.15.3:0/2129311824 shutdown_connections 2024-05-08T21:23:43.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.423+0000 7f90576cf700 1 --2- 172.21.15.3:0/2129311824 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f90500b48f0 0x7f90500b4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.423+0000 7f90576cf700 1 --2- 172.21.15.3:0/2129311824 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f903406acc0 0x7f903406d180 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.423+0000 7f90576cf700 1 --2- 172.21.15.3:0/2129311824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f903c005210 0x7f905009edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.423+0000 7f90576cf700 1 --2- 172.21.15.3:0/2129311824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90500cb3f0 0x7f905009cf60 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.423+0000 7f90576cf700 1 -- 172.21.15.3:0/2129311824 >> 172.21.15.3:0/2129311824 conn(0x7f90500b2f20 msgr2=0x7f90500b6010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:43.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.423+0000 7f90576cf700 1 -- 172.21.15.3:0/2129311824 shutdown_connections 2024-05-08T21:23:43.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.423+0000 7f90576cf700 1 -- 172.21.15.3:0/2129311824 wait complete. 2024-05-08T21:23:43.426 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 0 to deep-scrub 2024-05-08T21:23:43.438 INFO:tasks.ceph:Scrubbing osd.1 2024-05-08T21:23:43.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-08T21:23:43.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.577+0000 7f7e070cb700 1 Processor -- start 2024-05-08T21:23:43.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.577+0000 7f7e070cb700 1 -- start start 2024-05-08T21:23:43.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.579+0000 7f7e070cb700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7e0015b710 0x7f7e00099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.579+0000 7f7e070cb700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e000b26e0 0x7f7e00099f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.579+0000 7f7e070cb700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e000d04b0 con 0x7f7e000cff30 2024-05-08T21:23:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.579+0000 7f7e070cb700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e000b0910 con 0x7f7e0015b710 2024-05-08T21:23:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.579+0000 7f7e070cb700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e000b0bf0 con 0x7f7e000b26e0 2024-05-08T21:23:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.579+0000 7f7dfffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7e000cff30 0x7f7e000cc130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34150/0 (socket says 172.21.15.3:34150) 2024-05-08T21:23:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.579+0000 7f7dfffff700 1 -- 172.21.15.3:0/3208896320 learned_addr learned my addr 172.21.15.3:0/3208896320 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.579+0000 7f7e04e67700 1 --2- 172.21.15.3:0/3208896320 >> v2:172.21.15.186:3300/0 conn(0x7f7e0015b710 0x7f7e00099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.579+0000 7f7e05668700 1 --2- 172.21.15.3:0/3208896320 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e000b26e0 0x7f7e00099f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.579+0000 7f7dff7fe700 1 -- 172.21.15.3:0/3208896320 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854272473 0 0) 0x7f7e000d04b0 con 0x7f7e000cff30 2024-05-08T21:23:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.579+0000 7f7dff7fe700 1 -- 172.21.15.3:0/3208896320 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7de80035b0 con 0x7f7e000cff30 2024-05-08T21:23:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.579+0000 7f7e05668700 1 -- 172.21.15.3:0/3208896320 >> v2:172.21.15.186:3300/0 conn(0x7f7e0015b710 msgr2=0x7f7e00099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.579+0000 7f7e05668700 1 --2- 172.21.15.3:0/3208896320 >> v2:172.21.15.186:3300/0 conn(0x7f7e0015b710 0x7f7e00099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.579+0000 7f7e05668700 1 -- 172.21.15.3:0/3208896320 >> v1:172.21.15.3:6789/0 conn(0x7f7e000cff30 legacy=0x7f7e000cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.579+0000 7f7e05668700 1 -- 172.21.15.3:0/3208896320 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e000b0ed0 con 0x7f7e000b26e0 2024-05-08T21:23:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.579+0000 7f7e04e67700 1 --2- 172.21.15.3:0/3208896320 >> v2:172.21.15.186:3300/0 conn(0x7f7e0015b710 0x7f7e00099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.580+0000 7f7e05668700 1 --2- 172.21.15.3:0/3208896320 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e000b26e0 0x7f7e00099f90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7df8009dc0 tx=0x7f7df800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e07f5f14927cb1d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:43.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.580+0000 7f7dff7fe700 1 -- 172.21.15.3:0/3208896320 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df800e070 con 0x7f7e000b26e0 2024-05-08T21:23:43.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.580+0000 7f7dff7fe700 1 -- 172.21.15.3:0/3208896320 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df800bae0 con 0x7f7e000b26e0 2024-05-08T21:23:43.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.580+0000 7f7dff7fe700 1 -- 172.21.15.3:0/3208896320 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df800be90 con 0x7f7e000b26e0 2024-05-08T21:23:43.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.580+0000 7f7e070cb700 1 -- 172.21.15.3:0/3208896320 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e000b26e0 msgr2=0x7f7e00099f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.580+0000 7f7e070cb700 1 --2- 172.21.15.3:0/3208896320 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e000b26e0 0x7f7e00099f90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7df8009dc0 tx=0x7f7df800b330 comp rx=0 tx=0).stop 2024-05-08T21:23:43.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.580+0000 7f7e070cb700 1 -- 172.21.15.3:0/3208896320 shutdown_connections 2024-05-08T21:23:43.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.581+0000 7f7e070cb700 1 --2- 172.21.15.3:0/3208896320 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e000b26e0 0x7f7e00099f90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.581+0000 7f7e070cb700 1 --2- 172.21.15.3:0/3208896320 >> v2:172.21.15.186:3300/0 conn(0x7f7e0015b710 0x7f7e00099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.581+0000 7f7e070cb700 1 -- 172.21.15.3:0/3208896320 >> 172.21.15.3:0/3208896320 conn(0x7f7e000b2f60 msgr2=0x7f7e000b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:43.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.581+0000 7f7e070cb700 1 -- 172.21.15.3:0/3208896320 shutdown_connections 2024-05-08T21:23:43.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.581+0000 7f7e070cb700 1 -- 172.21.15.3:0/3208896320 wait complete. 2024-05-08T21:23:43.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.582+0000 7f7e070cb700 1 Processor -- start 2024-05-08T21:23:43.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.582+0000 7f7e070cb700 1 -- start start 2024-05-08T21:23:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7e070cb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7e000b26e0 0x7f7e00159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7e070cb700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e000cff30 0x7f7e001e4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7e070cb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e0015b710 0x7f7e000b5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7e070cb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e001e54b0 con 0x7f7e0015b710 2024-05-08T21:23:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7e070cb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e001e56c0 con 0x7f7e000b26e0 2024-05-08T21:23:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7e070cb700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e001e59a0 con 0x7f7e000cff30 2024-05-08T21:23:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7e05668700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e0015b710 0x7f7e000b5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7e05668700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e0015b710 0x7f7e000b5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34502/0 (socket says 172.21.15.3:34502) 2024-05-08T21:23:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7e05668700 1 -- 172.21.15.3:0/971517097 learned_addr learned my addr 172.21.15.3:0/971517097 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7e04e67700 1 --2- 172.21.15.3:0/971517097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7e000b26e0 0x7f7e00159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7dfffff700 1 --2- 172.21.15.3:0/971517097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e000cff30 0x7f7e001e4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7e05668700 1 -- 172.21.15.3:0/971517097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e000cff30 msgr2=0x7f7e001e4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7e05668700 1 --2- 172.21.15.3:0/971517097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e000cff30 0x7f7e001e4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7e05668700 1 -- 172.21.15.3:0/971517097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7e000b26e0 msgr2=0x7f7e00159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7e05668700 1 --2- 172.21.15.3:0/971517097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7e000b26e0 0x7f7e00159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7e05668700 1 -- 172.21.15.3:0/971517097 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df00094a0 con 0x7f7e0015b710 2024-05-08T21:23:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.583+0000 7f7e04e67700 1 --2- 172.21.15.3:0/971517097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7e000b26e0 0x7f7e00159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.584+0000 7f7e05668700 1 --2- 172.21.15.3:0/971517097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e0015b710 0x7f7e000b5ce0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7df8014040 tx=0x7f7df8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.584+0000 7f7dfd7fa700 1 -- 172.21.15.3:0/971517097 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df80044d0 con 0x7f7e0015b710 2024-05-08T21:23:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.584+0000 7f7dfd7fa700 1 -- 172.21.15.3:0/971517097 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df8020070 con 0x7f7e0015b710 2024-05-08T21:23:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.584+0000 7f7e070cb700 1 -- 172.21.15.3:0/971517097 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df8009a70 con 0x7f7e0015b710 2024-05-08T21:23:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.584+0000 7f7dfd7fa700 1 -- 172.21.15.3:0/971517097 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df80048b0 con 0x7f7e0015b710 2024-05-08T21:23:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.584+0000 7f7e070cb700 1 -- 172.21.15.3:0/971517097 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e001e5f60 con 0x7f7e0015b710 2024-05-08T21:23:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.585+0000 7f7e070cb700 1 -- 172.21.15.3:0/971517097 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7e000cd390 con 0x7f7e0015b710 2024-05-08T21:23:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.585+0000 7f7dfd7fa700 1 -- 172.21.15.3:0/971517097 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7df8012430 con 0x7f7e0015b710 2024-05-08T21:23:43.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.586+0000 7f7dfd7fa700 1 --2- 172.21.15.3:0/971517097 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7dec06ac90 0x7f7dec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.586+0000 7f7dfd7fa700 1 -- 172.21.15.3:0/971517097 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f7df8022070 con 0x7f7e0015b710 2024-05-08T21:23:43.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.586+0000 7f7dfd7fa700 1 --2- 172.21.15.3:0/971517097 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x7f7dec070b90 0x7f7dec072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.586+0000 7f7dfd7fa700 1 -- 172.21.15.3:0/971517097 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7dec073660 con 0x7f7dec070b90 2024-05-08T21:23:43.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.586+0000 7f7dfd7fa700 1 -- 172.21.15.3:0/971517097 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7df808b9b0 con 0x7f7e0015b710 2024-05-08T21:23:43.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.586+0000 7f7dfffff700 1 --2- 172.21.15.3:0/971517097 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x7f7dec070b90 0x7f7dec072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.586+0000 7f7e04e67700 1 --2- 172.21.15.3:0/971517097 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7dec06ac90 0x7f7dec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.587+0000 7f7dfffff700 1 --2- 172.21.15.3:0/971517097 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x7f7dec070b90 0x7f7dec072fb0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:43.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.587+0000 7f7e04e67700 1 --2- 172.21.15.3:0/971517097 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7dec06ac90 0x7f7dec06d150 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7df00114b0 tx=0x7f7df0009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:43.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.587+0000 7f7dfd7fa700 1 -- 172.21.15.3:0/971517097 <== osd.1 v2:172.21.15.3:6816/47119 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f7dec073660 con 0x7f7dec070b90 2024-05-08T21:23:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.603+0000 7f7e070cb700 1 -- 172.21.15.3:0/971517097 --> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7e00074bc0 con 0x7f7dec070b90 2024-05-08T21:23:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.603+0000 7f7dfd7fa700 1 -- 172.21.15.3:0/971517097 <== osd.1 v2:172.21.15.3:6816/47119 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f7e00074bc0 con 0x7f7dec070b90 2024-05-08T21:23:43.605 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:23:43.605 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T21:23:43.605 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:23:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.604+0000 7f7e070cb700 1 -- 172.21.15.3:0/971517097 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x7f7dec070b90 msgr2=0x7f7dec072fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.604+0000 7f7e070cb700 1 --2- 172.21.15.3:0/971517097 >> [v2:172.21.15.3:6816/47119,v1:172.21.15.3:6817/47119] conn(0x7f7dec070b90 0x7f7dec072fb0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.604+0000 7f7e070cb700 1 -- 172.21.15.3:0/971517097 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7dec06ac90 msgr2=0x7f7dec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.604+0000 7f7e070cb700 1 --2- 172.21.15.3:0/971517097 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f7dec06ac90 0x7f7dec06d150 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7df00114b0 tx=0x7f7df0009210 comp rx=0 tx=0).stop 2024-05-08T21:23:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.604+0000 7f7e070cb700 1 -- 172.21.15.3:0/971517097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e0015b710 msgr2=0x7f7e000b5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.604+0000 7f7e070cb700 1 --2- 172.21.15.3:0/971517097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e0015b710 0x7f7e000b5ce0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7df8014040 tx=0x7f7df8004ad0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.604+0000 7f7e05668700 1 -- 172.21.15.3:0/971517097 reap_dead start 2024-05-08T21:23:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.604+0000 7f7e070cb700 1 -- 172.21.15.3:0/971517097 shutdown_connections 2024-05-08T21:23:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.604+0000 7f7e070cb700 1 -- 172.21.15.3:0/971517097 >> 172.21.15.3:0/971517097 conn(0x7f7e000b2f60 msgr2=0x7f7e000b3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.604+0000 7f7e070cb700 1 -- 172.21.15.3:0/971517097 shutdown_connections 2024-05-08T21:23:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.604+0000 7f7e070cb700 1 -- 172.21.15.3:0/971517097 wait complete. 2024-05-08T21:23:43.619 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-05-08T21:23:43.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.754+0000 7fe9828fc700 1 Processor -- start 2024-05-08T21:23:43.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.754+0000 7fe9828fc700 1 -- start start 2024-05-08T21:23:43.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.755+0000 7fe9828fc700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe97c15a890 0x7fe97c06e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.755+0000 7fe9828fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe97c0c5900 0x7fe97c06e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.755+0000 7fe9828fc700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe97c0ce470 con 0x7fe97c0ca9a0 2024-05-08T21:23:43.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.755+0000 7fe9828fc700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe97c0c63c0 con 0x7fe97c15a890 2024-05-08T21:23:43.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.755+0000 7fe9828fc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c0b2e50 con 0x7fe97c0c5900 2024-05-08T21:23:43.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.755+0000 7fe97b7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe97c0ca9a0 0x7fe97c0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34160/0 (socket says 172.21.15.3:34160) 2024-05-08T21:23:43.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.755+0000 7fe97b7fe700 1 -- 172.21.15.3:0/3990206199 learned_addr learned my addr 172.21.15.3:0/3990206199 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:43.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.755+0000 7fe97bfff700 1 --2- 172.21.15.3:0/3990206199 >> v2:172.21.15.186:3300/0 conn(0x7fe97c15a890 0x7fe97c06e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.755+0000 7fe980e99700 1 --2- 172.21.15.3:0/3990206199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe97c0c5900 0x7fe97c06e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.755+0000 7fe97affd700 1 -- 172.21.15.3:0/3990206199 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890942012 0 0) 0x7fe97c0ce470 con 0x7fe97c0ca9a0 2024-05-08T21:23:43.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.755+0000 7fe97affd700 1 -- 172.21.15.3:0/3990206199 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9680035b0 con 0x7fe97c0ca9a0 2024-05-08T21:23:43.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.756+0000 7fe97bfff700 1 -- 172.21.15.3:0/3990206199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe97c0c5900 msgr2=0x7fe97c06e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:43.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.756+0000 7fe97bfff700 1 --2- 172.21.15.3:0/3990206199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe97c0c5900 0x7fe97c06e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.756+0000 7fe97affd700 1 -- 172.21.15.3:0/3990206199 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 646259419 0 0) 0x7fe9680035b0 con 0x7fe97c0ca9a0 2024-05-08T21:23:43.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.756+0000 7fe97bfff700 1 -- 172.21.15.3:0/3990206199 >> v1:172.21.15.3:6789/0 conn(0x7fe97c0ca9a0 legacy=0x7fe97c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.756+0000 7fe97bfff700 1 -- 172.21.15.3:0/3990206199 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97c0c5e00 con 0x7fe97c15a890 2024-05-08T21:23:43.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.756+0000 7fe980e99700 1 --2- 172.21.15.3:0/3990206199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe97c0c5900 0x7fe97c06e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-08T21:23:43.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.756+0000 7fe97bfff700 1 --2- 172.21.15.3:0/3990206199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97c15a890 0x7fe97c06e430 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe96c009840 tx=0x7fe96c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc35c5ceebc7efad server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:43.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.756+0000 7fe97affd700 1 -- 172.21.15.3:0/3990206199 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe964001670 con 0x7fe97c15a890 2024-05-08T21:23:43.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.756+0000 7fe97affd700 1 -- 172.21.15.3:0/3990206199 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe96c004030 con 0x7fe97c15a890 2024-05-08T21:23:43.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.756+0000 7fe97affd700 1 -- 172.21.15.3:0/3990206199 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96c0043c0 con 0x7fe97c15a890 2024-05-08T21:23:43.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.757+0000 7fe9828fc700 1 -- 172.21.15.3:0/3990206199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97c15a890 msgr2=0x7fe97c06e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.757+0000 7fe9828fc700 1 --2- 172.21.15.3:0/3990206199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97c15a890 0x7fe97c06e430 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe96c009840 tx=0x7fe96c009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:43.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.757+0000 7fe9828fc700 1 -- 172.21.15.3:0/3990206199 shutdown_connections 2024-05-08T21:23:43.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.757+0000 7fe9828fc700 1 --2- 172.21.15.3:0/3990206199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe97c0c5900 0x7fe97c06e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.757+0000 7fe9828fc700 1 --2- 172.21.15.3:0/3990206199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97c15a890 0x7fe97c06e430 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.757+0000 7fe9828fc700 1 -- 172.21.15.3:0/3990206199 >> 172.21.15.3:0/3990206199 conn(0x7fe97c0b1f80 msgr2=0x7fe97c0b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:43.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.757+0000 7fe9828fc700 1 -- 172.21.15.3:0/3990206199 shutdown_connections 2024-05-08T21:23:43.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.757+0000 7fe9828fc700 1 -- 172.21.15.3:0/3990206199 wait complete. 2024-05-08T21:23:43.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.758+0000 7fe9828fc700 1 Processor -- start 2024-05-08T21:23:43.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.758+0000 7fe9828fc700 1 -- start start 2024-05-08T21:23:43.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.759+0000 7fe9828fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97c0c5900 0x7fe97c1f1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.759+0000 7fe9828fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe97c0ca9a0 0x7fe97c0c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.759+0000 7fe9828fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe97c15a890 0x7fe97c157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.759+0000 7fe9828fc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c1f19a0 con 0x7fe97c0ca9a0 2024-05-08T21:23:43.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.759+0000 7fe9828fc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c1e0f80 con 0x7fe97c0c5900 2024-05-08T21:23:43.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.759+0000 7fe9828fc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c158560 con 0x7fe97c15a890 2024-05-08T21:23:43.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.759+0000 7fe97b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe97c0ca9a0 0x7fe97c0c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.759+0000 7fe97b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe97c0ca9a0 0x7fe97c0c9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34514/0 (socket says 172.21.15.3:34514) 2024-05-08T21:23:43.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.759+0000 7fe97b7fe700 1 -- 172.21.15.3:0/3334505251 learned_addr learned my addr 172.21.15.3:0/3334505251 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:43.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.759+0000 7fe980e99700 1 --2- 172.21.15.3:0/3334505251 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe97c15a890 0x7fe97c157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.759+0000 7fe97bfff700 1 --2- 172.21.15.3:0/3334505251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97c0c5900 0x7fe97c1f1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.760+0000 7fe97b7fe700 1 -- 172.21.15.3:0/3334505251 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe97c15a890 msgr2=0x7fe97c157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.760+0000 7fe97b7fe700 1 --2- 172.21.15.3:0/3334505251 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe97c15a890 0x7fe97c157420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.760+0000 7fe97b7fe700 1 -- 172.21.15.3:0/3334505251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97c0c5900 msgr2=0x7fe97c1f1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.760+0000 7fe97b7fe700 1 --2- 172.21.15.3:0/3334505251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97c0c5900 0x7fe97c1f1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.760+0000 7fe97b7fe700 1 -- 172.21.15.3:0/3334505251 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe974009a70 con 0x7fe97c0ca9a0 2024-05-08T21:23:43.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.760+0000 7fe97b7fe700 1 --2- 172.21.15.3:0/3334505251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe97c0ca9a0 0x7fe97c0c9f70 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe96400c780 tx=0x7fe96400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:43.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.760+0000 7fe978ff9700 1 -- 172.21.15.3:0/3334505251 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe964004c80 con 0x7fe97c0ca9a0 2024-05-08T21:23:43.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.760+0000 7fe978ff9700 1 -- 172.21.15.3:0/3334505251 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe964004de0 con 0x7fe97c0ca9a0 2024-05-08T21:23:43.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.760+0000 7fe9828fc700 1 -- 172.21.15.3:0/3334505251 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96c0094a0 con 0x7fe97c0ca9a0 2024-05-08T21:23:43.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.760+0000 7fe978ff9700 1 -- 172.21.15.3:0/3334505251 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9640072d0 con 0x7fe97c0ca9a0 2024-05-08T21:23:43.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.760+0000 7fe9828fc700 1 -- 172.21.15.3:0/3334505251 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe97c1e1ac0 con 0x7fe97c0ca9a0 2024-05-08T21:23:43.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.761+0000 7fe9828fc700 1 -- 172.21.15.3:0/3334505251 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe97c0cb700 con 0x7fe97c0ca9a0 2024-05-08T21:23:43.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.762+0000 7fe978ff9700 1 -- 172.21.15.3:0/3334505251 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe964007430 con 0x7fe97c0ca9a0 2024-05-08T21:23:43.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.762+0000 7fe978ff9700 1 --2- 172.21.15.3:0/3334505251 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe95c06ac90 0x7fe95c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:43.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.762+0000 7fe978ff9700 1 -- 172.21.15.3:0/3334505251 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe964088640 con 0x7fe97c0ca9a0 2024-05-08T21:23:43.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.762+0000 7fe97bfff700 1 --2- 172.21.15.3:0/3334505251 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe95c06ac90 0x7fe95c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:43.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.763+0000 7fe97bfff700 1 --2- 172.21.15.3:0/3334505251 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe95c06ac90 0x7fe95c06d150 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe96c0090b0 tx=0x7fe96c013040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:43.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.767+0000 7fe978ff9700 1 -- 172.21.15.3:0/3334505251 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe964053b50 con 0x7fe97c0ca9a0 2024-05-08T21:23:43.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.989+0000 7fe9828fc700 1 -- 172.21.15.3:0/3334505251 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe97c0ce9f0 con 0x7fe95c06ac90 2024-05-08T21:23:43.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.990+0000 7fe978ff9700 1 -- 172.21.15.3:0/3334505251 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe96c004520 con 0x7fe95c06ac90 2024-05-08T21:23:43.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.992+0000 7fe9828fc700 1 -- 172.21.15.3:0/3334505251 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe95c06ac90 msgr2=0x7fe95c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.992+0000 7fe9828fc700 1 --2- 172.21.15.3:0/3334505251 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe95c06ac90 0x7fe95c06d150 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe96c0090b0 tx=0x7fe96c013040 comp rx=0 tx=0).stop 2024-05-08T21:23:43.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.992+0000 7fe9828fc700 1 -- 172.21.15.3:0/3334505251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe97c0ca9a0 msgr2=0x7fe97c0c9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:43.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.992+0000 7fe9828fc700 1 --2- 172.21.15.3:0/3334505251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe97c0ca9a0 0x7fe97c0c9f70 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe96400c780 tx=0x7fe96400cb40 comp rx=0 tx=0).stop 2024-05-08T21:23:43.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.992+0000 7fe9828fc700 1 -- 172.21.15.3:0/3334505251 shutdown_connections 2024-05-08T21:23:43.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.992+0000 7fe9828fc700 1 --2- 172.21.15.3:0/3334505251 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe97c15a890 0x7fe97c157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.992+0000 7fe9828fc700 1 --2- 172.21.15.3:0/3334505251 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe95c06ac90 0x7fe95c06d150 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.992+0000 7fe9828fc700 1 --2- 172.21.15.3:0/3334505251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97c0c5900 0x7fe97c1f1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.992+0000 7fe9828fc700 1 --2- 172.21.15.3:0/3334505251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe97c0ca9a0 0x7fe97c0c9f70 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:43.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.992+0000 7fe9828fc700 1 -- 172.21.15.3:0/3334505251 >> 172.21.15.3:0/3334505251 conn(0x7fe97c0b1f80 msgr2=0x7fe97c0caff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:43.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.992+0000 7fe9828fc700 1 -- 172.21.15.3:0/3334505251 shutdown_connections 2024-05-08T21:23:43.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:43.993+0000 7fe9828fc700 1 -- 172.21.15.3:0/3334505251 wait complete. 2024-05-08T21:23:43.995 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 1 to deep-scrub 2024-05-08T21:23:44.008 INFO:tasks.ceph:Scrubbing osd.2 2024-05-08T21:23:44.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-08T21:23:44.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.147+0000 7fc0ec761700 1 Processor -- start 2024-05-08T21:23:44.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.147+0000 7fc0ec761700 1 -- start start 2024-05-08T21:23:44.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.148+0000 7fc0ec761700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc0e40b0d80 0x7fc0e40b3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.148+0000 7fc0ec761700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0e406b070 0x7fc0e41454d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.148+0000 7fc0ec761700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e4145040 con 0x7fc0e40b4d50 2024-05-08T21:23:44.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.148+0000 7fc0ec761700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e40b38c0 con 0x7fc0e40b0d80 2024-05-08T21:23:44.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.148+0000 7fc0ec761700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e40b16e0 con 0x7fc0e406b070 2024-05-08T21:23:44.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.148+0000 7fc0ea4fd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc0e40b0d80 0x7fc0e40b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.148+0000 7fc0ea4fd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc0e40b0d80 0x7fc0e40b3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34522/0 (socket says 172.21.15.3:34522) 2024-05-08T21:23:44.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.148+0000 7fc0ea4fd700 1 -- 172.21.15.3:0/3728532772 learned_addr learned my addr 172.21.15.3:0/3728532772 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:44.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.148+0000 7fc0eacfe700 1 --2- 172.21.15.3:0/3728532772 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0e406b070 0x7fc0e41454d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.148+0000 7fc0ea4fd700 1 -- 172.21.15.3:0/3728532772 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0e406b070 msgr2=0x7fc0e41454d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:44.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.148+0000 7fc0ea4fd700 1 --2- 172.21.15.3:0/3728532772 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0e406b070 0x7fc0e41454d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.148+0000 7fc0ea4fd700 1 -- 172.21.15.3:0/3728532772 >> v1:172.21.15.186:6789/0 conn(0x7fc0e40b4d50 legacy=0x7fc0e406a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.148+0000 7fc0ea4fd700 1 -- 172.21.15.3:0/3728532772 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e40b19c0 con 0x7fc0e40b0d80 2024-05-08T21:23:44.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.149+0000 7fc0ea4fd700 1 --2- 172.21.15.3:0/3728532772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0e40b0d80 0x7fc0e40b3380 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00098f0 tx=0x7fc0e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d007bf179909a92b server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:44.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.149+0000 7fc0e94fb700 1 -- 172.21.15.3:0/3728532772 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e000b8a0 con 0x7fc0e40b0d80 2024-05-08T21:23:44.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.149+0000 7fc0e94fb700 1 -- 172.21.15.3:0/3728532772 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e000ba00 con 0x7fc0e40b0d80 2024-05-08T21:23:44.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.149+0000 7fc0e94fb700 1 -- 172.21.15.3:0/3728532772 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e000bd90 con 0x7fc0e40b0d80 2024-05-08T21:23:44.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.149+0000 7fc0ec761700 1 -- 172.21.15.3:0/3728532772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0e40b0d80 msgr2=0x7fc0e40b3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.149+0000 7fc0ec761700 1 --2- 172.21.15.3:0/3728532772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0e40b0d80 0x7fc0e40b3380 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00098f0 tx=0x7fc0e0009c00 comp rx=0 tx=0).stop 2024-05-08T21:23:44.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.149+0000 7fc0ec761700 1 -- 172.21.15.3:0/3728532772 shutdown_connections 2024-05-08T21:23:44.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.149+0000 7fc0ec761700 1 --2- 172.21.15.3:0/3728532772 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0e406b070 0x7fc0e41454d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.149+0000 7fc0ec761700 1 --2- 172.21.15.3:0/3728532772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0e40b0d80 0x7fc0e40b3380 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.149+0000 7fc0ec761700 1 -- 172.21.15.3:0/3728532772 >> 172.21.15.3:0/3728532772 conn(0x7fc0e4099a50 msgr2=0x7fc0e40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:44.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.149+0000 7fc0ec761700 1 -- 172.21.15.3:0/3728532772 shutdown_connections 2024-05-08T21:23:44.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.149+0000 7fc0ec761700 1 -- 172.21.15.3:0/3728532772 wait complete. 2024-05-08T21:23:44.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.151+0000 7fc0ec761700 1 Processor -- start 2024-05-08T21:23:44.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.151+0000 7fc0ec761700 1 -- start start 2024-05-08T21:23:44.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0ec761700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0e406b070 0x7fc0e40aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0ec761700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0e40b0d80 0x7fc0e413e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0ec761700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0e40b4d50 0x7fc0e415c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0ec761700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e413f370 con 0x7fc0e406b070 2024-05-08T21:23:44.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0ec761700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e413f650 con 0x7fc0e40b0d80 2024-05-08T21:23:44.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0ec761700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e413f930 con 0x7fc0e40b4d50 2024-05-08T21:23:44.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0ea4fd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0e406b070 0x7fc0e40aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0ea4fd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0e406b070 0x7fc0e40aedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34530/0 (socket says 172.21.15.3:34530) 2024-05-08T21:23:44.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0ea4fd700 1 -- 172.21.15.3:0/4226609534 learned_addr learned my addr 172.21.15.3:0/4226609534 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:44.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0e9cfc700 1 --2- 172.21.15.3:0/4226609534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0e40b0d80 0x7fc0e413e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0eacfe700 1 --2- 172.21.15.3:0/4226609534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0e40b4d50 0x7fc0e415c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0ea4fd700 1 -- 172.21.15.3:0/4226609534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0e40b4d50 msgr2=0x7fc0e415c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0ea4fd700 1 --2- 172.21.15.3:0/4226609534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0e40b4d50 0x7fc0e415c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0ea4fd700 1 -- 172.21.15.3:0/4226609534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0e40b0d80 msgr2=0x7fc0e413e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0ea4fd700 1 --2- 172.21.15.3:0/4226609534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0e40b0d80 0x7fc0e413e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0ea4fd700 1 -- 172.21.15.3:0/4226609534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e0009550 con 0x7fc0e406b070 2024-05-08T21:23:44.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0ea4fd700 1 --2- 172.21.15.3:0/4226609534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0e406b070 0x7fc0e40aedb0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0000c00 tx=0x7fc0e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:44.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0daffd700 1 -- 172.21.15.3:0/4226609534 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0004480 con 0x7fc0e406b070 2024-05-08T21:23:44.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0daffd700 1 -- 172.21.15.3:0/4226609534 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e001f070 con 0x7fc0e406b070 2024-05-08T21:23:44.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0ec761700 1 -- 172.21.15.3:0/4226609534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e413fc10 con 0x7fc0e406b070 2024-05-08T21:23:44.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.152+0000 7fc0daffd700 1 -- 172.21.15.3:0/4226609534 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e00038c0 con 0x7fc0e406b070 2024-05-08T21:23:44.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.153+0000 7fc0ec761700 1 -- 172.21.15.3:0/4226609534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e415d520 con 0x7fc0e406b070 2024-05-08T21:23:44.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.153+0000 7fc0ec761700 1 -- 172.21.15.3:0/4226609534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc0e40b55a0 con 0x7fc0e406b070 2024-05-08T21:23:44.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.154+0000 7fc0daffd700 1 -- 172.21.15.3:0/4226609534 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc0e0021030 con 0x7fc0e406b070 2024-05-08T21:23:44.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.154+0000 7fc0daffd700 1 --2- 172.21.15.3:0/4226609534 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc0c806ac90 0x7fc0c806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.154+0000 7fc0daffd700 1 -- 172.21.15.3:0/4226609534 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fc0e008aef0 con 0x7fc0e406b070 2024-05-08T21:23:44.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.155+0000 7fc0daffd700 1 --2- 172.21.15.3:0/4226609534 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x7fc0c8070b90 0x7fc0c8072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.155+0000 7fc0daffd700 1 -- 172.21.15.3:0/4226609534 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0c8073660 con 0x7fc0c8070b90 2024-05-08T21:23:44.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.155+0000 7fc0daffd700 1 -- 172.21.15.3:0/4226609534 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0e008b2a0 con 0x7fc0e406b070 2024-05-08T21:23:44.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.155+0000 7fc0eacfe700 1 --2- 172.21.15.3:0/4226609534 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x7fc0c8070b90 0x7fc0c8072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.155+0000 7fc0e9cfc700 1 --2- 172.21.15.3:0/4226609534 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc0c806ac90 0x7fc0c806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.155+0000 7fc0eacfe700 1 --2- 172.21.15.3:0/4226609534 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x7fc0c8070b90 0x7fc0c8072fb0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:44.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.155+0000 7fc0e9cfc700 1 --2- 172.21.15.3:0/4226609534 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc0c806ac90 0x7fc0c806d150 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4140d70 tx=0x7fc0d4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:44.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.156+0000 7fc0daffd700 1 -- 172.21.15.3:0/4226609534 <== osd.2 v2:172.21.15.3:6808/47117 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fc0c8073660 con 0x7fc0c8070b90 2024-05-08T21:23:44.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.171+0000 7fc0ec761700 1 -- 172.21.15.3:0/4226609534 --> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc0e4074bc0 con 0x7fc0c8070b90 2024-05-08T21:23:44.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.172+0000 7fc0daffd700 1 -- 172.21.15.3:0/4226609534 <== osd.2 v2:172.21.15.3:6808/47117 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fc0e4074bc0 con 0x7fc0c8070b90 2024-05-08T21:23:44.173 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:23:44.173 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T21:23:44.173 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:23:44.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.172+0000 7fc0ec761700 1 -- 172.21.15.3:0/4226609534 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x7fc0c8070b90 msgr2=0x7fc0c8072fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.172+0000 7fc0ec761700 1 --2- 172.21.15.3:0/4226609534 >> [v2:172.21.15.3:6808/47117,v1:172.21.15.3:6809/47117] conn(0x7fc0c8070b90 0x7fc0c8072fb0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.172+0000 7fc0ec761700 1 -- 172.21.15.3:0/4226609534 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc0c806ac90 msgr2=0x7fc0c806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.172+0000 7fc0ec761700 1 --2- 172.21.15.3:0/4226609534 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc0c806ac90 0x7fc0c806d150 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4140d70 tx=0x7fc0d4009040 comp rx=0 tx=0).stop 2024-05-08T21:23:44.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.172+0000 7fc0ec761700 1 -- 172.21.15.3:0/4226609534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0e406b070 msgr2=0x7fc0e40aedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.172+0000 7fc0ec761700 1 --2- 172.21.15.3:0/4226609534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0e406b070 0x7fc0e40aedb0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0000c00 tx=0x7fc0e0004750 comp rx=0 tx=0).stop 2024-05-08T21:23:44.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.172+0000 7fc0eacfe700 1 -- 172.21.15.3:0/4226609534 reap_dead start 2024-05-08T21:23:44.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.172+0000 7fc0ec761700 1 -- 172.21.15.3:0/4226609534 shutdown_connections 2024-05-08T21:23:44.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.172+0000 7fc0ec761700 1 -- 172.21.15.3:0/4226609534 >> 172.21.15.3:0/4226609534 conn(0x7fc0e4099a50 msgr2=0x7fc0e41453d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:44.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.172+0000 7fc0ec761700 1 -- 172.21.15.3:0/4226609534 shutdown_connections 2024-05-08T21:23:44.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.172+0000 7fc0ec761700 1 -- 172.21.15.3:0/4226609534 wait complete. 2024-05-08T21:23:44.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-05-08T21:23:44.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.323+0000 7fc422da4700 1 Processor -- start 2024-05-08T21:23:44.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.323+0000 7fc422da4700 1 -- start start 2024-05-08T21:23:44.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.324+0000 7fc422da4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc41c0b28b0 0x7fc41c166350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.324+0000 7fc422da4700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c0b1410 con 0x7fc41c0cf9e0 2024-05-08T21:23:44.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.324+0000 7fc422da4700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c0b0e30 con 0x7fc41c15b900 2024-05-08T21:23:44.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.324+0000 7fc422da4700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc41c09a870 con 0x7fc41c0b28b0 2024-05-08T21:23:44.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.324+0000 7fc421341700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc41c0b28b0 0x7fc41c166350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.324+0000 7fc41bfff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc41c0cf9e0 0x7fc41c09a160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34170/0 (socket says 172.21.15.3:34170) 2024-05-08T21:23:44.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.324+0000 7fc41bfff700 1 -- 172.21.15.3:0/2332167530 learned_addr learned my addr 172.21.15.3:0/2332167530 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:44.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.325+0000 7fc421341700 1 -- 172.21.15.3:0/2332167530 >> v1:172.21.15.186:6789/0 conn(0x7fc41c15b900 legacy=0x7fc41c099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.325+0000 7fc421341700 1 -- 172.21.15.3:0/2332167530 >> v1:172.21.15.3:6789/0 conn(0x7fc41c0cf9e0 legacy=0x7fc41c09a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.325+0000 7fc41b7fe700 1 -- 172.21.15.3:0/2332167530 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390499194 0 0) 0x7fc41c0b1410 con 0x7fc41c0cf9e0 2024-05-08T21:23:44.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.325+0000 7fc421341700 1 -- 172.21.15.3:0/2332167530 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c0cffd0 con 0x7fc41c0b28b0 2024-05-08T21:23:44.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.325+0000 7fc421341700 1 --2- 172.21.15.3:0/2332167530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc41c0b28b0 0x7fc41c166350 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc414009d90 tx=0x7fc41400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c14a3baadc8d9f server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:44.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.325+0000 7fc41b7fe700 1 -- 172.21.15.3:0/2332167530 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41400e070 con 0x7fc41c0b28b0 2024-05-08T21:23:44.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.325+0000 7fc41b7fe700 1 -- 172.21.15.3:0/2332167530 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41400bbd0 con 0x7fc41c0b28b0 2024-05-08T21:23:44.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.325+0000 7fc41b7fe700 1 -- 172.21.15.3:0/2332167530 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc414012430 con 0x7fc41c0b28b0 2024-05-08T21:23:44.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.325+0000 7fc422da4700 1 -- 172.21.15.3:0/2332167530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc41c0b28b0 msgr2=0x7fc41c166350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.325+0000 7fc422da4700 1 --2- 172.21.15.3:0/2332167530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc41c0b28b0 0x7fc41c166350 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc414009d90 tx=0x7fc41400b330 comp rx=0 tx=0).stop 2024-05-08T21:23:44.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.326+0000 7fc422da4700 1 -- 172.21.15.3:0/2332167530 shutdown_connections 2024-05-08T21:23:44.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.326+0000 7fc422da4700 1 --2- 172.21.15.3:0/2332167530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc41c0b28b0 0x7fc41c166350 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.326+0000 7fc422da4700 1 -- 172.21.15.3:0/2332167530 >> 172.21.15.3:0/2332167530 conn(0x7fc41c0b2f20 msgr2=0x7fc41c0b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:44.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.326+0000 7fc422da4700 1 -- 172.21.15.3:0/2332167530 shutdown_connections 2024-05-08T21:23:44.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.326+0000 7fc422da4700 1 -- 172.21.15.3:0/2332167530 wait complete. 2024-05-08T21:23:44.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.327+0000 7fc422da4700 1 Processor -- start 2024-05-08T21:23:44.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.327+0000 7fc422da4700 1 -- start start 2024-05-08T21:23:44.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.328+0000 7fc422da4700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc41c0b28b0 0x7fc41c15e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.328+0000 7fc422da4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc41c0cf9e0 0x7fc41c14f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.328+0000 7fc422da4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc41c15b900 0x7fc41c14f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.328+0000 7fc422da4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c06d350 con 0x7fc41c0cf9e0 2024-05-08T21:23:44.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.328+0000 7fc422da4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c06c430 con 0x7fc41c15b900 2024-05-08T21:23:44.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.328+0000 7fc422da4700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c06c6d0 con 0x7fc41c0b28b0 2024-05-08T21:23:44.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.328+0000 7fc41bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc41c0cf9e0 0x7fc41c14f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.328+0000 7fc41bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc41c0cf9e0 0x7fc41c14f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34542/0 (socket says 172.21.15.3:34542) 2024-05-08T21:23:44.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.328+0000 7fc41bfff700 1 -- 172.21.15.3:0/2973863034 learned_addr learned my addr 172.21.15.3:0/2973863034 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:44.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.328+0000 7fc420b40700 1 --2- 172.21.15.3:0/2973863034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc41c0b28b0 0x7fc41c15e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.328+0000 7fc421341700 1 --2- 172.21.15.3:0/2973863034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc41c15b900 0x7fc41c14f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.329+0000 7fc41bfff700 1 -- 172.21.15.3:0/2973863034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc41c0b28b0 msgr2=0x7fc41c15e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.329+0000 7fc41bfff700 1 --2- 172.21.15.3:0/2973863034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc41c0b28b0 0x7fc41c15e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.329+0000 7fc41bfff700 1 -- 172.21.15.3:0/2973863034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc41c15b900 msgr2=0x7fc41c14f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.329+0000 7fc41bfff700 1 --2- 172.21.15.3:0/2973863034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc41c15b900 0x7fc41c14f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.329+0000 7fc41bfff700 1 -- 172.21.15.3:0/2973863034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc414009a70 con 0x7fc41c0cf9e0 2024-05-08T21:23:44.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.329+0000 7fc41bfff700 1 --2- 172.21.15.3:0/2973863034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc41c0cf9e0 0x7fc41c14f0d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc40400c520 tx=0x7fc40400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:44.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.329+0000 7fc4197fa700 1 -- 172.21.15.3:0/2973863034 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404004c80 con 0x7fc41c0cf9e0 2024-05-08T21:23:44.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.329+0000 7fc4197fa700 1 -- 172.21.15.3:0/2973863034 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc404004e20 con 0x7fc41c0cf9e0 2024-05-08T21:23:44.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.329+0000 7fc422da4700 1 -- 172.21.15.3:0/2973863034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41c06c9b0 con 0x7fc41c0cf9e0 2024-05-08T21:23:44.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.329+0000 7fc4197fa700 1 -- 172.21.15.3:0/2973863034 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404007310 con 0x7fc41c0cf9e0 2024-05-08T21:23:44.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.329+0000 7fc422da4700 1 -- 172.21.15.3:0/2973863034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41c06cf70 con 0x7fc41c0cf9e0 2024-05-08T21:23:44.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.330+0000 7fc422da4700 1 -- 172.21.15.3:0/2973863034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41c072f10 con 0x7fc41c0cf9e0 2024-05-08T21:23:44.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.330+0000 7fc4197fa700 1 -- 172.21.15.3:0/2973863034 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc404007470 con 0x7fc41c0cf9e0 2024-05-08T21:23:44.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.331+0000 7fc4197fa700 1 --2- 172.21.15.3:0/2973863034 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc3fc06ac90 0x7fc3fc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.331+0000 7fc4197fa700 1 -- 172.21.15.3:0/2973863034 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fc404088e60 con 0x7fc41c0cf9e0 2024-05-08T21:23:44.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.331+0000 7fc420b40700 1 --2- 172.21.15.3:0/2973863034 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc3fc06ac90 0x7fc3fc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.332+0000 7fc420b40700 1 --2- 172.21.15.3:0/2973863034 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc3fc06ac90 0x7fc3fc06d150 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc40c000f60 tx=0x7fc40c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:44.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.336+0000 7fc4197fa700 1 -- 172.21.15.3:0/2973863034 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc404054250 con 0x7fc41c0cf9e0 2024-05-08T21:23:44.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.554+0000 7fc422da4700 1 -- 172.21.15.3:0/2973863034 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc41c09aa60 con 0x7fc3fc06ac90 2024-05-08T21:23:44.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.555+0000 7fc4197fa700 1 -- 172.21.15.3:0/2973863034 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc41c09aa60 con 0x7fc3fc06ac90 2024-05-08T21:23:44.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.557+0000 7fc422da4700 1 -- 172.21.15.3:0/2973863034 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc3fc06ac90 msgr2=0x7fc3fc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.557+0000 7fc422da4700 1 --2- 172.21.15.3:0/2973863034 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc3fc06ac90 0x7fc3fc06d150 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc40c000f60 tx=0x7fc40c009040 comp rx=0 tx=0).stop 2024-05-08T21:23:44.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.557+0000 7fc422da4700 1 -- 172.21.15.3:0/2973863034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc41c0cf9e0 msgr2=0x7fc41c14f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.557+0000 7fc422da4700 1 --2- 172.21.15.3:0/2973863034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc41c0cf9e0 0x7fc41c14f0d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc40400c520 tx=0x7fc40400c8e0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.557+0000 7fc422da4700 1 -- 172.21.15.3:0/2973863034 shutdown_connections 2024-05-08T21:23:44.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.557+0000 7fc422da4700 1 --2- 172.21.15.3:0/2973863034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc41c0b28b0 0x7fc41c15e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.557+0000 7fc422da4700 1 --2- 172.21.15.3:0/2973863034 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fc3fc06ac90 0x7fc3fc06d150 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.557+0000 7fc422da4700 1 --2- 172.21.15.3:0/2973863034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc41c15b900 0x7fc41c14f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.557+0000 7fc422da4700 1 --2- 172.21.15.3:0/2973863034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc41c0cf9e0 0x7fc41c14f0d0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.557+0000 7fc422da4700 1 -- 172.21.15.3:0/2973863034 >> 172.21.15.3:0/2973863034 conn(0x7fc41c0b2f20 msgr2=0x7fc41c0aea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:44.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.557+0000 7fc422da4700 1 -- 172.21.15.3:0/2973863034 shutdown_connections 2024-05-08T21:23:44.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.558+0000 7fc422da4700 1 -- 172.21.15.3:0/2973863034 wait complete. 2024-05-08T21:23:44.560 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 2 to deep-scrub 2024-05-08T21:23:44.572 INFO:tasks.ceph:Scrubbing osd.3 2024-05-08T21:23:44.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-05-08T21:23:44.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.712+0000 7faeff4c6700 1 Processor -- start 2024-05-08T21:23:44.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.712+0000 7faeff4c6700 1 -- start start 2024-05-08T21:23:44.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.713+0000 7faeff4c6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faef8154df0 0x7faef80d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.713+0000 7faeff4c6700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faef80d0890 0x7faef80b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.713+0000 7faeff4c6700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef815add0 con 0x7faef80b43b0 2024-05-08T21:23:44.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.713+0000 7faeff4c6700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef815abe0 con 0x7faef8154df0 2024-05-08T21:23:44.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.713+0000 7faeff4c6700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef80d0d20 con 0x7faef80d0890 2024-05-08T21:23:44.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.713+0000 7faefd262700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faef8154df0 0x7faef80d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.713+0000 7faefda63700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7faef80b43b0 0x7faef80ada60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34186/0 (socket says 172.21.15.3:34186) 2024-05-08T21:23:44.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.713+0000 7faefda63700 1 -- 172.21.15.3:0/2442045426 learned_addr learned my addr 172.21.15.3:0/2442045426 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:44.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.714+0000 7faefd262700 1 -- 172.21.15.3:0/2442045426 >> v2:172.21.15.186:3300/0 conn(0x7faef80d0890 msgr2=0x7faef80b3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:44.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.714+0000 7faefd262700 1 --2- 172.21.15.3:0/2442045426 >> v2:172.21.15.186:3300/0 conn(0x7faef80d0890 0x7faef80b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.714+0000 7faefd262700 1 -- 172.21.15.3:0/2442045426 >> v1:172.21.15.3:6789/0 conn(0x7faef80b43b0 legacy=0x7faef80ada60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.714+0000 7faeebfff700 1 -- 172.21.15.3:0/2442045426 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887817726 0 0) 0x7faef815add0 con 0x7faef80b43b0 2024-05-08T21:23:44.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.714+0000 7faefd262700 1 -- 172.21.15.3:0/2442045426 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef815b070 con 0x7faef8154df0 2024-05-08T21:23:44.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.714+0000 7faefd262700 1 --2- 172.21.15.3:0/2442045426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef8154df0 0x7faef80d0350 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7faef80714a0 tx=0x7faef4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531d4d99852f56b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:44.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.714+0000 7faeebfff700 1 -- 172.21.15.3:0/2442045426 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef4011830 con 0x7faef8154df0 2024-05-08T21:23:44.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.714+0000 7faeebfff700 1 -- 172.21.15.3:0/2442045426 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef4011990 con 0x7faef8154df0 2024-05-08T21:23:44.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.714+0000 7faeebfff700 1 -- 172.21.15.3:0/2442045426 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef4011d20 con 0x7faef8154df0 2024-05-08T21:23:44.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.714+0000 7faeff4c6700 1 -- 172.21.15.3:0/2442045426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef8154df0 msgr2=0x7faef80d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.714+0000 7faeff4c6700 1 --2- 172.21.15.3:0/2442045426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef8154df0 0x7faef80d0350 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7faef80714a0 tx=0x7faef4009b20 comp rx=0 tx=0).stop 2024-05-08T21:23:44.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.715+0000 7faeff4c6700 1 -- 172.21.15.3:0/2442045426 shutdown_connections 2024-05-08T21:23:44.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.715+0000 7faeff4c6700 1 --2- 172.21.15.3:0/2442045426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef8154df0 0x7faef80d0350 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.715+0000 7faeff4c6700 1 --2- 172.21.15.3:0/2442045426 >> v2:172.21.15.186:3300/0 conn(0x7faef80d0890 0x7faef80b3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.715+0000 7faeff4c6700 1 -- 172.21.15.3:0/2442045426 >> 172.21.15.3:0/2442045426 conn(0x7faef8099a50 msgr2=0x7faef80b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:44.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.715+0000 7faeff4c6700 1 -- 172.21.15.3:0/2442045426 shutdown_connections 2024-05-08T21:23:44.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.715+0000 7faeff4c6700 1 -- 172.21.15.3:0/2442045426 wait complete. 2024-05-08T21:23:44.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.716+0000 7faeff4c6700 1 Processor -- start 2024-05-08T21:23:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.716+0000 7faeff4c6700 1 -- start start 2024-05-08T21:23:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.717+0000 7faeff4c6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faef80b43b0 0x7faef8133a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.717+0000 7faeff4c6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef80d0890 0x7faef8133f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.717+0000 7faeff4c6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faef8154df0 0x7faef8149000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.717+0000 7faeff4c6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef8131970 con 0x7faef80d0890 2024-05-08T21:23:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.717+0000 7faeff4c6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef80afea0 con 0x7faef80b43b0 2024-05-08T21:23:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.717+0000 7faeff4c6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef80b0180 con 0x7faef8154df0 2024-05-08T21:23:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.717+0000 7faefca61700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef80d0890 0x7faef8133f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.717+0000 7faefca61700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef80d0890 0x7faef8133f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34560/0 (socket says 172.21.15.3:34560) 2024-05-08T21:23:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.717+0000 7faefca61700 1 -- 172.21.15.3:0/1419102748 learned_addr learned my addr 172.21.15.3:0/1419102748 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.717+0000 7faefd262700 1 --2- 172.21.15.3:0/1419102748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faef80b43b0 0x7faef8133a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.717+0000 7faefca61700 1 -- 172.21.15.3:0/1419102748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faef8154df0 msgr2=0x7faef8149000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.717+0000 7faefca61700 1 --2- 172.21.15.3:0/1419102748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faef8154df0 0x7faef8149000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.717+0000 7faefca61700 1 -- 172.21.15.3:0/1419102748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faef80b43b0 msgr2=0x7faef8133a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.717+0000 7faefca61700 1 --2- 172.21.15.3:0/1419102748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faef80b43b0 0x7faef8133a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.717+0000 7faefca61700 1 -- 172.21.15.3:0/1419102748 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef40094a0 con 0x7faef80d0890 2024-05-08T21:23:44.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.717+0000 7faefca61700 1 --2- 172.21.15.3:0/1419102748 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef80d0890 0x7faef8133f40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7faeec00b530 tx=0x7faeec00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:44.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.718+0000 7faee9ffb700 1 -- 172.21.15.3:0/1419102748 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec011830 con 0x7faef80d0890 2024-05-08T21:23:44.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.718+0000 7faee9ffb700 1 -- 172.21.15.3:0/1419102748 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeec011990 con 0x7faef80d0890 2024-05-08T21:23:44.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.718+0000 7faeff4c6700 1 -- 172.21.15.3:0/1419102748 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef80b0460 con 0x7faef80d0890 2024-05-08T21:23:44.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.718+0000 7faee9ffb700 1 -- 172.21.15.3:0/1419102748 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec011d20 con 0x7faef80d0890 2024-05-08T21:23:44.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.718+0000 7faeff4c6700 1 -- 172.21.15.3:0/1419102748 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef814a160 con 0x7faef80d0890 2024-05-08T21:23:44.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.719+0000 7faeff4c6700 1 -- 172.21.15.3:0/1419102748 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faef814a380 con 0x7faef80d0890 2024-05-08T21:23:44.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.719+0000 7faee9ffb700 1 -- 172.21.15.3:0/1419102748 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faeec01a400 con 0x7faef80d0890 2024-05-08T21:23:44.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.720+0000 7faee9ffb700 1 --2- 172.21.15.3:0/1419102748 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faedc06ac90 0x7faedc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.720+0000 7faee9ffb700 1 -- 172.21.15.3:0/1419102748 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7faeec0890a0 con 0x7faef80d0890 2024-05-08T21:23:44.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.720+0000 7faee9ffb700 1 --2- 172.21.15.3:0/1419102748 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x7faedc070b90 0x7faedc072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.720+0000 7faee9ffb700 1 -- 172.21.15.3:0/1419102748 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faedc073660 con 0x7faedc070b90 2024-05-08T21:23:44.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.720+0000 7faefd262700 1 --2- 172.21.15.3:0/1419102748 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faedc06ac90 0x7faedc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.720+0000 7faee9ffb700 1 -- 172.21.15.3:0/1419102748 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7faeec08b760 con 0x7faef80d0890 2024-05-08T21:23:44.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.720+0000 7faefda63700 1 --2- 172.21.15.3:0/1419102748 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x7faedc070b90 0x7faedc072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.720+0000 7faefd262700 1 --2- 172.21.15.3:0/1419102748 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faedc06ac90 0x7faedc06d150 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7faef40090b0 tx=0x7faef401c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:44.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.720+0000 7faefda63700 1 --2- 172.21.15.3:0/1419102748 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x7faedc070b90 0x7faedc072fb0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:44.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.721+0000 7faee9ffb700 1 -- 172.21.15.3:0/1419102748 <== osd.3 v2:172.21.15.3:6800/47118 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7faedc073660 con 0x7faedc070b90 2024-05-08T21:23:44.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.737+0000 7faeff4c6700 1 -- 172.21.15.3:0/1419102748 --> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7faef815b910 con 0x7faedc070b90 2024-05-08T21:23:44.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.737+0000 7faee9ffb700 1 -- 172.21.15.3:0/1419102748 <== osd.3 v2:172.21.15.3:6800/47118 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7faef0001db0 con 0x7faedc070b90 2024-05-08T21:23:44.739 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:23:44.739 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T21:23:44.739 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:23:44.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.737+0000 7faeff4c6700 1 -- 172.21.15.3:0/1419102748 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x7faedc070b90 msgr2=0x7faedc072fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.737+0000 7faeff4c6700 1 --2- 172.21.15.3:0/1419102748 >> [v2:172.21.15.3:6800/47118,v1:172.21.15.3:6801/47118] conn(0x7faedc070b90 0x7faedc072fb0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.737+0000 7faeff4c6700 1 -- 172.21.15.3:0/1419102748 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faedc06ac90 msgr2=0x7faedc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.737+0000 7faeff4c6700 1 --2- 172.21.15.3:0/1419102748 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7faedc06ac90 0x7faedc06d150 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7faef40090b0 tx=0x7faef401c040 comp rx=0 tx=0).stop 2024-05-08T21:23:44.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.737+0000 7faeff4c6700 1 -- 172.21.15.3:0/1419102748 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef80d0890 msgr2=0x7faef8133f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.737+0000 7faeff4c6700 1 --2- 172.21.15.3:0/1419102748 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef80d0890 0x7faef8133f40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7faeec00b530 tx=0x7faeec00b840 comp rx=0 tx=0).stop 2024-05-08T21:23:44.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.737+0000 7faefda63700 1 -- 172.21.15.3:0/1419102748 reap_dead start 2024-05-08T21:23:44.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.737+0000 7faeff4c6700 1 -- 172.21.15.3:0/1419102748 shutdown_connections 2024-05-08T21:23:44.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.737+0000 7faeff4c6700 1 -- 172.21.15.3:0/1419102748 >> 172.21.15.3:0/1419102748 conn(0x7faef8099a50 msgr2=0x7faef815bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:44.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.738+0000 7faeff4c6700 1 -- 172.21.15.3:0/1419102748 shutdown_connections 2024-05-08T21:23:44.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.738+0000 7faeff4c6700 1 -- 172.21.15.3:0/1419102748 wait complete. 2024-05-08T21:23:44.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-05-08T21:23:44.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.887+0000 7f83bd0d5700 1 Processor -- start 2024-05-08T21:23:44.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.887+0000 7f83bd0d5700 1 -- start start 2024-05-08T21:23:44.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.888+0000 7f83bd0d5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83b80cd4e0 0x7f83b806f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.888+0000 7f83bd0d5700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b815c280 con 0x7f83b806f130 2024-05-08T21:23:44.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.888+0000 7f83bd0d5700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b806e780 con 0x7f83b80cecf0 2024-05-08T21:23:44.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.888+0000 7f83bd0d5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b806ea60 con 0x7f83b80cd4e0 2024-05-08T21:23:44.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.888+0000 7f83b659c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f83b806f130 0x7f83b80ce5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34198/0 (socket says 172.21.15.3:34198) 2024-05-08T21:23:44.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.888+0000 7f83b659c700 1 -- 172.21.15.3:0/2263541682 learned_addr learned my addr 172.21.15.3:0/2263541682 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:44.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.888+0000 7f83b6d9d700 1 --2- 172.21.15.3:0/2263541682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83b80cd4e0 0x7f83b806f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.888+0000 7f83b5d9b700 1 -- 172.21.15.3:0/2263541682 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852631219 0 0) 0x7f83b815c280 con 0x7f83b806f130 2024-05-08T21:23:44.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.888+0000 7f83b5d9b700 1 -- 172.21.15.3:0/2263541682 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83a00035b0 con 0x7f83b806f130 2024-05-08T21:23:44.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.888+0000 7f83b5d9b700 1 -- 172.21.15.3:0/2263541682 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869417670 0 0) 0x7f83b806e780 con 0x7f83b80cecf0 2024-05-08T21:23:44.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.889+0000 7f83b5d9b700 1 -- 172.21.15.3:0/2263541682 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b815c280 con 0x7f83b80cecf0 2024-05-08T21:23:44.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.889+0000 7f83b6d9d700 1 -- 172.21.15.3:0/2263541682 >> v1:172.21.15.186:6789/0 conn(0x7f83b80cecf0 legacy=0x7f83b815c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.889+0000 7f83b6d9d700 1 -- 172.21.15.3:0/2263541682 >> v1:172.21.15.3:6789/0 conn(0x7f83b806f130 legacy=0x7f83b80ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.889+0000 7f83b5d9b700 1 -- 172.21.15.3:0/2263541682 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1200115645 0 0) 0x7f83a00035b0 con 0x7f83b806f130 2024-05-08T21:23:44.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.889+0000 7f83b6d9d700 1 -- 172.21.15.3:0/2263541682 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b806ed40 con 0x7f83b80cd4e0 2024-05-08T21:23:44.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.889+0000 7f83b6d9d700 1 --2- 172.21.15.3:0/2263541682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83b80cd4e0 0x7f83b806f910 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f83a8009870 tx=0x7f83a8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a74f5a257813f826 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:44.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.889+0000 7f83b5d9b700 1 -- 172.21.15.3:0/2263541682 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac001670 con 0x7f83b80cd4e0 2024-05-08T21:23:44.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.889+0000 7f83b5d9b700 1 -- 172.21.15.3:0/2263541682 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a8004030 con 0x7f83b80cd4e0 2024-05-08T21:23:44.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.889+0000 7f83b5d9b700 1 -- 172.21.15.3:0/2263541682 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a80043c0 con 0x7f83b80cd4e0 2024-05-08T21:23:44.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.890+0000 7f83bd0d5700 1 -- 172.21.15.3:0/2263541682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83b80cd4e0 msgr2=0x7f83b806f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.890+0000 7f83bd0d5700 1 --2- 172.21.15.3:0/2263541682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83b80cd4e0 0x7f83b806f910 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f83a8009870 tx=0x7f83a8009b80 comp rx=0 tx=0).stop 2024-05-08T21:23:44.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.890+0000 7f83bd0d5700 1 -- 172.21.15.3:0/2263541682 shutdown_connections 2024-05-08T21:23:44.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.890+0000 7f83bd0d5700 1 --2- 172.21.15.3:0/2263541682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83b80cd4e0 0x7f83b806f910 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.890+0000 7f83bd0d5700 1 -- 172.21.15.3:0/2263541682 >> 172.21.15.3:0/2263541682 conn(0x7f83b80d0320 msgr2=0x7f83b80d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:44.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.890+0000 7f83bd0d5700 1 -- 172.21.15.3:0/2263541682 shutdown_connections 2024-05-08T21:23:44.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.890+0000 7f83bd0d5700 1 -- 172.21.15.3:0/2263541682 wait complete. 2024-05-08T21:23:44.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.891+0000 7f83bd0d5700 1 Processor -- start 2024-05-08T21:23:44.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.892+0000 7f83bd0d5700 1 -- start start 2024-05-08T21:23:44.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.892+0000 7f83bd0d5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83b806f130 0x7f83b815aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.892+0000 7f83bd0d5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83b80cd4e0 0x7f83b81f6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.892+0000 7f83bd0d5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83b80cecf0 0x7f83b81f4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.892+0000 7f83bd0d5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b815afa0 con 0x7f83b806f130 2024-05-08T21:23:44.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.892+0000 7f83bd0d5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b81f4ff0 con 0x7f83b80cecf0 2024-05-08T21:23:44.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.892+0000 7f83bd0d5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b81f52d0 con 0x7f83b80cd4e0 2024-05-08T21:23:44.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.892+0000 7f83b6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83b806f130 0x7f83b815aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.892+0000 7f83b6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83b806f130 0x7f83b815aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34568/0 (socket says 172.21.15.3:34568) 2024-05-08T21:23:44.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.892+0000 7f83b6d9d700 1 -- 172.21.15.3:0/898264069 learned_addr learned my addr 172.21.15.3:0/898264069 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:44.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.893+0000 7f83b759e700 1 --2- 172.21.15.3:0/898264069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83b80cecf0 0x7f83b81f4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.893+0000 7f83b659c700 1 --2- 172.21.15.3:0/898264069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83b80cd4e0 0x7f83b81f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.893+0000 7f83b6d9d700 1 -- 172.21.15.3:0/898264069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83b80cd4e0 msgr2=0x7f83b81f6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.893+0000 7f83b6d9d700 1 --2- 172.21.15.3:0/898264069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83b80cd4e0 0x7f83b81f6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.893+0000 7f83b6d9d700 1 -- 172.21.15.3:0/898264069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83b80cecf0 msgr2=0x7f83b81f4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:44.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.893+0000 7f83b6d9d700 1 --2- 172.21.15.3:0/898264069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83b80cecf0 0x7f83b81f4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:44.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.893+0000 7f83b6d9d700 1 -- 172.21.15.3:0/898264069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a80094a0 con 0x7f83b806f130 2024-05-08T21:23:44.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.893+0000 7f83b659c700 1 --2- 172.21.15.3:0/898264069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83b80cd4e0 0x7f83b81f6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:44.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.893+0000 7f83b6d9d700 1 --2- 172.21.15.3:0/898264069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83b806f130 0x7f83b815aa60 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f83a8000c00 tx=0x7f83a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:44.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.893+0000 7f839f7fe700 1 -- 172.21.15.3:0/898264069 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8011630 con 0x7f83b806f130 2024-05-08T21:23:44.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.893+0000 7f839f7fe700 1 -- 172.21.15.3:0/898264069 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a801f070 con 0x7f83b806f130 2024-05-08T21:23:44.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.893+0000 7f83bd0d5700 1 -- 172.21.15.3:0/898264069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b81f55b0 con 0x7f83b806f130 2024-05-08T21:23:44.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.893+0000 7f839f7fe700 1 -- 172.21.15.3:0/898264069 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a80119f0 con 0x7f83b806f130 2024-05-08T21:23:44.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.893+0000 7f83bd0d5700 1 -- 172.21.15.3:0/898264069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b81f5b70 con 0x7f83b806f130 2024-05-08T21:23:44.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.894+0000 7f83bd0d5700 1 -- 172.21.15.3:0/898264069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b8068000 con 0x7f83b806f130 2024-05-08T21:23:44.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.895+0000 7f839f7fe700 1 -- 172.21.15.3:0/898264069 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f83a8011b50 con 0x7f83b806f130 2024-05-08T21:23:44.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.895+0000 7f839f7fe700 1 --2- 172.21.15.3:0/898264069 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f83a406ac90 0x7f83a406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:44.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.895+0000 7f839f7fe700 1 -- 172.21.15.3:0/898264069 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f83a8021070 con 0x7f83b806f130 2024-05-08T21:23:44.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.896+0000 7f83b659c700 1 --2- 172.21.15.3:0/898264069 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f83a406ac90 0x7f83a406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:44.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.896+0000 7f83b659c700 1 --2- 172.21.15.3:0/898264069 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f83a406ac90 0x7f83a406d150 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f83ac001eb0 tx=0x7f83ac001ca0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:44.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:44.900+0000 7f839f7fe700 1 -- 172.21.15.3:0/898264069 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f83a8055f10 con 0x7f83b806f130 2024-05-08T21:23:45.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.116+0000 7f83bd0d5700 1 -- 172.21.15.3:0/898264069 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f83b81f5440 con 0x7f83a406ac90 2024-05-08T21:23:45.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.116+0000 7f839f7fe700 1 -- 172.21.15.3:0/898264069 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f83b81f5440 con 0x7f83a406ac90 2024-05-08T21:23:45.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.118+0000 7f83bd0d5700 1 -- 172.21.15.3:0/898264069 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f83a406ac90 msgr2=0x7f83a406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.118+0000 7f83bd0d5700 1 --2- 172.21.15.3:0/898264069 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f83a406ac90 0x7f83a406d150 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f83ac001eb0 tx=0x7f83ac001ca0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.118+0000 7f83bd0d5700 1 -- 172.21.15.3:0/898264069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83b806f130 msgr2=0x7f83b815aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.118+0000 7f83bd0d5700 1 --2- 172.21.15.3:0/898264069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83b806f130 0x7f83b815aa60 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f83a8000c00 tx=0x7f83a8004750 comp rx=0 tx=0).stop 2024-05-08T21:23:45.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.119+0000 7f83bd0d5700 1 -- 172.21.15.3:0/898264069 shutdown_connections 2024-05-08T21:23:45.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.119+0000 7f83bd0d5700 1 --2- 172.21.15.3:0/898264069 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f83a406ac90 0x7f83a406d150 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.119+0000 7f83bd0d5700 1 --2- 172.21.15.3:0/898264069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83b80cecf0 0x7f83b81f4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.119+0000 7f83bd0d5700 1 --2- 172.21.15.3:0/898264069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83b806f130 0x7f83b815aa60 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.119+0000 7f83bd0d5700 1 --2- 172.21.15.3:0/898264069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83b80cd4e0 0x7f83b81f6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.119+0000 7f83bd0d5700 1 -- 172.21.15.3:0/898264069 >> 172.21.15.3:0/898264069 conn(0x7f83b80d0320 msgr2=0x7f83b80cb910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:45.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.119+0000 7f83bd0d5700 1 -- 172.21.15.3:0/898264069 shutdown_connections 2024-05-08T21:23:45.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.119+0000 7f83bd0d5700 1 -- 172.21.15.3:0/898264069 wait complete. 2024-05-08T21:23:45.121 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 3 to deep-scrub 2024-05-08T21:23:45.134 INFO:tasks.ceph:Scrubbing osd.4 2024-05-08T21:23:45.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-05-08T21:23:45.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.273+0000 7fe13718d700 1 Processor -- start 2024-05-08T21:23:45.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.274+0000 7fe13718d700 1 -- start start 2024-05-08T21:23:45.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.274+0000 7fe13718d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe13015b4e0 0x7fe1300b1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.274+0000 7fe13718d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe13015b8c0 0x7fe1300d1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.274+0000 7fe13718d700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1300cf560 con 0x7fe1300d1740 2024-05-08T21:23:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.274+0000 7fe13718d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1300cefe0 con 0x7fe13015b8c0 2024-05-08T21:23:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.274+0000 7fe13718d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1300d1bb0 con 0x7fe13015b4e0 2024-05-08T21:23:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.275+0000 7fe12ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe13015b8c0 0x7fe1300d1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.275+0000 7fe12ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe13015b8c0 0x7fe1300d1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34584/0 (socket says 172.21.15.3:34584) 2024-05-08T21:23:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.275+0000 7fe12ffff700 1 -- 172.21.15.3:0/1728298864 learned_addr learned my addr 172.21.15.3:0/1728298864 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.275+0000 7fe134f29700 1 --2- 172.21.15.3:0/1728298864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe13015b4e0 0x7fe1300b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.275+0000 7fe12ffff700 1 -- 172.21.15.3:0/1728298864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe13015b4e0 msgr2=0x7fe1300b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.275+0000 7fe12ffff700 1 --2- 172.21.15.3:0/1728298864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe13015b4e0 0x7fe1300b1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.275+0000 7fe12ffff700 1 -- 172.21.15.3:0/1728298864 >> v1:172.21.15.186:6789/0 conn(0x7fe1300d1740 legacy=0x7fe1300cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.275+0000 7fe12ffff700 1 -- 172.21.15.3:0/1728298864 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1300ccff0 con 0x7fe13015b8c0 2024-05-08T21:23:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.275+0000 7fe134f29700 1 --2- 172.21.15.3:0/1728298864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe13015b4e0 0x7fe1300b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.275+0000 7fe12ffff700 1 --2- 172.21.15.3:0/1728298864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe13015b8c0 0x7fe1300d1200 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe1240098f0 tx=0x7fe124009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97eabe6f097b13c server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.275+0000 7fe12f7fe700 1 -- 172.21.15.3:0/1728298864 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe124004230 con 0x7fe13015b8c0 2024-05-08T21:23:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.275+0000 7fe12f7fe700 1 -- 172.21.15.3:0/1728298864 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe124004390 con 0x7fe13015b8c0 2024-05-08T21:23:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.275+0000 7fe12f7fe700 1 -- 172.21.15.3:0/1728298864 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1240036a0 con 0x7fe13015b8c0 2024-05-08T21:23:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.276+0000 7fe13718d700 1 -- 172.21.15.3:0/1728298864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe13015b8c0 msgr2=0x7fe1300d1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.276+0000 7fe13718d700 1 --2- 172.21.15.3:0/1728298864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe13015b8c0 0x7fe1300d1200 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe1240098f0 tx=0x7fe124009c00 comp rx=0 tx=0).stop 2024-05-08T21:23:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.276+0000 7fe13718d700 1 -- 172.21.15.3:0/1728298864 shutdown_connections 2024-05-08T21:23:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.276+0000 7fe13718d700 1 --2- 172.21.15.3:0/1728298864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe13015b4e0 0x7fe1300b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.276+0000 7fe13718d700 1 --2- 172.21.15.3:0/1728298864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe13015b8c0 0x7fe1300d1200 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.276+0000 7fe13718d700 1 -- 172.21.15.3:0/1728298864 >> 172.21.15.3:0/1728298864 conn(0x7fe13008afe0 msgr2=0x7fe13008b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.276+0000 7fe13718d700 1 -- 172.21.15.3:0/1728298864 shutdown_connections 2024-05-08T21:23:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.276+0000 7fe13718d700 1 -- 172.21.15.3:0/1728298864 wait complete. 2024-05-08T21:23:45.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.277+0000 7fe13718d700 1 Processor -- start 2024-05-08T21:23:45.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.277+0000 7fe13718d700 1 -- start start 2024-05-08T21:23:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.278+0000 7fe13718d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1300d1740 0x7fe1301ee6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.278+0000 7fe13718d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe13015b4e0 0x7fe130158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.278+0000 7fe13718d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe13015b8c0 0x7fe1301f2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.278+0000 7fe13718d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1301eec10 con 0x7fe13015b8c0 2024-05-08T21:23:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.278+0000 7fe13718d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe130157b60 con 0x7fe1300d1740 2024-05-08T21:23:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.278+0000 7fe13718d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe130157e40 con 0x7fe13015b4e0 2024-05-08T21:23:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.278+0000 7fe13572a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe13015b8c0 0x7fe1301f2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.278+0000 7fe13572a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe13015b8c0 0x7fe1301f2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34594/0 (socket says 172.21.15.3:34594) 2024-05-08T21:23:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.278+0000 7fe13572a700 1 -- 172.21.15.3:0/3635244565 learned_addr learned my addr 172.21.15.3:0/3635244565 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.279+0000 7fe134f29700 1 --2- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1300d1740 0x7fe1301ee6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.279+0000 7fe12ffff700 1 --2- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe13015b4e0 0x7fe130158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.279+0000 7fe13572a700 1 -- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe13015b4e0 msgr2=0x7fe130158c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.279+0000 7fe13572a700 1 --2- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe13015b4e0 0x7fe130158c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.279+0000 7fe13572a700 1 -- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1300d1740 msgr2=0x7fe1301ee6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.279+0000 7fe13572a700 1 --2- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1300d1740 0x7fe1301ee6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.279+0000 7fe13572a700 1 -- 172.21.15.3:0/3635244565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe124009550 con 0x7fe13015b8c0 2024-05-08T21:23:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.279+0000 7fe12ffff700 1 --2- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe13015b4e0 0x7fe130158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:23:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.279+0000 7fe13572a700 1 --2- 172.21.15.3:0/3635244565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe13015b8c0 0x7fe1301f2350 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe12800cc10 tx=0x7fe12800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.279+0000 7fe12d7fa700 1 -- 172.21.15.3:0/3635244565 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12800abb0 con 0x7fe13015b8c0 2024-05-08T21:23:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.279+0000 7fe12d7fa700 1 -- 172.21.15.3:0/3635244565 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12800ad10 con 0x7fe13015b8c0 2024-05-08T21:23:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.279+0000 7fe13718d700 1 -- 172.21.15.3:0/3635244565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe130158120 con 0x7fe13015b8c0 2024-05-08T21:23:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.279+0000 7fe12d7fa700 1 -- 172.21.15.3:0/3635244565 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe128012660 con 0x7fe13015b8c0 2024-05-08T21:23:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.279+0000 7fe13718d700 1 -- 172.21.15.3:0/3635244565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1301586e0 con 0x7fe13015b8c0 2024-05-08T21:23:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.280+0000 7fe13718d700 1 -- 172.21.15.3:0/3635244565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe1300b1740 con 0x7fe13015b8c0 2024-05-08T21:23:45.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.281+0000 7fe12d7fa700 1 -- 172.21.15.3:0/3635244565 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe1280127c0 con 0x7fe13015b8c0 2024-05-08T21:23:45.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.281+0000 7fe12d7fa700 1 --2- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe11c06ac90 0x7fe11c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.281+0000 7fe12d7fa700 1 -- 172.21.15.3:0/3635244565 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe1280899c0 con 0x7fe13015b8c0 2024-05-08T21:23:45.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.281+0000 7fe12d7fa700 1 --2- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x7fe11c070b90 0x7fe11c072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.281+0000 7fe12d7fa700 1 -- 172.21.15.3:0/3635244565 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe11c073660 con 0x7fe11c070b90 2024-05-08T21:23:45.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.281+0000 7fe12d7fa700 1 -- 172.21.15.3:0/3635244565 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe12808e2a0 con 0x7fe13015b8c0 2024-05-08T21:23:45.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.282+0000 7fe134f29700 1 --2- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe11c06ac90 0x7fe11c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.282+0000 7fe12ffff700 1 --2- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x7fe11c070b90 0x7fe11c072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.282+0000 7fe134f29700 1 --2- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe11c06ac90 0x7fe11c06d150 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe120006fd0 tx=0x7fe120008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:45.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.282+0000 7fe12ffff700 1 --2- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x7fe11c070b90 0x7fe11c072fb0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:45.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.283+0000 7fe12d7fa700 1 -- 172.21.15.3:0/3635244565 <== osd.4 v2:172.21.15.186:6810/47344 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fe11c073660 con 0x7fe11c070b90 2024-05-08T21:23:45.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.299+0000 7fe13718d700 1 -- 172.21.15.3:0/3635244565 --> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe1300cf1f0 con 0x7fe11c070b90 2024-05-08T21:23:45.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.300+0000 7fe12d7fa700 1 -- 172.21.15.3:0/3635244565 <== osd.4 v2:172.21.15.186:6810/47344 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe1240036a0 con 0x7fe11c070b90 2024-05-08T21:23:45.301 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:23:45.301 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T21:23:45.301 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:23:45.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.300+0000 7fe13718d700 1 -- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x7fe11c070b90 msgr2=0x7fe11c072fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.300+0000 7fe13718d700 1 --2- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:6810/47344,v1:172.21.15.186:6813/47344] conn(0x7fe11c070b90 0x7fe11c072fb0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.300+0000 7fe13718d700 1 -- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe11c06ac90 msgr2=0x7fe11c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.300+0000 7fe13718d700 1 --2- 172.21.15.3:0/3635244565 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fe11c06ac90 0x7fe11c06d150 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe120006fd0 tx=0x7fe120008040 comp rx=0 tx=0).stop 2024-05-08T21:23:45.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.300+0000 7fe13718d700 1 -- 172.21.15.3:0/3635244565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe13015b8c0 msgr2=0x7fe1301f2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.300+0000 7fe13718d700 1 --2- 172.21.15.3:0/3635244565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe13015b8c0 0x7fe1301f2350 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe12800cc10 tx=0x7fe12800a060 comp rx=0 tx=0).stop 2024-05-08T21:23:45.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.300+0000 7fe13572a700 1 -- 172.21.15.3:0/3635244565 reap_dead start 2024-05-08T21:23:45.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.300+0000 7fe13718d700 1 -- 172.21.15.3:0/3635244565 shutdown_connections 2024-05-08T21:23:45.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.300+0000 7fe13718d700 1 -- 172.21.15.3:0/3635244565 >> 172.21.15.3:0/3635244565 conn(0x7fe13008afe0 msgr2=0x7fe1300705c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:45.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.300+0000 7fe13718d700 1 -- 172.21.15.3:0/3635244565 shutdown_connections 2024-05-08T21:23:45.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.300+0000 7fe13718d700 1 -- 172.21.15.3:0/3635244565 wait complete. 2024-05-08T21:23:45.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-05-08T21:23:45.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.454+0000 7fb4e67d9700 1 Processor -- start 2024-05-08T21:23:45.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.454+0000 7fb4e67d9700 1 -- start start 2024-05-08T21:23:45.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4e67d9700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb4e015a8e0 0x7fb4e00c5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4e67d9700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb4e00b1450 0x7fb4e00c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4e67d9700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e00ce5c0 con 0x7fb4e00cea50 2024-05-08T21:23:45.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4e67d9700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4e00ce430 con 0x7fb4e00b1450 2024-05-08T21:23:45.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4e67d9700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4e00b08f0 con 0x7fb4e015a8e0 2024-05-08T21:23:45.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4df7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb4e00b1450 0x7fb4e00c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4df7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb4e00b1450 0x7fb4e00c5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34608/0 (socket says 172.21.15.3:34608) 2024-05-08T21:23:45.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4df7fe700 1 -- 172.21.15.3:0/3315345026 learned_addr learned my addr 172.21.15.3:0/3315345026 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:45.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4dffff700 1 --2- 172.21.15.3:0/3315345026 >> v2:172.21.15.186:3300/0 conn(0x7fb4e015a8e0 0x7fb4e00c5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4df7fe700 1 -- 172.21.15.3:0/3315345026 >> v2:172.21.15.186:3300/0 conn(0x7fb4e015a8e0 msgr2=0x7fb4e00c5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:45.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4df7fe700 1 --2- 172.21.15.3:0/3315345026 >> v2:172.21.15.186:3300/0 conn(0x7fb4e015a8e0 0x7fb4e00c5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4df7fe700 1 -- 172.21.15.3:0/3315345026 >> v1:172.21.15.186:6789/0 conn(0x7fb4e00cea50 legacy=0x7fb4e00ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4df7fe700 1 -- 172.21.15.3:0/3315345026 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e00b0b60 con 0x7fb4e00b1450 2024-05-08T21:23:45.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4dffff700 1 --2- 172.21.15.3:0/3315345026 >> v2:172.21.15.186:3300/0 conn(0x7fb4e015a8e0 0x7fb4e00c5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:45.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4df7fe700 1 --2- 172.21.15.3:0/3315345026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4e00b1450 0x7fb4e00c5b60 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0067520 tx=0x7fb4d4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4aa252c2a36f8d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:45.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4deffd700 1 -- 172.21.15.3:0/3315345026 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d400b870 con 0x7fb4e00b1450 2024-05-08T21:23:45.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4deffd700 1 -- 172.21.15.3:0/3315345026 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d400b9d0 con 0x7fb4e00b1450 2024-05-08T21:23:45.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.455+0000 7fb4deffd700 1 -- 172.21.15.3:0/3315345026 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d400bd60 con 0x7fb4e00b1450 2024-05-08T21:23:45.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.456+0000 7fb4e67d9700 1 -- 172.21.15.3:0/3315345026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4e00b1450 msgr2=0x7fb4e00c5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.456+0000 7fb4e67d9700 1 --2- 172.21.15.3:0/3315345026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4e00b1450 0x7fb4e00c5b60 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0067520 tx=0x7fb4d4009bd0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.456+0000 7fb4e67d9700 1 -- 172.21.15.3:0/3315345026 shutdown_connections 2024-05-08T21:23:45.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.456+0000 7fb4e67d9700 1 --2- 172.21.15.3:0/3315345026 >> v2:172.21.15.186:3300/0 conn(0x7fb4e015a8e0 0x7fb4e00c5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.456+0000 7fb4e67d9700 1 --2- 172.21.15.3:0/3315345026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4e00b1450 0x7fb4e00c5b60 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.456+0000 7fb4e67d9700 1 -- 172.21.15.3:0/3315345026 >> 172.21.15.3:0/3315345026 conn(0x7fb4e00d03a0 msgr2=0x7fb4e00d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:45.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.456+0000 7fb4e67d9700 1 -- 172.21.15.3:0/3315345026 shutdown_connections 2024-05-08T21:23:45.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.456+0000 7fb4e67d9700 1 -- 172.21.15.3:0/3315345026 wait complete. 2024-05-08T21:23:45.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.457+0000 7fb4e67d9700 1 Processor -- start 2024-05-08T21:23:45.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.458+0000 7fb4e67d9700 1 -- start start 2024-05-08T21:23:45.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.458+0000 7fb4e67d9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4e00b1450 0x7fb4e0147c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.458+0000 7fb4e67d9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4e00cea50 0x7fb4e015cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.458+0000 7fb4e67d9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4e015a8e0 0x7fb4e015bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.458+0000 7fb4e67d9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e00ad300 con 0x7fb4e00cea50 2024-05-08T21:23:45.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.458+0000 7fb4e67d9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e00ad5e0 con 0x7fb4e015a8e0 2024-05-08T21:23:45.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.458+0000 7fb4e67d9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e00ad8c0 con 0x7fb4e00b1450 2024-05-08T21:23:45.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.459+0000 7fb4df7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4e00cea50 0x7fb4e015cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.459+0000 7fb4df7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4e00cea50 0x7fb4e015cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34616/0 (socket says 172.21.15.3:34616) 2024-05-08T21:23:45.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.459+0000 7fb4df7fe700 1 -- 172.21.15.3:0/101404547 learned_addr learned my addr 172.21.15.3:0/101404547 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:45.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.459+0000 7fb4e4d76700 1 --2- 172.21.15.3:0/101404547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4e015a8e0 0x7fb4e015bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.459+0000 7fb4dffff700 1 --2- 172.21.15.3:0/101404547 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4e00b1450 0x7fb4e0147c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.459+0000 7fb4df7fe700 1 -- 172.21.15.3:0/101404547 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4e00b1450 msgr2=0x7fb4e0147c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.459+0000 7fb4df7fe700 1 --2- 172.21.15.3:0/101404547 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4e00b1450 0x7fb4e0147c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.459+0000 7fb4df7fe700 1 -- 172.21.15.3:0/101404547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4e015a8e0 msgr2=0x7fb4e015bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.459+0000 7fb4df7fe700 1 --2- 172.21.15.3:0/101404547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4e015a8e0 0x7fb4e015bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.459+0000 7fb4df7fe700 1 -- 172.21.15.3:0/101404547 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d4009550 con 0x7fb4e00cea50 2024-05-08T21:23:45.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.459+0000 7fb4df7fe700 1 --2- 172.21.15.3:0/101404547 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4e00cea50 0x7fb4e015cdc0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4000c00 tx=0x7fb4d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:45.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.459+0000 7fb4dcff9700 1 -- 172.21.15.3:0/101404547 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4004310 con 0x7fb4e00cea50 2024-05-08T21:23:45.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.459+0000 7fb4dcff9700 1 -- 172.21.15.3:0/101404547 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d401e070 con 0x7fb4e00cea50 2024-05-08T21:23:45.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.459+0000 7fb4e67d9700 1 -- 172.21.15.3:0/101404547 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e00adba0 con 0x7fb4e00cea50 2024-05-08T21:23:45.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.459+0000 7fb4dcff9700 1 -- 172.21.15.3:0/101404547 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d40036a0 con 0x7fb4e00cea50 2024-05-08T21:23:45.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.459+0000 7fb4e67d9700 1 -- 172.21.15.3:0/101404547 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e0148140 con 0x7fb4e00cea50 2024-05-08T21:23:45.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.460+0000 7fb4e67d9700 1 -- 172.21.15.3:0/101404547 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4e00ae0c0 con 0x7fb4e00cea50 2024-05-08T21:23:45.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.461+0000 7fb4dcff9700 1 -- 172.21.15.3:0/101404547 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb4d4020030 con 0x7fb4e00cea50 2024-05-08T21:23:45.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.461+0000 7fb4dcff9700 1 --2- 172.21.15.3:0/101404547 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4cc06ac90 0x7fb4cc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.461+0000 7fb4dcff9700 1 -- 172.21.15.3:0/101404547 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb4d4089fd0 con 0x7fb4e00cea50 2024-05-08T21:23:45.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.462+0000 7fb4dffff700 1 --2- 172.21.15.3:0/101404547 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4cc06ac90 0x7fb4cc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.462+0000 7fb4dffff700 1 --2- 172.21.15.3:0/101404547 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4cc06ac90 0x7fb4cc06d150 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb4d00099d0 tx=0x7fb4d0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:45.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.466+0000 7fb4dcff9700 1 -- 172.21.15.3:0/101404547 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb4e00ae0c0 con 0x7fb4e00cea50 2024-05-08T21:23:45.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.695+0000 7fb4e67d9700 1 -- 172.21.15.3:0/101404547 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4e00ad750 con 0x7fb4cc06ac90 2024-05-08T21:23:45.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.695+0000 7fb4dcff9700 1 -- 172.21.15.3:0/101404547 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb4e00ad750 con 0x7fb4cc06ac90 2024-05-08T21:23:45.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.698+0000 7fb4e67d9700 1 -- 172.21.15.3:0/101404547 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4cc06ac90 msgr2=0x7fb4cc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.698+0000 7fb4e67d9700 1 --2- 172.21.15.3:0/101404547 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4cc06ac90 0x7fb4cc06d150 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb4d00099d0 tx=0x7fb4d0008040 comp rx=0 tx=0).stop 2024-05-08T21:23:45.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.698+0000 7fb4e67d9700 1 -- 172.21.15.3:0/101404547 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4e00cea50 msgr2=0x7fb4e015cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.698+0000 7fb4e67d9700 1 --2- 172.21.15.3:0/101404547 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4e00cea50 0x7fb4e015cdc0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4000c00 tx=0x7fb4d4004750 comp rx=0 tx=0).stop 2024-05-08T21:23:45.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.698+0000 7fb4e67d9700 1 -- 172.21.15.3:0/101404547 shutdown_connections 2024-05-08T21:23:45.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.698+0000 7fb4e67d9700 1 --2- 172.21.15.3:0/101404547 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4e00b1450 0x7fb4e0147c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.698+0000 7fb4e67d9700 1 --2- 172.21.15.3:0/101404547 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb4cc06ac90 0x7fb4cc06d150 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.698+0000 7fb4e67d9700 1 --2- 172.21.15.3:0/101404547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4e015a8e0 0x7fb4e015bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.698+0000 7fb4e67d9700 1 --2- 172.21.15.3:0/101404547 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4e00cea50 0x7fb4e015cdc0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.698+0000 7fb4e67d9700 1 -- 172.21.15.3:0/101404547 >> 172.21.15.3:0/101404547 conn(0x7fb4e00d03a0 msgr2=0x7fb4e0159ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:45.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.698+0000 7fb4e67d9700 1 -- 172.21.15.3:0/101404547 shutdown_connections 2024-05-08T21:23:45.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.698+0000 7fb4e67d9700 1 -- 172.21.15.3:0/101404547 wait complete. 2024-05-08T21:23:45.701 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 4 to deep-scrub 2024-05-08T21:23:45.713 INFO:tasks.ceph:Scrubbing osd.5 2024-05-08T21:23:45.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-05-08T21:23:45.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.852+0000 7fd6759b7700 1 Processor -- start 2024-05-08T21:23:45.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.852+0000 7fd6759b7700 1 -- start start 2024-05-08T21:23:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.853+0000 7fd6759b7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6700d1e50 0x7fd6700ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.853+0000 7fd6759b7700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6700cf560 con 0x7fd6700d2230 2024-05-08T21:23:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.853+0000 7fd6759b7700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6700ce950 con 0x7fd670161320 2024-05-08T21:23:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.853+0000 7fd6759b7700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6700cec30 con 0x7fd6700d1e50 2024-05-08T21:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.853+0000 7fd66e7fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd6700d2230 0x7fd670160c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34214/0 (socket says 172.21.15.3:34214) 2024-05-08T21:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.853+0000 7fd66e7fc700 1 -- 172.21.15.3:0/2149026961 learned_addr learned my addr 172.21.15.3:0/2149026961 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.853+0000 7fd66effd700 1 --2- 172.21.15.3:0/2149026961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6700d1e50 0x7fd6700ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.853+0000 7fd66dffb700 1 -- 172.21.15.3:0/2149026961 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685852989 0 0) 0x7fd6700cf560 con 0x7fd6700d2230 2024-05-08T21:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.853+0000 7fd66dffb700 1 -- 172.21.15.3:0/2149026961 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6580035b0 con 0x7fd6700d2230 2024-05-08T21:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66dffb700 1 -- 172.21.15.3:0/2149026961 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274740148 0 0) 0x7fd6700ce950 con 0x7fd670161320 2024-05-08T21:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66dffb700 1 -- 172.21.15.3:0/2149026961 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6700cf560 con 0x7fd670161320 2024-05-08T21:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66dffb700 1 -- 172.21.15.3:0/2149026961 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1381955466 0 0) 0x7fd6580035b0 con 0x7fd6700d2230 2024-05-08T21:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66dffb700 1 -- 172.21.15.3:0/2149026961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6700d1e50 msgr2=0x7fd6700ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66dffb700 1 --2- 172.21.15.3:0/2149026961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6700d1e50 0x7fd6700ce2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66dffb700 1 -- 172.21.15.3:0/2149026961 >> v1:172.21.15.186:6789/0 conn(0x7fd670161320 legacy=0x7fd67006bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66dffb700 1 -- 172.21.15.3:0/2149026961 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6700cef10 con 0x7fd6700d2230 2024-05-08T21:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66dffb700 1 -- 172.21.15.3:0/2149026961 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7fd664001670 con 0x7fd6700d2230 2024-05-08T21:23:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66dffb700 1 -- 172.21.15.3:0/2149026961 >> v1:172.21.15.3:6789/0 conn(0x7fd6700d2230 legacy=0x7fd670160c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66dffb700 1 --2- 172.21.15.3:0/2149026961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd658003bc0 0x7fd658003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66dffb700 1 --2- 172.21.15.3:0/2149026961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd658004850 0x7fd658004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66dffb700 1 --2- 172.21.15.3:0/2149026961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6700d1e50 0x7fd658005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66dffb700 1 -- 172.21.15.3:0/2149026961 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6700cec30 con 0x7fd658004850 2024-05-08T21:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66dffb700 1 -- 172.21.15.3:0/2149026961 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65800bac0 con 0x7fd658003bc0 2024-05-08T21:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66dffb700 1 -- 172.21.15.3:0/2149026961 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd65800bc80 con 0x7fd6700d1e50 2024-05-08T21:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66e7fc700 1 --2- 172.21.15.3:0/2149026961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd658004850 0x7fd658004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66effd700 1 --2- 172.21.15.3:0/2149026961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd658003bc0 0x7fd658003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66f7fe700 1 --2- 172.21.15.3:0/2149026961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6700d1e50 0x7fd658005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66e7fc700 1 -- 172.21.15.3:0/2149026961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6700d1e50 msgr2=0x7fd658005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66e7fc700 1 --2- 172.21.15.3:0/2149026961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6700d1e50 0x7fd658005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66e7fc700 1 -- 172.21.15.3:0/2149026961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd658003bc0 msgr2=0x7fd658003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66e7fc700 1 --2- 172.21.15.3:0/2149026961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd658003bc0 0x7fd658003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.854+0000 7fd66e7fc700 1 -- 172.21.15.3:0/2149026961 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6580035b0 con 0x7fd658004850 2024-05-08T21:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.855+0000 7fd66e7fc700 1 --2- 172.21.15.3:0/2149026961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd658004850 0x7fd658004cd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd66400c160 tx=0x7fd658005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c585ca88909b3a server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.855+0000 7fd66dffb700 1 -- 172.21.15.3:0/2149026961 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66400cca0 con 0x7fd658004850 2024-05-08T21:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.855+0000 7fd66dffb700 1 -- 172.21.15.3:0/2149026961 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66400ce20 con 0x7fd658004850 2024-05-08T21:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.855+0000 7fd66dffb700 1 -- 172.21.15.3:0/2149026961 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664005ba0 con 0x7fd658004850 2024-05-08T21:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.855+0000 7fd6759b7700 1 -- 172.21.15.3:0/2149026961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd658004850 msgr2=0x7fd658004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.855+0000 7fd6759b7700 1 --2- 172.21.15.3:0/2149026961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd658004850 0x7fd658004cd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd66400c160 tx=0x7fd658005880 comp rx=0 tx=0).stop 2024-05-08T21:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.855+0000 7fd66f7fe700 1 -- 172.21.15.3:0/2149026961 reap_dead start 2024-05-08T21:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.855+0000 7fd6759b7700 1 -- 172.21.15.3:0/2149026961 shutdown_connections 2024-05-08T21:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.855+0000 7fd6759b7700 1 -- 172.21.15.3:0/2149026961 >> 172.21.15.3:0/2149026961 conn(0x7fd67008afe0 msgr2=0x7fd6700d35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.855+0000 7fd6759b7700 1 -- 172.21.15.3:0/2149026961 shutdown_connections 2024-05-08T21:23:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.855+0000 7fd6759b7700 1 -- 172.21.15.3:0/2149026961 wait complete. 2024-05-08T21:23:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.857+0000 7fd6759b7700 1 Processor -- start 2024-05-08T21:23:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.857+0000 7fd6759b7700 1 -- start start 2024-05-08T21:23:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd6759b7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd658004850 0x7fd670155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd6759b7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6700cc640 0x7fd6700b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd6759b7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd67006c5c0 0x7fd67006ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd6759b7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6700b6420 con 0x7fd67006c5c0 2024-05-08T21:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd6759b7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67006cf90 con 0x7fd658004850 2024-05-08T21:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd6759b7700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6701560f0 con 0x7fd6700cc640 2024-05-08T21:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd66f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd67006c5c0 0x7fd67006ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd66f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd67006c5c0 0x7fd67006ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34646/0 (socket says 172.21.15.3:34646) 2024-05-08T21:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd66f7fe700 1 -- 172.21.15.3:0/151837944 learned_addr learned my addr 172.21.15.3:0/151837944 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd66effd700 1 --2- 172.21.15.3:0/151837944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd658004850 0x7fd670155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd66e7fc700 1 --2- 172.21.15.3:0/151837944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6700cc640 0x7fd6700b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd66f7fe700 1 -- 172.21.15.3:0/151837944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6700cc640 msgr2=0x7fd6700b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd66f7fe700 1 --2- 172.21.15.3:0/151837944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6700cc640 0x7fd6700b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd66f7fe700 1 -- 172.21.15.3:0/151837944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd658004850 msgr2=0x7fd670155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd66f7fe700 1 --2- 172.21.15.3:0/151837944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd658004850 0x7fd670155bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd66f7fe700 1 -- 172.21.15.3:0/151837944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6700b8130 con 0x7fd67006c5c0 2024-05-08T21:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd66f7fe700 1 --2- 172.21.15.3:0/151837944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd67006c5c0 0x7fd67006ca50 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd670072ef0 tx=0x7fd66800d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.858+0000 7fd657fff700 1 -- 172.21.15.3:0/151837944 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66800dbc0 con 0x7fd67006c5c0 2024-05-08T21:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.859+0000 7fd657fff700 1 -- 172.21.15.3:0/151837944 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66800dd40 con 0x7fd67006c5c0 2024-05-08T21:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.859+0000 7fd6759b7700 1 -- 172.21.15.3:0/151837944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6700b8410 con 0x7fd67006c5c0 2024-05-08T21:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.859+0000 7fd657fff700 1 -- 172.21.15.3:0/151837944 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd668003c40 con 0x7fd67006c5c0 2024-05-08T21:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.859+0000 7fd6759b7700 1 -- 172.21.15.3:0/151837944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6700b89d0 con 0x7fd67006c5c0 2024-05-08T21:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.859+0000 7fd6759b7700 1 -- 172.21.15.3:0/151837944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd6700cf9f0 con 0x7fd67006c5c0 2024-05-08T21:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.860+0000 7fd657fff700 1 -- 172.21.15.3:0/151837944 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd6680050d0 con 0x7fd67006c5c0 2024-05-08T21:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.861+0000 7fd657fff700 1 --2- 172.21.15.3:0/151837944 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd65c06acc0 0x7fd65c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.861+0000 7fd657fff700 1 -- 172.21.15.3:0/151837944 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd668089ce0 con 0x7fd67006c5c0 2024-05-08T21:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.861+0000 7fd657fff700 1 --2- 172.21.15.3:0/151837944 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x7fd65c070bc0 0x7fd65c072fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.861+0000 7fd657fff700 1 -- 172.21.15.3:0/151837944 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd65c073690 con 0x7fd65c070bc0 2024-05-08T21:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.861+0000 7fd657fff700 1 -- 172.21.15.3:0/151837944 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd66808a130 con 0x7fd67006c5c0 2024-05-08T21:23:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.861+0000 7fd66effd700 1 --2- 172.21.15.3:0/151837944 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd65c06acc0 0x7fd65c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.861+0000 7fd66e7fc700 1 --2- 172.21.15.3:0/151837944 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x7fd65c070bc0 0x7fd65c072fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.862+0000 7fd66e7fc700 1 --2- 172.21.15.3:0/151837944 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x7fd65c070bc0 0x7fd65c072fe0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.862+0000 7fd66effd700 1 --2- 172.21.15.3:0/151837944 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd65c06acc0 0x7fd65c06d180 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd660009b00 tx=0x7fd660009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.862+0000 7fd657fff700 1 -- 172.21.15.3:0/151837944 <== osd.5 v2:172.21.15.186:6800/47340 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fd65c073690 con 0x7fd65c070bc0 2024-05-08T21:23:45.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.878+0000 7fd6759b7700 1 -- 172.21.15.3:0/151837944 --> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd67006d840 con 0x7fd65c070bc0 2024-05-08T21:23:45.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.879+0000 7fd657fff700 1 -- 172.21.15.3:0/151837944 <== osd.5 v2:172.21.15.186:6800/47340 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd66400cb20 con 0x7fd65c070bc0 2024-05-08T21:23:45.880 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:23:45.880 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T21:23:45.880 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:23:45.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.879+0000 7fd6759b7700 1 -- 172.21.15.3:0/151837944 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x7fd65c070bc0 msgr2=0x7fd65c072fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.879+0000 7fd6759b7700 1 --2- 172.21.15.3:0/151837944 >> [v2:172.21.15.186:6800/47340,v1:172.21.15.186:6801/47340] conn(0x7fd65c070bc0 0x7fd65c072fe0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:45.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.879+0000 7fd6759b7700 1 -- 172.21.15.3:0/151837944 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd65c06acc0 msgr2=0x7fd65c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.879+0000 7fd6759b7700 1 --2- 172.21.15.3:0/151837944 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd65c06acc0 0x7fd65c06d180 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd660009b00 tx=0x7fd660009210 comp rx=0 tx=0).stop 2024-05-08T21:23:45.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.879+0000 7fd6759b7700 1 -- 172.21.15.3:0/151837944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd67006c5c0 msgr2=0x7fd67006ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:45.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.879+0000 7fd6759b7700 1 --2- 172.21.15.3:0/151837944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd67006c5c0 0x7fd67006ca50 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd670072ef0 tx=0x7fd66800d660 comp rx=0 tx=0).stop 2024-05-08T21:23:45.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.879+0000 7fd66f7fe700 1 -- 172.21.15.3:0/151837944 reap_dead start 2024-05-08T21:23:45.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.879+0000 7fd6759b7700 1 -- 172.21.15.3:0/151837944 shutdown_connections 2024-05-08T21:23:45.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.879+0000 7fd6759b7700 1 -- 172.21.15.3:0/151837944 >> 172.21.15.3:0/151837944 conn(0x7fd67008afe0 msgr2=0x7fd67015e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:45.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.879+0000 7fd6759b7700 1 -- 172.21.15.3:0/151837944 shutdown_connections 2024-05-08T21:23:45.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:45.879+0000 7fd6759b7700 1 -- 172.21.15.3:0/151837944 wait complete. 2024-05-08T21:23:45.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-05-08T21:23:46.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.030+0000 7f3b094a3700 1 Processor -- start 2024-05-08T21:23:46.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.030+0000 7f3b094a3700 1 -- start start 2024-05-08T21:23:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.031+0000 7f3b094a3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3b0415dc40 0x7f3b0409a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.031+0000 7f3b094a3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b040d0270 0x7f3b04099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.031+0000 7f3b094a3700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b040d0700 con 0x7f3b04099f90 2024-05-08T21:23:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.031+0000 7f3b094a3700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b0409a440 con 0x7f3b0415dc40 2024-05-08T21:23:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.031+0000 7f3b094a3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b040d09e0 con 0x7f3b040d0270 2024-05-08T21:23:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.031+0000 7f3b02ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3b0415dc40 0x7f3b0409a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.031+0000 7f3b02ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3b0415dc40 0x7f3b0409a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34648/0 (socket says 172.21.15.3:34648) 2024-05-08T21:23:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.031+0000 7f3b02ffd700 1 -- 172.21.15.3:0/625922645 learned_addr learned my addr 172.21.15.3:0/625922645 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.032+0000 7f3b027fc700 1 --2- 172.21.15.3:0/625922645 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b040d0270 0x7f3b04099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.032+0000 7f3b02ffd700 1 -- 172.21.15.3:0/625922645 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b040d0270 msgr2=0x7f3b04099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.032+0000 7f3b02ffd700 1 --2- 172.21.15.3:0/625922645 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b040d0270 0x7f3b04099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.032+0000 7f3b02ffd700 1 -- 172.21.15.3:0/625922645 >> v1:172.21.15.186:6789/0 conn(0x7f3b04099f90 legacy=0x7f3b0406be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.032+0000 7f3b02ffd700 1 -- 172.21.15.3:0/625922645 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b040d0cc0 con 0x7f3b0415dc40 2024-05-08T21:23:46.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.032+0000 7f3b02ffd700 1 --2- 172.21.15.3:0/625922645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b0415dc40 0x7f3b0409a790 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3af40098f0 tx=0x7f3af4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d640e8a325f8086 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:46.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.032+0000 7f3b01ffb700 1 -- 172.21.15.3:0/625922645 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af400b8a0 con 0x7f3b0415dc40 2024-05-08T21:23:46.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.032+0000 7f3b01ffb700 1 -- 172.21.15.3:0/625922645 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af400ba00 con 0x7f3b0415dc40 2024-05-08T21:23:46.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.032+0000 7f3b01ffb700 1 -- 172.21.15.3:0/625922645 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af400bd90 con 0x7f3b0415dc40 2024-05-08T21:23:46.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.033+0000 7f3b094a3700 1 -- 172.21.15.3:0/625922645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b0415dc40 msgr2=0x7f3b0409a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.033+0000 7f3b094a3700 1 --2- 172.21.15.3:0/625922645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b0415dc40 0x7f3b0409a790 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3af40098f0 tx=0x7f3af4009c00 comp rx=0 tx=0).stop 2024-05-08T21:23:46.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.033+0000 7f3b094a3700 1 -- 172.21.15.3:0/625922645 shutdown_connections 2024-05-08T21:23:46.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.033+0000 7f3b094a3700 1 --2- 172.21.15.3:0/625922645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b0415dc40 0x7f3b0409a790 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.033+0000 7f3b094a3700 1 --2- 172.21.15.3:0/625922645 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b040d0270 0x7f3b04099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.033+0000 7f3b094a3700 1 -- 172.21.15.3:0/625922645 >> 172.21.15.3:0/625922645 conn(0x7f3b040b2f20 msgr2=0x7f3b040b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:46.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.033+0000 7f3b094a3700 1 -- 172.21.15.3:0/625922645 shutdown_connections 2024-05-08T21:23:46.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.033+0000 7f3b094a3700 1 -- 172.21.15.3:0/625922645 wait complete. 2024-05-08T21:23:46.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.034+0000 7f3b094a3700 1 Processor -- start 2024-05-08T21:23:46.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.034+0000 7f3b094a3700 1 -- start start 2024-05-08T21:23:46.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.035+0000 7f3b094a3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b04099f90 0x7f3b0414ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.035+0000 7f3b094a3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b0415dc40 0x7f3b0406ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.035+0000 7f3b094a3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b040b6cf0 0x7f3b0406bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.035+0000 7f3b094a3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b04148170 con 0x7f3b0415dc40 2024-05-08T21:23:46.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.035+0000 7f3b094a3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b040b59b0 con 0x7f3b04099f90 2024-05-08T21:23:46.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.035+0000 7f3b094a3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0414b3d0 con 0x7f3b040b6cf0 2024-05-08T21:23:46.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.035+0000 7f3b027fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b0415dc40 0x7f3b0406ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.035+0000 7f3b027fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b0415dc40 0x7f3b0406ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34664/0 (socket says 172.21.15.3:34664) 2024-05-08T21:23:46.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.035+0000 7f3b027fc700 1 -- 172.21.15.3:0/3339670252 learned_addr learned my addr 172.21.15.3:0/3339670252 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:46.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.035+0000 7f3b02ffd700 1 --2- 172.21.15.3:0/3339670252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b04099f90 0x7f3b0414ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.036+0000 7f3b037fe700 1 --2- 172.21.15.3:0/3339670252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b040b6cf0 0x7f3b0406bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.036+0000 7f3b027fc700 1 -- 172.21.15.3:0/3339670252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b040b6cf0 msgr2=0x7f3b0406bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.036+0000 7f3b027fc700 1 --2- 172.21.15.3:0/3339670252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b040b6cf0 0x7f3b0406bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.036+0000 7f3b027fc700 1 -- 172.21.15.3:0/3339670252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b04099f90 msgr2=0x7f3b0414ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.036+0000 7f3b027fc700 1 --2- 172.21.15.3:0/3339670252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b04099f90 0x7f3b0414ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.036+0000 7f3b027fc700 1 -- 172.21.15.3:0/3339670252 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af4009550 con 0x7f3b0415dc40 2024-05-08T21:23:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.036+0000 7f3b037fe700 1 --2- 172.21.15.3:0/3339670252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b040b6cf0 0x7f3b0406bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:23:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.036+0000 7f3b027fc700 1 --2- 172.21.15.3:0/3339670252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b0415dc40 0x7f3b0406ba70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3af800e810 tx=0x7f3af800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.036+0000 7f3aeb7fe700 1 -- 172.21.15.3:0/3339670252 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af8013610 con 0x7f3b0415dc40 2024-05-08T21:23:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.036+0000 7f3aeb7fe700 1 -- 172.21.15.3:0/3339670252 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af8013770 con 0x7f3b0415dc40 2024-05-08T21:23:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.036+0000 7f3b094a3700 1 -- 172.21.15.3:0/3339670252 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b040b5c50 con 0x7f3b0415dc40 2024-05-08T21:23:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.036+0000 7f3aeb7fe700 1 -- 172.21.15.3:0/3339670252 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af8013b20 con 0x7f3b0415dc40 2024-05-08T21:23:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.036+0000 7f3b094a3700 1 -- 172.21.15.3:0/3339670252 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b040b6210 con 0x7f3b0415dc40 2024-05-08T21:23:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.037+0000 7f3b094a3700 1 -- 172.21.15.3:0/3339670252 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b040b6b40 con 0x7f3b0415dc40 2024-05-08T21:23:46.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.038+0000 7f3aeb7fe700 1 -- 172.21.15.3:0/3339670252 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3af8013c80 con 0x7f3b0415dc40 2024-05-08T21:23:46.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.038+0000 7f3aeb7fe700 1 --2- 172.21.15.3:0/3339670252 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3af006ac90 0x7f3af006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.038+0000 7f3aeb7fe700 1 -- 172.21.15.3:0/3339670252 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f3af8088c80 con 0x7f3b0415dc40 2024-05-08T21:23:46.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.038+0000 7f3b02ffd700 1 --2- 172.21.15.3:0/3339670252 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3af006ac90 0x7f3af006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.039+0000 7f3b02ffd700 1 --2- 172.21.15.3:0/3339670252 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3af006ac90 0x7f3af006d150 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3af4014040 tx=0x7f3af4008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:46.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.043+0000 7f3aeb7fe700 1 -- 172.21.15.3:0/3339670252 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f3af8054110 con 0x7f3b0415dc40 2024-05-08T21:23:46.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.271+0000 7f3b094a3700 1 -- 172.21.15.3:0/3339670252 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b040b2930 con 0x7f3af006ac90 2024-05-08T21:23:46.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.272+0000 7f3aeb7fe700 1 -- 172.21.15.3:0/3339670252 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3af400bb70 con 0x7f3af006ac90 2024-05-08T21:23:46.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.274+0000 7f3b094a3700 1 -- 172.21.15.3:0/3339670252 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3af006ac90 msgr2=0x7f3af006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.274+0000 7f3b094a3700 1 --2- 172.21.15.3:0/3339670252 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3af006ac90 0x7f3af006d150 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3af4014040 tx=0x7f3af4008030 comp rx=0 tx=0).stop 2024-05-08T21:23:46.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.274+0000 7f3b094a3700 1 -- 172.21.15.3:0/3339670252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b0415dc40 msgr2=0x7f3b0406ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.274+0000 7f3b094a3700 1 --2- 172.21.15.3:0/3339670252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b0415dc40 0x7f3b0406ba70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3af800e810 tx=0x7f3af800eb20 comp rx=0 tx=0).stop 2024-05-08T21:23:46.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.274+0000 7f3b094a3700 1 -- 172.21.15.3:0/3339670252 shutdown_connections 2024-05-08T21:23:46.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.274+0000 7f3b094a3700 1 --2- 172.21.15.3:0/3339670252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b040b6cf0 0x7f3b0406bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.274+0000 7f3b094a3700 1 --2- 172.21.15.3:0/3339670252 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f3af006ac90 0x7f3af006d150 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.274+0000 7f3b094a3700 1 --2- 172.21.15.3:0/3339670252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b04099f90 0x7f3b0414ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.274+0000 7f3b094a3700 1 --2- 172.21.15.3:0/3339670252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b0415dc40 0x7f3b0406ba70 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.274+0000 7f3b094a3700 1 -- 172.21.15.3:0/3339670252 >> 172.21.15.3:0/3339670252 conn(0x7f3b040b2f20 msgr2=0x7f3b0406f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:46.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.274+0000 7f3b094a3700 1 -- 172.21.15.3:0/3339670252 shutdown_connections 2024-05-08T21:23:46.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.274+0000 7f3b094a3700 1 -- 172.21.15.3:0/3339670252 wait complete. 2024-05-08T21:23:46.277 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 5 to deep-scrub 2024-05-08T21:23:46.289 INFO:tasks.ceph:Scrubbing osd.6 2024-05-08T21:23:46.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-05-08T21:23:46.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.429+0000 7fb49144d700 1 Processor -- start 2024-05-08T21:23:46.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.429+0000 7fb49144d700 1 -- start start 2024-05-08T21:23:46.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.430+0000 7fb49144d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb48c15dae0 0x7fb48c1614f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.430+0000 7fb49144d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c161e80 con 0x7fb48c161a30 2024-05-08T21:23:46.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.430+0000 7fb49144d700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c0b49b0 con 0x7fb48c15d700 2024-05-08T21:23:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.430+0000 7fb49144d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb48c0b3da0 con 0x7fb48c15dae0 2024-05-08T21:23:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.430+0000 7fb48a7fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb48c15dae0 0x7fb48c1614f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.430+0000 7fb48b7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb48c161a30 0x7fb48c0b0140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34224/0 (socket says 172.21.15.3:34224) 2024-05-08T21:23:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.430+0000 7fb48b7fe700 1 -- 172.21.15.3:0/4199211940 learned_addr learned my addr 172.21.15.3:0/4199211940 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.431+0000 7fb48a7fc700 1 -- 172.21.15.3:0/4199211940 >> v1:172.21.15.186:6789/0 conn(0x7fb48c15d700 legacy=0x7fb48c0b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.431+0000 7fb48a7fc700 1 -- 172.21.15.3:0/4199211940 >> v1:172.21.15.3:6789/0 conn(0x7fb48c161a30 legacy=0x7fb48c0b0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.431+0000 7fb489ffb700 1 -- 172.21.15.3:0/4199211940 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024246440 0 0) 0x7fb484001490 con 0x7fb48c161a30 2024-05-08T21:23:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.431+0000 7fb48a7fc700 1 -- 172.21.15.3:0/4199211940 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c0b4080 con 0x7fb48c15dae0 2024-05-08T21:23:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.431+0000 7fb48a7fc700 1 --2- 172.21.15.3:0/4199211940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb48c15dae0 0x7fb48c1614f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fb480009840 tx=0x7fb480009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7adf9e4cf2a6ef8a server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.431+0000 7fb489ffb700 1 -- 172.21.15.3:0/4199211940 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480011830 con 0x7fb48c15dae0 2024-05-08T21:23:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.431+0000 7fb489ffb700 1 -- 172.21.15.3:0/4199211940 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb480011990 con 0x7fb48c15dae0 2024-05-08T21:23:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.431+0000 7fb489ffb700 1 -- 172.21.15.3:0/4199211940 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480011d20 con 0x7fb48c15dae0 2024-05-08T21:23:46.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.432+0000 7fb49144d700 1 -- 172.21.15.3:0/4199211940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb48c15dae0 msgr2=0x7fb48c1614f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.432+0000 7fb49144d700 1 --2- 172.21.15.3:0/4199211940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb48c15dae0 0x7fb48c1614f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fb480009840 tx=0x7fb480009b50 comp rx=0 tx=0).stop 2024-05-08T21:23:46.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.432+0000 7fb49144d700 1 -- 172.21.15.3:0/4199211940 shutdown_connections 2024-05-08T21:23:46.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.432+0000 7fb49144d700 1 --2- 172.21.15.3:0/4199211940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb48c15dae0 0x7fb48c1614f0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.432+0000 7fb49144d700 1 -- 172.21.15.3:0/4199211940 >> 172.21.15.3:0/4199211940 conn(0x7fb48c08afe0 msgr2=0x7fb48c0d3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:46.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.432+0000 7fb49144d700 1 -- 172.21.15.3:0/4199211940 shutdown_connections 2024-05-08T21:23:46.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.432+0000 7fb49144d700 1 -- 172.21.15.3:0/4199211940 wait complete. 2024-05-08T21:23:46.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.434+0000 7fb49144d700 1 Processor -- start 2024-05-08T21:23:46.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.434+0000 7fb49144d700 1 -- start start 2024-05-08T21:23:46.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb49144d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb48c15d700 0x7fb48c0c7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb49144d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb48c15dae0 0x7fb48c1f4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb49144d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb48c161a30 0x7fb48c1f6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb49144d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c0c8150 con 0x7fb48c15d700 2024-05-08T21:23:46.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb49144d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c1f5180 con 0x7fb48c161a30 2024-05-08T21:23:46.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb49144d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c0cb880 con 0x7fb48c15dae0 2024-05-08T21:23:46.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb48affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb48c15d700 0x7fb48c0c7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb48affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb48c15d700 0x7fb48c0c7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34682/0 (socket says 172.21.15.3:34682) 2024-05-08T21:23:46.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb48affd700 1 -- 172.21.15.3:0/3863850044 learned_addr learned my addr 172.21.15.3:0/3863850044 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:46.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb48a7fc700 1 --2- 172.21.15.3:0/3863850044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb48c15dae0 0x7fb48c1f4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb48b7fe700 1 --2- 172.21.15.3:0/3863850044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb48c161a30 0x7fb48c1f6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb48affd700 1 -- 172.21.15.3:0/3863850044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb48c15dae0 msgr2=0x7fb48c1f4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb48affd700 1 --2- 172.21.15.3:0/3863850044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb48c15dae0 0x7fb48c1f4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb48affd700 1 -- 172.21.15.3:0/3863850044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb48c161a30 msgr2=0x7fb48c1f6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb48affd700 1 --2- 172.21.15.3:0/3863850044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb48c161a30 0x7fb48c1f6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb48affd700 1 -- 172.21.15.3:0/3863850044 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4800094a0 con 0x7fb48c15d700 2024-05-08T21:23:46.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.435+0000 7fb48affd700 1 --2- 172.21.15.3:0/3863850044 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb48c15d700 0x7fb48c0c7c10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fb47c00c520 tx=0x7fb47c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:46.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.436+0000 7fb4737fe700 1 -- 172.21.15.3:0/3863850044 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47c0052d0 con 0x7fb48c15d700 2024-05-08T21:23:46.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.436+0000 7fb4737fe700 1 -- 172.21.15.3:0/3863850044 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb47c005430 con 0x7fb48c15d700 2024-05-08T21:23:46.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.436+0000 7fb49144d700 1 -- 172.21.15.3:0/3863850044 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48c0c8340 con 0x7fb48c15d700 2024-05-08T21:23:46.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.436+0000 7fb4737fe700 1 -- 172.21.15.3:0/3863850044 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47c0038f0 con 0x7fb48c15d700 2024-05-08T21:23:46.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.436+0000 7fb49144d700 1 -- 172.21.15.3:0/3863850044 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48c0c86d0 con 0x7fb48c15d700 2024-05-08T21:23:46.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.437+0000 7fb49144d700 1 -- 172.21.15.3:0/3863850044 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb48c159390 con 0x7fb48c15d700 2024-05-08T21:23:46.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.437+0000 7fb4737fe700 1 -- 172.21.15.3:0/3863850044 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb47c003a50 con 0x7fb48c15d700 2024-05-08T21:23:46.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.438+0000 7fb4737fe700 1 --2- 172.21.15.3:0/3863850044 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb47806ac90 0x7fb47806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.438+0000 7fb4737fe700 1 -- 172.21.15.3:0/3863850044 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb47c0884e0 con 0x7fb48c15d700 2024-05-08T21:23:46.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.438+0000 7fb4737fe700 1 --2- 172.21.15.3:0/3863850044 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x7fb478070b90 0x7fb478072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.438+0000 7fb48a7fc700 1 --2- 172.21.15.3:0/3863850044 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb47806ac90 0x7fb47806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.438+0000 7fb4737fe700 1 -- 172.21.15.3:0/3863850044 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb478073660 con 0x7fb478070b90 2024-05-08T21:23:46.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.438+0000 7fb4737fe700 1 -- 172.21.15.3:0/3863850044 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb47c08aba0 con 0x7fb48c15d700 2024-05-08T21:23:46.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.438+0000 7fb48b7fe700 1 --2- 172.21.15.3:0/3863850044 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x7fb478070b90 0x7fb478072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.438+0000 7fb48a7fc700 1 --2- 172.21.15.3:0/3863850044 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb47806ac90 0x7fb47806d150 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb4800090b0 tx=0x7fb48001c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:46.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.438+0000 7fb48b7fe700 1 --2- 172.21.15.3:0/3863850044 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x7fb478070b90 0x7fb478072fb0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:46.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.439+0000 7fb4737fe700 1 -- 172.21.15.3:0/3863850044 <== osd.6 v2:172.21.15.186:6808/47351 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fb478073660 con 0x7fb478070b90 2024-05-08T21:23:46.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.455+0000 7fb49144d700 1 -- 172.21.15.3:0/3863850044 --> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb48c0b3a80 con 0x7fb478070b90 2024-05-08T21:23:46.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.456+0000 7fb4737fe700 1 -- 172.21.15.3:0/3863850044 <== osd.6 v2:172.21.15.186:6808/47351 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb4840071c0 con 0x7fb478070b90 2024-05-08T21:23:46.458 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:23:46.458 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T21:23:46.458 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:23:46.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.456+0000 7fb49144d700 1 -- 172.21.15.3:0/3863850044 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x7fb478070b90 msgr2=0x7fb478072fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.456+0000 7fb49144d700 1 --2- 172.21.15.3:0/3863850044 >> [v2:172.21.15.186:6808/47351,v1:172.21.15.186:6811/47351] conn(0x7fb478070b90 0x7fb478072fb0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.456+0000 7fb49144d700 1 -- 172.21.15.3:0/3863850044 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb47806ac90 msgr2=0x7fb47806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.456+0000 7fb49144d700 1 --2- 172.21.15.3:0/3863850044 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fb47806ac90 0x7fb47806d150 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb4800090b0 tx=0x7fb48001c040 comp rx=0 tx=0).stop 2024-05-08T21:23:46.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.457+0000 7fb49144d700 1 -- 172.21.15.3:0/3863850044 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb48c15d700 msgr2=0x7fb48c0c7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.457+0000 7fb49144d700 1 --2- 172.21.15.3:0/3863850044 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb48c15d700 0x7fb48c0c7c10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fb47c00c520 tx=0x7fb47c00c830 comp rx=0 tx=0).stop 2024-05-08T21:23:46.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.457+0000 7fb48b7fe700 1 -- 172.21.15.3:0/3863850044 reap_dead start 2024-05-08T21:23:46.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.457+0000 7fb49144d700 1 -- 172.21.15.3:0/3863850044 shutdown_connections 2024-05-08T21:23:46.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.457+0000 7fb49144d700 1 -- 172.21.15.3:0/3863850044 >> 172.21.15.3:0/3863850044 conn(0x7fb48c08afe0 msgr2=0x7fb48c0d15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:46.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.457+0000 7fb49144d700 1 -- 172.21.15.3:0/3863850044 shutdown_connections 2024-05-08T21:23:46.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.457+0000 7fb49144d700 1 -- 172.21.15.3:0/3863850044 wait complete. 2024-05-08T21:23:46.472 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-05-08T21:23:46.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.607+0000 7f4f9bb36700 1 Processor -- start 2024-05-08T21:23:46.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.607+0000 7f4f9bb36700 1 -- start start 2024-05-08T21:23:46.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.608+0000 7f4f9bb36700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4f9415dc40 0x7f4f9409a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.608+0000 7f4f9bb36700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4f940d0270 0x7f4f94099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.608+0000 7f4f9bb36700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f9409a3e0 con 0x7f4f94099f90 2024-05-08T21:23:46.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.608+0000 7f4f9bb36700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f940d0700 con 0x7f4f940d0270 2024-05-08T21:23:46.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.608+0000 7f4f9bb36700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f940d09e0 con 0x7f4f9415dc40 2024-05-08T21:23:46.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.608+0000 7f4f990d1700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4f940d0270 0x7f4f94099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.608+0000 7f4f9a0d3700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f4f94099f90 0x7f4f9406be30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34228/0 (socket says 172.21.15.3:34228) 2024-05-08T21:23:46.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.609+0000 7f4f9a0d3700 1 -- 172.21.15.3:0/3201500905 learned_addr learned my addr 172.21.15.3:0/3201500905 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:46.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.609+0000 7f4f998d2700 1 --2- 172.21.15.3:0/3201500905 >> v2:172.21.15.186:3300/0 conn(0x7f4f9415dc40 0x7f4f9409a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.609+0000 7f4f990d1700 1 -- 172.21.15.3:0/3201500905 >> v2:172.21.15.186:3300/0 conn(0x7f4f9415dc40 msgr2=0x7f4f9409a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:46.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.609+0000 7f4f990d1700 1 --2- 172.21.15.3:0/3201500905 >> v2:172.21.15.186:3300/0 conn(0x7f4f9415dc40 0x7f4f9409a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.609+0000 7f4f988d0700 1 -- 172.21.15.3:0/3201500905 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573325976 0 0) 0x7f4f8c001490 con 0x7f4f94099f90 2024-05-08T21:23:46.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.609+0000 7f4f990d1700 1 -- 172.21.15.3:0/3201500905 >> v1:172.21.15.3:6789/0 conn(0x7f4f94099f90 legacy=0x7f4f9406be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.609+0000 7f4f990d1700 1 -- 172.21.15.3:0/3201500905 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f940d0cc0 con 0x7f4f940d0270 2024-05-08T21:23:46.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.609+0000 7f4f990d1700 1 --2- 172.21.15.3:0/3201500905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f940d0270 0x7f4f94099a50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4f94072250 tx=0x7f4f90009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df3fc4c42a2d676 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:46.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.609+0000 7f4f988d0700 1 -- 172.21.15.3:0/3201500905 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f90011830 con 0x7f4f940d0270 2024-05-08T21:23:46.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.609+0000 7f4f988d0700 1 -- 172.21.15.3:0/3201500905 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f90011990 con 0x7f4f940d0270 2024-05-08T21:23:46.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.609+0000 7f4f988d0700 1 -- 172.21.15.3:0/3201500905 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f90011d20 con 0x7f4f940d0270 2024-05-08T21:23:46.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.610+0000 7f4f9bb36700 1 -- 172.21.15.3:0/3201500905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f940d0270 msgr2=0x7f4f94099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.610+0000 7f4f9bb36700 1 --2- 172.21.15.3:0/3201500905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f940d0270 0x7f4f94099a50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4f94072250 tx=0x7f4f90009b20 comp rx=0 tx=0).stop 2024-05-08T21:23:46.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.610+0000 7f4f9bb36700 1 -- 172.21.15.3:0/3201500905 shutdown_connections 2024-05-08T21:23:46.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.610+0000 7f4f9bb36700 1 --2- 172.21.15.3:0/3201500905 >> v2:172.21.15.186:3300/0 conn(0x7f4f9415dc40 0x7f4f9409a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.610+0000 7f4f9bb36700 1 --2- 172.21.15.3:0/3201500905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f940d0270 0x7f4f94099a50 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.610+0000 7f4f9bb36700 1 -- 172.21.15.3:0/3201500905 >> 172.21.15.3:0/3201500905 conn(0x7f4f940b2f20 msgr2=0x7f4f940b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:46.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.610+0000 7f4f9bb36700 1 -- 172.21.15.3:0/3201500905 shutdown_connections 2024-05-08T21:23:46.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.610+0000 7f4f9bb36700 1 -- 172.21.15.3:0/3201500905 wait complete. 2024-05-08T21:23:46.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.611+0000 7f4f9bb36700 1 Processor -- start 2024-05-08T21:23:46.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.611+0000 7f4f9bb36700 1 -- start start 2024-05-08T21:23:46.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.612+0000 7f4f9bb36700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f94099f90 0x7f4f94070000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.612+0000 7f4f9bb36700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f9415dc40 0x7f4f94157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.612+0000 7f4f9bb36700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f94157a70 0x7f4f941414c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.612+0000 7f4f9bb36700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f941570a0 con 0x7f4f9415dc40 2024-05-08T21:23:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.612+0000 7f4f9bb36700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f941567e0 con 0x7f4f94099f90 2024-05-08T21:23:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.612+0000 7f4f9bb36700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f94156a50 con 0x7f4f94157a70 2024-05-08T21:23:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.612+0000 7f4f990d1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f9415dc40 0x7f4f94157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.612+0000 7f4f990d1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f9415dc40 0x7f4f94157530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34696/0 (socket says 172.21.15.3:34696) 2024-05-08T21:23:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.612+0000 7f4f990d1700 1 -- 172.21.15.3:0/2736242580 learned_addr learned my addr 172.21.15.3:0/2736242580 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.613+0000 7f4f998d2700 1 --2- 172.21.15.3:0/2736242580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f94099f90 0x7f4f94070000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.613+0000 7f4f9a0d3700 1 --2- 172.21.15.3:0/2736242580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f94157a70 0x7f4f941414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.613+0000 7f4f990d1700 1 -- 172.21.15.3:0/2736242580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f94157a70 msgr2=0x7f4f941414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.613+0000 7f4f990d1700 1 --2- 172.21.15.3:0/2736242580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f94157a70 0x7f4f941414c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.613+0000 7f4f990d1700 1 -- 172.21.15.3:0/2736242580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f94099f90 msgr2=0x7f4f94070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.613+0000 7f4f990d1700 1 --2- 172.21.15.3:0/2736242580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f94099f90 0x7f4f94070000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.613+0000 7f4f990d1700 1 -- 172.21.15.3:0/2736242580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f900094a0 con 0x7f4f9415dc40 2024-05-08T21:23:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.613+0000 7f4f990d1700 1 --2- 172.21.15.3:0/2736242580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f9415dc40 0x7f4f94157530 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f4f940726b0 tx=0x7f4f90004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.613+0000 7f4f867fc700 1 -- 172.21.15.3:0/2736242580 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f900036a0 con 0x7f4f9415dc40 2024-05-08T21:23:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.613+0000 7f4f867fc700 1 -- 172.21.15.3:0/2736242580 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9001f070 con 0x7f4f9415dc40 2024-05-08T21:23:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.613+0000 7f4f9bb36700 1 -- 172.21.15.3:0/2736242580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f94156d30 con 0x7f4f9415dc40 2024-05-08T21:23:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.613+0000 7f4f867fc700 1 -- 172.21.15.3:0/2736242580 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f90003a60 con 0x7f4f9415dc40 2024-05-08T21:23:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.613+0000 7f4f9bb36700 1 -- 172.21.15.3:0/2736242580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f94070540 con 0x7f4f9415dc40 2024-05-08T21:23:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.614+0000 7f4f9bb36700 1 -- 172.21.15.3:0/2736242580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f94157250 con 0x7f4f9415dc40 2024-05-08T21:23:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.615+0000 7f4f867fc700 1 -- 172.21.15.3:0/2736242580 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4f90003bc0 con 0x7f4f9415dc40 2024-05-08T21:23:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.615+0000 7f4f867fc700 1 --2- 172.21.15.3:0/2736242580 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4f7806ac90 0x7f4f7806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.615+0000 7f4f867fc700 1 -- 172.21.15.3:0/2736242580 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f4f90021070 con 0x7f4f9415dc40 2024-05-08T21:23:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.615+0000 7f4f998d2700 1 --2- 172.21.15.3:0/2736242580 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4f7806ac90 0x7f4f7806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.616+0000 7f4f998d2700 1 --2- 172.21.15.3:0/2736242580 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4f7806ac90 0x7f4f7806d150 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4f88006b20 tx=0x7f4f88008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.620+0000 7f4f867fc700 1 -- 172.21.15.3:0/2736242580 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f4f94157250 con 0x7f4f9415dc40 2024-05-08T21:23:46.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.835+0000 7f4f9bb36700 1 -- 172.21.15.3:0/2736242580 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f940b2930 con 0x7f4f7806ac90 2024-05-08T21:23:46.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.836+0000 7f4f867fc700 1 -- 172.21.15.3:0/2736242580 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4f880076b0 con 0x7f4f7806ac90 2024-05-08T21:23:46.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.838+0000 7f4f9bb36700 1 -- 172.21.15.3:0/2736242580 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4f7806ac90 msgr2=0x7f4f7806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.838+0000 7f4f9bb36700 1 --2- 172.21.15.3:0/2736242580 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4f7806ac90 0x7f4f7806d150 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4f88006b20 tx=0x7f4f88008040 comp rx=0 tx=0).stop 2024-05-08T21:23:46.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.838+0000 7f4f9bb36700 1 -- 172.21.15.3:0/2736242580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f9415dc40 msgr2=0x7f4f94157530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.838+0000 7f4f9bb36700 1 --2- 172.21.15.3:0/2736242580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f9415dc40 0x7f4f94157530 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f4f940726b0 tx=0x7f4f90004750 comp rx=0 tx=0).stop 2024-05-08T21:23:46.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.838+0000 7f4f9bb36700 1 -- 172.21.15.3:0/2736242580 shutdown_connections 2024-05-08T21:23:46.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.838+0000 7f4f9bb36700 1 --2- 172.21.15.3:0/2736242580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f94157a70 0x7f4f941414c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.838+0000 7f4f9bb36700 1 --2- 172.21.15.3:0/2736242580 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f4f7806ac90 0x7f4f7806d150 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.838+0000 7f4f9bb36700 1 --2- 172.21.15.3:0/2736242580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f94099f90 0x7f4f94070000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.838+0000 7f4f9bb36700 1 --2- 172.21.15.3:0/2736242580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f9415dc40 0x7f4f94157530 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.838+0000 7f4f9bb36700 1 -- 172.21.15.3:0/2736242580 >> 172.21.15.3:0/2736242580 conn(0x7f4f940b2f20 msgr2=0x7f4f9415b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:46.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.838+0000 7f4f9bb36700 1 -- 172.21.15.3:0/2736242580 shutdown_connections 2024-05-08T21:23:46.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.838+0000 7f4f9bb36700 1 -- 172.21.15.3:0/2736242580 wait complete. 2024-05-08T21:23:46.842 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 6 to deep-scrub 2024-05-08T21:23:46.854 INFO:tasks.ceph:Scrubbing osd.7 2024-05-08T21:23:46.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-05-08T21:23:46.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.993+0000 7efde5162700 1 Processor -- start 2024-05-08T21:23:46.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.993+0000 7efde5162700 1 -- start start 2024-05-08T21:23:46.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.994+0000 7efde5162700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00cff30 0x7efde00d0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:46.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.994+0000 7efde5162700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde00d0850 con 0x7efde00b4a90 2024-05-08T21:23:46.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.994+0000 7efde5162700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde00cf610 con 0x7efde00d3d20 2024-05-08T21:23:46.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.994+0000 7efde5162700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde00d1fc0 con 0x7efde00cff30 2024-05-08T21:23:46.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.994+0000 7efddf59e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7efde00b4a90 0x7efde00ce970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34238/0 (socket says 172.21.15.3:34238) 2024-05-08T21:23:46.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.995+0000 7efddf59e700 1 -- 172.21.15.3:0/3550036334 learned_addr learned my addr 172.21.15.3:0/3550036334 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:23:46.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.995+0000 7efdded9d700 1 --2- 172.21.15.3:0/3550036334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00cff30 0x7efde00d0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:46.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.995+0000 7efdddd9b700 1 -- 172.21.15.3:0/3550036334 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121248717 0 0) 0x7efde00d0850 con 0x7efde00b4a90 2024-05-08T21:23:46.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.995+0000 7efdddd9b700 1 -- 172.21.15.3:0/3550036334 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc80035b0 con 0x7efde00b4a90 2024-05-08T21:23:46.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.995+0000 7efdddd9b700 1 -- 172.21.15.3:0/3550036334 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797485708 0 0) 0x7efde00cf610 con 0x7efde00d3d20 2024-05-08T21:23:46.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.995+0000 7efdddd9b700 1 -- 172.21.15.3:0/3550036334 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efde00d0850 con 0x7efde00d3d20 2024-05-08T21:23:46.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.995+0000 7efdddd9b700 1 -- 172.21.15.3:0/3550036334 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3218621921 0 0) 0x7efdc80035b0 con 0x7efde00b4a90 2024-05-08T21:23:46.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.995+0000 7efdded9d700 1 -- 172.21.15.3:0/3550036334 >> v1:172.21.15.186:6789/0 conn(0x7efde00d3d20 legacy=0x7efde00ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.995+0000 7efdded9d700 1 -- 172.21.15.3:0/3550036334 >> v1:172.21.15.3:6789/0 conn(0x7efde00b4a90 legacy=0x7efde00ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.995+0000 7efdded9d700 1 -- 172.21.15.3:0/3550036334 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde00d22a0 con 0x7efde00cff30 2024-05-08T21:23:46.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.995+0000 7efdded9d700 1 --2- 172.21.15.3:0/3550036334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00cff30 0x7efde00d0310 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009870 tx=0x7efdd0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68dcebb85c7cfefe server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:46.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.995+0000 7efdddd9b700 1 -- 172.21.15.3:0/3550036334 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd80019f0 con 0x7efde00cff30 2024-05-08T21:23:46.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.995+0000 7efdddd9b700 1 -- 172.21.15.3:0/3550036334 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd0004030 con 0x7efde00cff30 2024-05-08T21:23:46.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.996+0000 7efdddd9b700 1 -- 172.21.15.3:0/3550036334 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd00043c0 con 0x7efde00cff30 2024-05-08T21:23:46.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.996+0000 7efde5162700 1 -- 172.21.15.3:0/3550036334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00cff30 msgr2=0x7efde00d0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:46.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.996+0000 7efde5162700 1 --2- 172.21.15.3:0/3550036334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00cff30 0x7efde00d0310 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009870 tx=0x7efdd0009b80 comp rx=0 tx=0).stop 2024-05-08T21:23:46.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.996+0000 7efde5162700 1 -- 172.21.15.3:0/3550036334 shutdown_connections 2024-05-08T21:23:46.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.996+0000 7efde5162700 1 --2- 172.21.15.3:0/3550036334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00cff30 0x7efde00d0310 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:46.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.996+0000 7efde5162700 1 -- 172.21.15.3:0/3550036334 >> 172.21.15.3:0/3550036334 conn(0x7efde008afe0 msgr2=0x7efde00d3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:46.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.996+0000 7efde5162700 1 -- 172.21.15.3:0/3550036334 shutdown_connections 2024-05-08T21:23:46.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.996+0000 7efde5162700 1 -- 172.21.15.3:0/3550036334 wait complete. 2024-05-08T21:23:47.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.998+0000 7efde5162700 1 Processor -- start 2024-05-08T21:23:47.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.998+0000 7efde5162700 1 -- start start 2024-05-08T21:23:47.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efde5162700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efde00b4a90 0x7efde00b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efde5162700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00cff30 0x7efde01f4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efde5162700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde00d3d20 0x7efde015bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efde5162700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde00b6c70 con 0x7efde00b4a90 2024-05-08T21:23:47.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efde5162700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde01f4dd0 con 0x7efde00d3d20 2024-05-08T21:23:47.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efde5162700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde015c100 con 0x7efde00cff30 2024-05-08T21:23:47.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efdded9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efde00b4a90 0x7efde00b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efdded9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efde00b4a90 0x7efde00b7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34700/0 (socket says 172.21.15.3:34700) 2024-05-08T21:23:47.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efdded9d700 1 -- 172.21.15.3:0/3258946584 learned_addr learned my addr 172.21.15.3:0/3258946584 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:47.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efddf59e700 1 --2- 172.21.15.3:0/3258946584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde00d3d20 0x7efde015bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efdde59c700 1 --2- 172.21.15.3:0/3258946584 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00cff30 0x7efde01f4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efdded9d700 1 -- 172.21.15.3:0/3258946584 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00cff30 msgr2=0x7efde01f4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:47.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efdded9d700 1 --2- 172.21.15.3:0/3258946584 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00cff30 0x7efde01f4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efdded9d700 1 -- 172.21.15.3:0/3258946584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde00d3d20 msgr2=0x7efde015bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:47.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efdded9d700 1 --2- 172.21.15.3:0/3258946584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde00d3d20 0x7efde015bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efdded9d700 1 -- 172.21.15.3:0/3258946584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd00094a0 con 0x7efde00b4a90 2024-05-08T21:23:47.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efdded9d700 1 --2- 172.21.15.3:0/3258946584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efde00b4a90 0x7efde00b7df0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7efdd0000c00 tx=0x7efdd0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:47.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:46.999+0000 7efdc77fe700 1 -- 172.21.15.3:0/3258946584 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd0003e60 con 0x7efde00b4a90 2024-05-08T21:23:47.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.000+0000 7efdc77fe700 1 -- 172.21.15.3:0/3258946584 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd001f070 con 0x7efde00b4a90 2024-05-08T21:23:47.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.000+0000 7efde5162700 1 -- 172.21.15.3:0/3258946584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde00b62d0 con 0x7efde00b4a90 2024-05-08T21:23:47.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.000+0000 7efdc77fe700 1 -- 172.21.15.3:0/3258946584 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd0011850 con 0x7efde00b4a90 2024-05-08T21:23:47.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.000+0000 7efde5162700 1 -- 172.21.15.3:0/3258946584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde00b6890 con 0x7efde00b4a90 2024-05-08T21:23:47.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.000+0000 7efde5162700 1 -- 172.21.15.3:0/3258946584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efde00cf8f0 con 0x7efde00b4a90 2024-05-08T21:23:47.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.001+0000 7efdc77fe700 1 -- 172.21.15.3:0/3258946584 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efdd00119b0 con 0x7efde00b4a90 2024-05-08T21:23:47.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.002+0000 7efdc77fe700 1 --2- 172.21.15.3:0/3258946584 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efdcc06ac90 0x7efdcc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.002+0000 7efdc77fe700 1 -- 172.21.15.3:0/3258946584 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7efdd0021070 con 0x7efde00b4a90 2024-05-08T21:23:47.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.002+0000 7efdc77fe700 1 --2- 172.21.15.3:0/3258946584 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x7efdcc070b90 0x7efdcc072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.002+0000 7efdc77fe700 1 -- 172.21.15.3:0/3258946584 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efdcc073660 con 0x7efdcc070b90 2024-05-08T21:23:47.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.002+0000 7efdc77fe700 1 -- 172.21.15.3:0/3258946584 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdd008cd40 con 0x7efde00b4a90 2024-05-08T21:23:47.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.002+0000 7efdde59c700 1 --2- 172.21.15.3:0/3258946584 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efdcc06ac90 0x7efdcc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.002+0000 7efddf59e700 1 --2- 172.21.15.3:0/3258946584 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x7efdcc070b90 0x7efdcc072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.003+0000 7efdde59c700 1 --2- 172.21.15.3:0/3258946584 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efdcc06ac90 0x7efdcc06d150 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efdd40011e0 tx=0x7efdd4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:47.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.003+0000 7efddf59e700 1 --2- 172.21.15.3:0/3258946584 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x7efdcc070b90 0x7efdcc072fb0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:47.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.003+0000 7efdc77fe700 1 -- 172.21.15.3:0/3258946584 <== osd.7 v2:172.21.15.186:6809/47341 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7efdcc073660 con 0x7efdcc070b90 2024-05-08T21:23:47.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.019+0000 7efde5162700 1 -- 172.21.15.3:0/3258946584 --> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7efde00b2740 con 0x7efdcc070b90 2024-05-08T21:23:47.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.019+0000 7efdc77fe700 1 -- 172.21.15.3:0/3258946584 <== osd.7 v2:172.21.15.186:6809/47341 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7efdd8007240 con 0x7efdcc070b90 2024-05-08T21:23:47.021 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-05-08T21:23:47.021 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T21:23:47.021 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-05-08T21:23:47.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.020+0000 7efde5162700 1 -- 172.21.15.3:0/3258946584 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x7efdcc070b90 msgr2=0x7efdcc072fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:47.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.020+0000 7efde5162700 1 --2- 172.21.15.3:0/3258946584 >> [v2:172.21.15.186:6809/47341,v1:172.21.15.186:6812/47341] conn(0x7efdcc070b90 0x7efdcc072fb0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.020+0000 7efde5162700 1 -- 172.21.15.3:0/3258946584 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efdcc06ac90 msgr2=0x7efdcc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:47.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.020+0000 7efde5162700 1 --2- 172.21.15.3:0/3258946584 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7efdcc06ac90 0x7efdcc06d150 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efdd40011e0 tx=0x7efdd4009040 comp rx=0 tx=0).stop 2024-05-08T21:23:47.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.020+0000 7efde5162700 1 -- 172.21.15.3:0/3258946584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efde00b4a90 msgr2=0x7efde00b7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:47.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.020+0000 7efde5162700 1 --2- 172.21.15.3:0/3258946584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efde00b4a90 0x7efde00b7df0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7efdd0000c00 tx=0x7efdd0004750 comp rx=0 tx=0).stop 2024-05-08T21:23:47.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.020+0000 7efddf59e700 1 -- 172.21.15.3:0/3258946584 reap_dead start 2024-05-08T21:23:47.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.020+0000 7efde5162700 1 -- 172.21.15.3:0/3258946584 shutdown_connections 2024-05-08T21:23:47.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.020+0000 7efde5162700 1 -- 172.21.15.3:0/3258946584 >> 172.21.15.3:0/3258946584 conn(0x7efde008afe0 msgr2=0x7efde00b01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:47.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.020+0000 7efde5162700 1 -- 172.21.15.3:0/3258946584 shutdown_connections 2024-05-08T21:23:47.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.020+0000 7efde5162700 1 -- 172.21.15.3:0/3258946584 wait complete. 2024-05-08T21:23:47.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-05-08T21:23:47.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.170+0000 7fa516a3e700 1 Processor -- start 2024-05-08T21:23:47.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.170+0000 7fa516a3e700 1 -- start start 2024-05-08T21:23:47.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.171+0000 7fa516a3e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa5101614f0 0x7fa510099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.171+0000 7fa516a3e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa51015d0a0 0x7fa51015d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.171+0000 7fa516a3e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51009a030 con 0x7fa5100d0650 2024-05-08T21:23:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.171+0000 7fa516a3e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa51009a310 con 0x7fa51015d0a0 2024-05-08T21:23:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.171+0000 7fa516a3e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa51009a5f0 con 0x7fa5101614f0 2024-05-08T21:23:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.171+0000 7fa514fdb700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa51015d0a0 0x7fa51015d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.171+0000 7fa514fdb700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa51015d0a0 0x7fa51015d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34714/0 (socket says 172.21.15.3:34714) 2024-05-08T21:23:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.171+0000 7fa514fdb700 1 -- 172.21.15.3:0/3231586900 learned_addr learned my addr 172.21.15.3:0/3231586900 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.172+0000 7fa50ffff700 1 --2- 172.21.15.3:0/3231586900 >> v2:172.21.15.186:3300/0 conn(0x7fa5101614f0 0x7fa510099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.172+0000 7fa514fdb700 1 -- 172.21.15.3:0/3231586900 >> v2:172.21.15.186:3300/0 conn(0x7fa5101614f0 msgr2=0x7fa510099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.172+0000 7fa514fdb700 1 --2- 172.21.15.3:0/3231586900 >> v2:172.21.15.186:3300/0 conn(0x7fa5101614f0 0x7fa510099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.172+0000 7fa514fdb700 1 -- 172.21.15.3:0/3231586900 >> v1:172.21.15.186:6789/0 conn(0x7fa5100d0650 legacy=0x7fa5101618d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.172+0000 7fa514fdb700 1 -- 172.21.15.3:0/3231586900 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51009a8d0 con 0x7fa51015d0a0 2024-05-08T21:23:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.172+0000 7fa514fdb700 1 --2- 172.21.15.3:0/3231586900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa51015d0a0 0x7fa51015d530 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa510067d10 tx=0x7fa50800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd56650470ad5034 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:47.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.172+0000 7fa50effd700 1 -- 172.21.15.3:0/3231586900 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50800e070 con 0x7fa51015d0a0 2024-05-08T21:23:47.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.172+0000 7fa50effd700 1 -- 172.21.15.3:0/3231586900 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa508003a20 con 0x7fa51015d0a0 2024-05-08T21:23:47.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.172+0000 7fa50effd700 1 -- 172.21.15.3:0/3231586900 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa508003dd0 con 0x7fa51015d0a0 2024-05-08T21:23:47.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.173+0000 7fa516a3e700 1 -- 172.21.15.3:0/3231586900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa51015d0a0 msgr2=0x7fa51015d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:47.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.173+0000 7fa516a3e700 1 --2- 172.21.15.3:0/3231586900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa51015d0a0 0x7fa51015d530 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa510067d10 tx=0x7fa50800b330 comp rx=0 tx=0).stop 2024-05-08T21:23:47.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.173+0000 7fa516a3e700 1 -- 172.21.15.3:0/3231586900 shutdown_connections 2024-05-08T21:23:47.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.173+0000 7fa516a3e700 1 --2- 172.21.15.3:0/3231586900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa51015d0a0 0x7fa51015d530 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.173+0000 7fa516a3e700 1 --2- 172.21.15.3:0/3231586900 >> v2:172.21.15.186:3300/0 conn(0x7fa5101614f0 0x7fa510099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.173+0000 7fa516a3e700 1 -- 172.21.15.3:0/3231586900 >> 172.21.15.3:0/3231586900 conn(0x7fa5100b2f20 msgr2=0x7fa5100b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:47.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.173+0000 7fa516a3e700 1 -- 172.21.15.3:0/3231586900 shutdown_connections 2024-05-08T21:23:47.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.173+0000 7fa516a3e700 1 -- 172.21.15.3:0/3231586900 wait complete. 2024-05-08T21:23:47.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.174+0000 7fa516a3e700 1 Processor -- start 2024-05-08T21:23:47.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.174+0000 7fa516a3e700 1 -- start start 2024-05-08T21:23:47.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.175+0000 7fa516a3e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5100d0650 0x7fa5100b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.175+0000 7fa516a3e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa51015d0a0 0x7fa5100b6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.175+0000 7fa516a3e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5101614f0 0x7fa510158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.175+0000 7fa516a3e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5100c95f0 con 0x7fa51015d0a0 2024-05-08T21:23:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.175+0000 7fa516a3e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5100c98d0 con 0x7fa5100d0650 2024-05-08T21:23:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.175+0000 7fa516a3e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5100c9bb0 con 0x7fa5101614f0 2024-05-08T21:23:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.175+0000 7fa50f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa51015d0a0 0x7fa5100b6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.175+0000 7fa50f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa51015d0a0 0x7fa5100b6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34716/0 (socket says 172.21.15.3:34716) 2024-05-08T21:23:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.175+0000 7fa50f7fe700 1 -- 172.21.15.3:0/3095960164 learned_addr learned my addr 172.21.15.3:0/3095960164 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.175+0000 7fa50ffff700 1 --2- 172.21.15.3:0/3095960164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5100d0650 0x7fa5100b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.175+0000 7fa514fdb700 1 --2- 172.21.15.3:0/3095960164 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5101614f0 0x7fa510158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.176+0000 7fa50f7fe700 1 -- 172.21.15.3:0/3095960164 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5101614f0 msgr2=0x7fa510158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.176+0000 7fa50f7fe700 1 --2- 172.21.15.3:0/3095960164 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5101614f0 0x7fa510158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.176+0000 7fa50f7fe700 1 -- 172.21.15.3:0/3095960164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5100d0650 msgr2=0x7fa5100b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.176+0000 7fa50f7fe700 1 --2- 172.21.15.3:0/3095960164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5100d0650 0x7fa5100b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.176+0000 7fa50f7fe700 1 -- 172.21.15.3:0/3095960164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa508009b30 con 0x7fa51015d0a0 2024-05-08T21:23:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.176+0000 7fa50f7fe700 1 --2- 172.21.15.3:0/3095960164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa51015d0a0 0x7fa5100b6e80 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa4f800c4d0 tx=0x7fa4f800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.176+0000 7fa50cff9700 1 -- 172.21.15.3:0/3095960164 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8011790 con 0x7fa51015d0a0 2024-05-08T21:23:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.176+0000 7fa50cff9700 1 -- 172.21.15.3:0/3095960164 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f80118f0 con 0x7fa51015d0a0 2024-05-08T21:23:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.176+0000 7fa516a3e700 1 -- 172.21.15.3:0/3095960164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5100c9e90 con 0x7fa51015d0a0 2024-05-08T21:23:47.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.176+0000 7fa50cff9700 1 -- 172.21.15.3:0/3095960164 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8011ca0 con 0x7fa51015d0a0 2024-05-08T21:23:47.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.176+0000 7fa516a3e700 1 -- 172.21.15.3:0/3095960164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5100b4c80 con 0x7fa51015d0a0 2024-05-08T21:23:47.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.177+0000 7fa516a3e700 1 -- 172.21.15.3:0/3095960164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5100ca0d0 con 0x7fa51015d0a0 2024-05-08T21:23:47.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.178+0000 7fa50cff9700 1 -- 172.21.15.3:0/3095960164 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa4f801c460 con 0x7fa51015d0a0 2024-05-08T21:23:47.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.178+0000 7fa50cff9700 1 --2- 172.21.15.3:0/3095960164 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa4f006ac90 0x7fa4f006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.178+0000 7fa50cff9700 1 -- 172.21.15.3:0/3095960164 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fa4f8088cf0 con 0x7fa51015d0a0 2024-05-08T21:23:47.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.179+0000 7fa50ffff700 1 --2- 172.21.15.3:0/3095960164 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa4f006ac90 0x7fa4f006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.179+0000 7fa50ffff700 1 --2- 172.21.15.3:0/3095960164 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa4f006ac90 0x7fa4f006d150 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa5100cf020 tx=0x7fa500008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:47.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.183+0000 7fa50cff9700 1 -- 172.21.15.3:0/3095960164 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fa5100ca0d0 con 0x7fa51015d0a0 2024-05-08T21:23:47.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.401+0000 7fa516a3e700 1 -- 172.21.15.3:0/3095960164 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5100cfa00 con 0x7fa4f006ac90 2024-05-08T21:23:47.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.401+0000 7fa50cff9700 1 -- 172.21.15.3:0/3095960164 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa5000076a0 con 0x7fa4f006ac90 2024-05-08T21:23:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.403+0000 7fa516a3e700 1 -- 172.21.15.3:0/3095960164 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa4f006ac90 msgr2=0x7fa4f006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.403+0000 7fa516a3e700 1 --2- 172.21.15.3:0/3095960164 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa4f006ac90 0x7fa4f006d150 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa5100cf020 tx=0x7fa500008040 comp rx=0 tx=0).stop 2024-05-08T21:23:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.403+0000 7fa516a3e700 1 -- 172.21.15.3:0/3095960164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa51015d0a0 msgr2=0x7fa5100b6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.403+0000 7fa516a3e700 1 --2- 172.21.15.3:0/3095960164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa51015d0a0 0x7fa5100b6e80 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa4f800c4d0 tx=0x7fa4f800c7e0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.404+0000 7fa516a3e700 1 -- 172.21.15.3:0/3095960164 shutdown_connections 2024-05-08T21:23:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.404+0000 7fa516a3e700 1 --2- 172.21.15.3:0/3095960164 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5101614f0 0x7fa510158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.404+0000 7fa516a3e700 1 --2- 172.21.15.3:0/3095960164 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fa4f006ac90 0x7fa4f006d150 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.404+0000 7fa516a3e700 1 --2- 172.21.15.3:0/3095960164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5100d0650 0x7fa5100b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.404+0000 7fa516a3e700 1 --2- 172.21.15.3:0/3095960164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa51015d0a0 0x7fa5100b6e80 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.404+0000 7fa516a3e700 1 -- 172.21.15.3:0/3095960164 >> 172.21.15.3:0/3095960164 conn(0x7fa5100b2f20 msgr2=0x7fa5101618d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.404+0000 7fa516a3e700 1 -- 172.21.15.3:0/3095960164 shutdown_connections 2024-05-08T21:23:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.404+0000 7fa516a3e700 1 -- 172.21.15.3:0/3095960164 wait complete. 2024-05-08T21:23:47.407 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 7 to deep-scrub 2024-05-08T21:23:47.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T21:23:47.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.555+0000 7f5d26425700 1 Processor -- start 2024-05-08T21:23:47.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.555+0000 7f5d26425700 1 -- start start 2024-05-08T21:23:47.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.556+0000 7f5d26425700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d2015da10 0x7f5d200d0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.556+0000 7f5d26425700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5d2006bd80 0x7f5d2009a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.556+0000 7f5d26425700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5d2006c3d0 0x7f5d2006c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.556+0000 7f5d26425700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d2006cdc0 con 0x7f5d2006bd80 2024-05-08T21:23:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.556+0000 7f5d26425700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d200cf990 con 0x7f5d2006c3d0 2024-05-08T21:23:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.556+0000 7f5d26425700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d200d04b0 con 0x7f5d2015da10 2024-05-08T21:23:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.556+0000 7f5d1f7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5d2006bd80 0x7f5d2009a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.557+0000 7f5d1f7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5d2006bd80 0x7f5d2009a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34724/0 (socket says 172.21.15.3:34724) 2024-05-08T21:23:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.557+0000 7f5d1f7fe700 1 -- 172.21.15.3:0/2340035418 learned_addr learned my addr 172.21.15.3:0/2340035418 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.557+0000 7f5d249c2700 1 --2- 172.21.15.3:0/2340035418 >> v2:172.21.15.186:3300/0 conn(0x7f5d2006c3d0 0x7f5d2006c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.557+0000 7f5d1ffff700 1 --2- 172.21.15.3:0/2340035418 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d2015da10 0x7f5d200d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.557+0000 7f5d1f7fe700 1 -- 172.21.15.3:0/2340035418 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d2015da10 msgr2=0x7f5d200d0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.557+0000 7f5d1f7fe700 1 --2- 172.21.15.3:0/2340035418 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d2015da10 0x7f5d200d0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.557+0000 7f5d1f7fe700 1 -- 172.21.15.3:0/2340035418 >> v2:172.21.15.186:3300/0 conn(0x7f5d2006c3d0 msgr2=0x7f5d2006c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:23:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.557+0000 7f5d1f7fe700 1 --2- 172.21.15.3:0/2340035418 >> v2:172.21.15.186:3300/0 conn(0x7f5d2006c3d0 0x7f5d2006c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.557+0000 7f5d1f7fe700 1 -- 172.21.15.3:0/2340035418 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2009ad60 con 0x7f5d2006bd80 2024-05-08T21:23:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.557+0000 7f5d1f7fe700 1 --2- 172.21.15.3:0/2340035418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d2006bd80 0x7f5d2009a430 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f5d200cb940 tx=0x7f5d08009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d282b061ba1335a server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.557+0000 7f5d1effd700 1 -- 172.21.15.3:0/2340035418 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0800b870 con 0x7f5d2006bd80 2024-05-08T21:23:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.557+0000 7f5d1effd700 1 -- 172.21.15.3:0/2340035418 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0800b9d0 con 0x7f5d2006bd80 2024-05-08T21:23:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.557+0000 7f5d1effd700 1 -- 172.21.15.3:0/2340035418 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0800bd60 con 0x7f5d2006bd80 2024-05-08T21:23:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.558+0000 7f5d26425700 1 -- 172.21.15.3:0/2340035418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d2006bd80 msgr2=0x7f5d2009a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.558+0000 7f5d26425700 1 --2- 172.21.15.3:0/2340035418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d2006bd80 0x7f5d2009a430 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f5d200cb940 tx=0x7f5d08009bd0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.558+0000 7f5d26425700 1 -- 172.21.15.3:0/2340035418 shutdown_connections 2024-05-08T21:23:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.558+0000 7f5d26425700 1 --2- 172.21.15.3:0/2340035418 >> v2:172.21.15.186:3300/0 conn(0x7f5d2006c3d0 0x7f5d2006c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.558+0000 7f5d26425700 1 --2- 172.21.15.3:0/2340035418 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d2015da10 0x7f5d200d0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.558+0000 7f5d26425700 1 --2- 172.21.15.3:0/2340035418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d2006bd80 0x7f5d2009a430 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.558+0000 7f5d26425700 1 -- 172.21.15.3:0/2340035418 >> 172.21.15.3:0/2340035418 conn(0x7f5d200b3010 msgr2=0x7f5d200b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.558+0000 7f5d26425700 1 -- 172.21.15.3:0/2340035418 shutdown_connections 2024-05-08T21:23:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.558+0000 7f5d26425700 1 -- 172.21.15.3:0/2340035418 wait complete. 2024-05-08T21:23:47.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.559+0000 7f5d26425700 1 Processor -- start 2024-05-08T21:23:47.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.559+0000 7f5d26425700 1 -- start start 2024-05-08T21:23:47.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.560+0000 7f5d26425700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d2006bd80 0x7f5d2014e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.560+0000 7f5d26425700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2006c3d0 0x7f5d20070300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.560+0000 7f5d26425700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d2015da10 0x7f5d201f6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.560+0000 7f5d26425700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2014e600 con 0x7f5d2006bd80 2024-05-08T21:23:47.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.560+0000 7f5d26425700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2014d670 con 0x7f5d2006c3d0 2024-05-08T21:23:47.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.560+0000 7f5d26425700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d201f73c0 con 0x7f5d2015da10 2024-05-08T21:23:47.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.560+0000 7f5d1ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d2006bd80 0x7f5d2014e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.560+0000 7f5d1ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d2006bd80 0x7f5d2014e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34740/0 (socket says 172.21.15.3:34740) 2024-05-08T21:23:47.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.560+0000 7f5d1ffff700 1 -- 172.21.15.3:0/3600131787 learned_addr learned my addr 172.21.15.3:0/3600131787 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:23:47.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.560+0000 7f5d1f7fe700 1 --2- 172.21.15.3:0/3600131787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2006c3d0 0x7f5d20070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.560+0000 7f5d249c2700 1 --2- 172.21.15.3:0/3600131787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d2015da10 0x7f5d201f6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.561+0000 7f5d1ffff700 1 -- 172.21.15.3:0/3600131787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d2015da10 msgr2=0x7f5d201f6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:47.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.561+0000 7f5d1ffff700 1 --2- 172.21.15.3:0/3600131787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d2015da10 0x7f5d201f6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.561+0000 7f5d1ffff700 1 -- 172.21.15.3:0/3600131787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2006c3d0 msgr2=0x7f5d20070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:47.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.561+0000 7f5d1ffff700 1 --2- 172.21.15.3:0/3600131787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2006c3d0 0x7f5d20070300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.561+0000 7f5d1ffff700 1 -- 172.21.15.3:0/3600131787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d08009550 con 0x7f5d2006bd80 2024-05-08T21:23:47.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.561+0000 7f5d1ffff700 1 --2- 172.21.15.3:0/3600131787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d2006bd80 0x7f5d2014e0c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f5d1000e810 tx=0x7f5d1000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:47.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.561+0000 7f5d1cff9700 1 -- 172.21.15.3:0/3600131787 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1000c070 con 0x7f5d2006bd80 2024-05-08T21:23:47.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.561+0000 7f5d1cff9700 1 -- 172.21.15.3:0/3600131787 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d10013610 con 0x7f5d2006bd80 2024-05-08T21:23:47.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.561+0000 7f5d26425700 1 -- 172.21.15.3:0/3600131787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d201f76a0 con 0x7f5d2006bd80 2024-05-08T21:23:47.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.561+0000 7f5d1cff9700 1 -- 172.21.15.3:0/3600131787 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d100139c0 con 0x7f5d2006bd80 2024-05-08T21:23:47.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.561+0000 7f5d26425700 1 -- 172.21.15.3:0/3600131787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2014dba0 con 0x7f5d2006bd80 2024-05-08T21:23:47.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.562+0000 7f5d26425700 1 -- 172.21.15.3:0/3600131787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2014df10 con 0x7f5d2006bd80 2024-05-08T21:23:47.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.562+0000 7f5d1cff9700 1 -- 172.21.15.3:0/3600131787 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5d10013b20 con 0x7f5d2006bd80 2024-05-08T21:23:47.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.563+0000 7f5d1cff9700 1 --2- 172.21.15.3:0/3600131787 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f5d0006ac90 0x7f5d0006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:23:47.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.563+0000 7f5d1cff9700 1 -- 172.21.15.3:0/3600131787 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f5d10088dc0 con 0x7f5d2006bd80 2024-05-08T21:23:47.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.563+0000 7f5d1f7fe700 1 --2- 172.21.15.3:0/3600131787 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f5d0006ac90 0x7f5d0006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:23:47.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.564+0000 7f5d1f7fe700 1 --2- 172.21.15.3:0/3600131787 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f5d0006ac90 0x7f5d0006d150 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5d2006c1c0 tx=0x7f5d08008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:23:47.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.568+0000 7f5d1cff9700 1 -- 172.21.15.3:0/3600131787 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f5d10054250 con 0x7f5d2006bd80 2024-05-08T21:23:47.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.780+0000 7f5d26425700 1 -- 172.21.15.3:0/3600131787 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5d200d0140 con 0x7f5d0006ac90 2024-05-08T21:23:47.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.783+0000 7f5d1cff9700 1 -- 172.21.15.3:0/3600131787 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50908 (secure 0 0 0) 0x7f5d0800bb40 con 0x7f5d0006ac90 2024-05-08T21:23:47.785 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:23:47.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.786+0000 7f5d26425700 1 -- 172.21.15.3:0/3600131787 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f5d0006ac90 msgr2=0x7f5d0006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:47.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.786+0000 7f5d26425700 1 --2- 172.21.15.3:0/3600131787 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f5d0006ac90 0x7f5d0006d150 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5d2006c1c0 tx=0x7f5d08008030 comp rx=0 tx=0).stop 2024-05-08T21:23:47.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.786+0000 7f5d26425700 1 -- 172.21.15.3:0/3600131787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d2006bd80 msgr2=0x7f5d2014e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:23:47.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.786+0000 7f5d26425700 1 --2- 172.21.15.3:0/3600131787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d2006bd80 0x7f5d2014e0c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f5d1000e810 tx=0x7f5d1000eb20 comp rx=0 tx=0).stop 2024-05-08T21:23:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.786+0000 7f5d26425700 1 -- 172.21.15.3:0/3600131787 shutdown_connections 2024-05-08T21:23:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.786+0000 7f5d26425700 1 --2- 172.21.15.3:0/3600131787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d2015da10 0x7f5d201f6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.786+0000 7f5d26425700 1 --2- 172.21.15.3:0/3600131787 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f5d0006ac90 0x7f5d0006d150 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.786+0000 7f5d26425700 1 --2- 172.21.15.3:0/3600131787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d2006bd80 0x7f5d2014e0c0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.786+0000 7f5d26425700 1 --2- 172.21.15.3:0/3600131787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2006c3d0 0x7f5d20070300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:23:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.786+0000 7f5d26425700 1 -- 172.21.15.3:0/3600131787 >> 172.21.15.3:0/3600131787 conn(0x7f5d200b3010 msgr2=0x7f5d200b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:23:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.786+0000 7f5d26425700 1 -- 172.21.15.3:0/3600131787 shutdown_connections 2024-05-08T21:23:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:23:47.786+0000 7f5d26425700 1 -- 172.21.15.3:0/3600131787 wait complete. 2024-05-08T21:23:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-08T21:23:47.801 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-05-08T21:23:46.558559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":221596,"kb_used_data":5416,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753124,"statfs":{"total":773094113280,"available":772867198976,"internally_reserved":0,"allocated":5545984,"data_stored":2999128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6291456,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":-16,"num_read_kb":-9,"num_write":-8,"num_write_kb":-6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001844"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":188,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-08T21:23:37.330912+0000","last_change":"2024-05-08T21:21:42.333097+0000","last_active":"2024-05-08T21:23:37.330912+0000","last_peered":"2024-05-08T21:23:37.330912+0000","last_clean":"2024-05-08T21:23:37.330912+0000","last_became_active":"2024-05-08T21:21:42.332033+0000","last_became_peered":"2024-05-08T21:21:42.332033+0000","last_unstale":"2024-05-08T21:23:37.330912+0000","last_undegraded":"2024-05-08T21:23:37.330912+0000","last_fullsized":"2024-05-08T21:23:37.330912+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T21:21:40.862622+0000","last_clean_scrub_stamp":"2024-05-08T21:21:40.862622+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:39:24.772551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28132,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343708,"statfs":{"total":96636764160,"available":96607956992,"internally_reserved":0,"allocated":1138688,"data_stored":817667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.52100000000000002,"15min":0.51500000000000001},"min":{"1min":0.32800000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.48699999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.49199999999999999}]},{"osd":1,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.39500000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.441},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55200000000000005,"15min":0.55100000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.55200000000000005}]},{"osd":2,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.58399999999999996,"15min":0.57499999999999996},"min":{"1min":0.51000000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.59399999999999997}]},{"osd":3,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.38300000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.45200000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.52900000000000003}]},{"osd":4,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.52600000000000002,"15min":0.53300000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42399999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.65700000000000003}]},{"osd":5,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.502,"15min":0.503},"min":{"1min":0.36299999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.748,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.51900000000000002,"15min":0.52300000000000002},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71299999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.505}]},{"osd":6,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57599999999999996,"15min":0.57799999999999996},"min":{"1min":0.38700000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78200000000000003,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.62}]}]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27552,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":544768,"data_stored":227299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.47699999999999998,"15min":0.46200000000000002},"min":{"1min":0.38,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.57899999999999996}]},{"osd":1,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.51100000000000001,"15min":0.505},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.622},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53500000000000003,"15min":0.53000000000000003},"min":{"1min":0.39200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.498}]},{"osd":2,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.35799999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.435,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.53900000000000003}]},{"osd":3,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.54000000000000004,"15min":0.53500000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.46000000000000002,"5min":0.376,"15min":0.376},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.65400000000000003}]},{"osd":4,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.496,"15min":0.49199999999999999},"min":{"1min":0.35199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.48999999999999999,"15min":0.48399999999999999},"min":{"1min":0.39400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.63800000000000001}]},{"osd":5,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.47999999999999998,"15min":0.47199999999999998},"min":{"1min":0.41199999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.40400000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.69499999999999995}]},{"osd":7,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.76700000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.443,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.68899999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27552,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":544768,"data_stored":227299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.375,"15min":0.375},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.56999999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.42599999999999999,"15min":0.42499999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.85399999999999998,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.41499999999999998}]},{"osd":2,"last update":"Wed May 8 21:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.39400000000000002,"15min":0.39500000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.621,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.308},{"interface":"front","average":{"1min":0.44,"5min":0.46600000000000003,"15min":0.46999999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.622,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.434}]},{"osd":3,"last update":"Wed May 8 21:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.46800000000000003,"15min":0.47299999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.59199999999999997,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.435,"15min":0.42899999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.36199999999999999}]},{"osd":4,"last update":"Wed May 8 21:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.64200000000000002,"15min":0.64400000000000002},"min":{"1min":0.46200000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.64600000000000002}]},{"osd":5,"last update":"Wed May 8 21:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.82799999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.63900000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.63600000000000001}]},{"osd":6,"last update":"Wed May 8 21:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.66800000000000004,"15min":0.67200000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.88600000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.60299999999999998}]},{"osd":7,"last update":"Wed May 8 21:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.70299999999999996,"15min":0.70399999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65100000000000002,"15min":0.64500000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.624}]}]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28136,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343704,"statfs":{"total":96636764160,"available":96607952896,"internally_reserved":0,"allocated":1138688,"data_stored":817667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.49099999999999999,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.378,"15min":0.376},"min":{"1min":0.29599999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.32500000000000001}]},{"osd":2,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.39500000000000002,"15min":0.39300000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.40300000000000002,"15min":0.40100000000000002},"min":{"1min":0.313,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.59899999999999998,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.379}]},{"osd":3,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.40500000000000003,"15min":0.41099999999999998},"min":{"1min":0.26100000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.57099999999999995,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68400000000000005,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.38800000000000001}]},{"osd":4,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.49199999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58699999999999997,"15min":0.58499999999999996},"min":{"1min":0.47399999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.59699999999999998}]},{"osd":5,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60499999999999998,"15min":0.60699999999999998},"min":{"1min":0.46600000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.64100000000000001,"15min":0.64200000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.82799999999999996,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.55600000000000005}]},{"osd":6,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66800000000000004,"15min":0.67200000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92000000000000004,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65100000000000002,"15min":0.64900000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.76400000000000001}]},{"osd":7,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.5,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66200000000000003,"15min":0.66400000000000003},"min":{"1min":0.52000000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.89700000000000002,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.56499999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27560,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344280,"statfs":{"total":96636764160,"available":96608542720,"internally_reserved":0,"allocated":544768,"data_stored":227299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.41299999999999998,"15min":0.41599999999999998},"min":{"1min":0.26900000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.55700000000000005,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.49099999999999999,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63,"5min":1.083,"15min":1.083},"last":0.40500000000000003}]},{"osd":1,"last update":"Wed May 8 21:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.434,"15min":0.438},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.61899999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.442,"15min":0.45000000000000001},"min":{"1min":0.30299999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.58299999999999996,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.44}]},{"osd":3,"last update":"Wed May 8 21:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.433,"15min":0.439},"min":{"1min":0.29999999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57999999999999996,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.5,"15min":0.50600000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.61499999999999999,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.52600000000000002}]},{"osd":4,"last update":"Wed May 8 21:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.65100000000000002,"15min":0.65700000000000003},"min":{"1min":0.46700000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79700000000000004,"5min":1.141,"15min":1.141},"last":0.503},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.628},"min":{"1min":0.42899999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84299999999999997,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.47499999999999998}]},{"osd":5,"last update":"Wed May 8 21:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.68600000000000005,"15min":0.68799999999999994},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.89200000000000002,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.63},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.70499999999999996,"15min":0.71599999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.82399999999999995,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.58299999999999996}]},{"osd":6,"last update":"Wed May 8 21:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.91700000000000004,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69699999999999995,"15min":0.70399999999999996},"min":{"1min":0.53200000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.81000000000000005,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.59699999999999998}]},{"osd":7,"last update":"Wed May 8 21:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.72299999999999998,"15min":0.73199999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.85899999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.71799999999999997,"15min":0.72299999999999998},"min":{"1min":0.54400000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.94899999999999995,"5min":1.177,"15min":1.177},"last":0.56299999999999994}]}]},{"osd":3,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27556,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344284,"statfs":{"total":96636764160,"available":96608546816,"internally_reserved":0,"allocated":544768,"data_stored":227299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.42899999999999999,"15min":0.435},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.59799999999999998,"5min":1.492,"15min":1.492},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.50600000000000001,"15min":0.52100000000000002},"min":{"1min":0.30099999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.68700000000000006,"5min":2.7919999999999998,"15min":2.7919999999999998},"last":0.27000000000000002}]},{"osd":1,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.46000000000000002,"15min":0.434},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.2370000000000001,"5min":4.2370000000000001,"15min":4.2370000000000001},"last":0.315},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.436,"15min":0.441},"min":{"1min":0.31,"5min":0.308,"15min":0.308},"max":{"1min":0.54500000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.40200000000000002}]},{"osd":2,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.53400000000000003,"15min":0.55200000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.65000000000000002,"5min":2.9369999999999998,"15min":2.9369999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.496,"15min":0.46999999999999997},"min":{"1min":0.35299999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.2969999999999997,"5min":4.2969999999999997,"15min":4.2969999999999997},"last":0.38700000000000001}]},{"osd":4,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.623,"15min":0.622},"min":{"1min":0.48299999999999998,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.90300000000000002,"5min":1.173,"15min":1.173},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.45000000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.7490000000000001,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":0.47999999999999998}]},{"osd":5,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.76000000000000001,"15min":0.78600000000000003},"min":{"1min":0.47399999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.79900000000000004,"5min":2.9060000000000001,"15min":2.9060000000000001},"last":0.498},{"interface":"front","average":{"1min":0.874,"5min":0.70399999999999996,"15min":0.67600000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":5.1699999999999999,"5min":5.1699999999999999,"15min":5.1699999999999999},"last":0.55900000000000005}]},{"osd":6,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.68000000000000005,"15min":0.68400000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.76500000000000001,"15min":0.78900000000000003},"min":{"1min":0.50600000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.81799999999999995,"5min":2.8730000000000002,"15min":2.8730000000000002},"last":0.45600000000000002}]},{"osd":7,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.74299999999999999,"15min":0.71999999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":5.2030000000000003,"5min":5.2030000000000003,"15min":5.2030000000000003},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.70099999999999996,"15min":0.70699999999999996},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.1399999999999999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.59699999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27556,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344284,"statfs":{"total":96636764160,"available":96608546816,"internally_reserved":0,"allocated":544768,"data_stored":227299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.52700000000000002,"15min":0.52000000000000002},"min":{"1min":0.40400000000000003,"5min":0.312,"15min":0.312},"max":{"1min":0.748,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.39700000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.68500000000000005,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.59899999999999998}]},{"osd":1,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35199999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76000000000000001,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52300000000000002,"15min":0.51700000000000002},"min":{"1min":0.38200000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.57299999999999995}]},{"osd":2,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58999999999999997,"15min":0.58899999999999997},"min":{"1min":0.47399999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.747,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.71999999999999997,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.65900000000000003}]},{"osd":3,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.36699999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.497}]},{"osd":5,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81299999999999994,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.625},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.51000000000000001,"15min":0.505},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.74299999999999999,"5min":0.754,"15min":0.754},"last":0.51500000000000001}]},{"osd":6,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.38600000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.62,"15min":0.622},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85399999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.68999999999999995}]},{"osd":7,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79600000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54100000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27552,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":544768,"data_stored":227299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.74399999999999999,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.83299999999999996,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.439}]},{"osd":1,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56000000000000005,"15min":0.56200000000000006},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.78700000000000003,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.53000000000000003,"15min":0.52400000000000002},"min":{"1min":0.41599999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94599999999999995,"5min":1.147,"15min":1.147},"last":0.65500000000000003}]},{"osd":2,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.47999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.90100000000000002,"5min":1.282,"15min":1.282},"last":0.73999999999999999}]},{"osd":3,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.96699999999999997,"5min":1.202,"15min":1.202},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.441,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.71599999999999997,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.61699999999999999}]},{"osd":4,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.502,"15min":0.499},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73199999999999998,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.372,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68700000000000006,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68700000000000006}]},{"osd":6,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53500000000000003,"15min":0.53100000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.75600000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39800000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.77700000000000002,"5min":1.119,"15min":1.119},"last":0.77700000000000002}]},{"osd":7,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53200000000000003,"15min":0.53000000000000003},"min":{"1min":0.31900000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.79900000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.39300000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T21:23:47.805 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-05-08T21:21:40.862622+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=21, tm_min=21, tm_sec=40, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=21, tm_min=23, tm_sec=41, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T21:23:47.805 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-08T21:24:07.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T21:24:07.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.950+0000 7f6fbf2cf700 1 Processor -- start 2024-05-08T21:24:07.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.951+0000 7f6fbf2cf700 1 -- start start 2024-05-08T21:24:07.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.951+0000 7f6fbf2cf700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6fb815e830 0x7f6fb80ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:07.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.951+0000 7f6fbf2cf700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6fb806b0e0 0x7f6fb80cf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:07.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.951+0000 7f6fbf2cf700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb80d0a90 con 0x7f6fb80d0f20 2024-05-08T21:24:07.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.951+0000 7f6fbf2cf700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb806b510 con 0x7f6fb806b0e0 2024-05-08T21:24:07.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.951+0000 7f6fbf2cf700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb806b720 con 0x7f6fb815e830 2024-05-08T21:24:07.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.952+0000 7f6fbd86c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6fb806b0e0 0x7f6fb80cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:07.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.952+0000 7f6fbc86a700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6fb80d0f20 0x7f6fb806a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53094/0 (socket says 172.21.15.3:53094) 2024-05-08T21:24:07.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.952+0000 7f6fbc86a700 1 -- 172.21.15.3:0/2234199169 learned_addr learned my addr 172.21.15.3:0/2234199169 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:24:07.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.952+0000 7f6fbd06b700 1 --2- 172.21.15.3:0/2234199169 >> v2:172.21.15.186:3300/0 conn(0x7f6fb815e830 0x7f6fb80ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:07.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.952+0000 7f6fbd86c700 1 -- 172.21.15.3:0/2234199169 >> v2:172.21.15.186:3300/0 conn(0x7f6fb815e830 msgr2=0x7f6fb80ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:24:07.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.952+0000 7f6fbd86c700 1 --2- 172.21.15.3:0/2234199169 >> v2:172.21.15.186:3300/0 conn(0x7f6fb815e830 0x7f6fb80ceb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:07.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.952+0000 7f6fbd86c700 1 -- 172.21.15.3:0/2234199169 >> v1:172.21.15.3:6789/0 conn(0x7f6fb80d0f20 legacy=0x7f6fb806a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:07.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.952+0000 7f6fbd86c700 1 -- 172.21.15.3:0/2234199169 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb80d0440 con 0x7f6fb806b0e0 2024-05-08T21:24:07.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.952+0000 7f6faffff700 1 -- 172.21.15.3:0/2234199169 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255393268 0 0) 0x7f6fb80d0a90 con 0x7f6fb80d0f20 2024-05-08T21:24:07.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.952+0000 7f6fbd86c700 1 --2- 172.21.15.3:0/2234199169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fb806b0e0 0x7f6fb80cf090 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80d13d0 tx=0x7f6fb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2850e2610b39bfb1 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:24:07.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.952+0000 7f6faffff700 1 -- 172.21.15.3:0/2234199169 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb000e070 con 0x7f6fb806b0e0 2024-05-08T21:24:07.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.952+0000 7f6faffff700 1 -- 172.21.15.3:0/2234199169 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb000bbd0 con 0x7f6fb806b0e0 2024-05-08T21:24:07.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.952+0000 7f6faffff700 1 -- 172.21.15.3:0/2234199169 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0012430 con 0x7f6fb806b0e0 2024-05-08T21:24:07.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.953+0000 7f6fbf2cf700 1 -- 172.21.15.3:0/2234199169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fb806b0e0 msgr2=0x7f6fb80cf090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:07.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.953+0000 7f6fbf2cf700 1 --2- 172.21.15.3:0/2234199169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fb806b0e0 0x7f6fb80cf090 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80d13d0 tx=0x7f6fb000b330 comp rx=0 tx=0).stop 2024-05-08T21:24:07.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.953+0000 7f6fbf2cf700 1 -- 172.21.15.3:0/2234199169 shutdown_connections 2024-05-08T21:24:07.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.953+0000 7f6fbf2cf700 1 --2- 172.21.15.3:0/2234199169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fb806b0e0 0x7f6fb80cf090 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:07.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.953+0000 7f6fbf2cf700 1 --2- 172.21.15.3:0/2234199169 >> v2:172.21.15.186:3300/0 conn(0x7f6fb815e830 0x7f6fb80ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:07.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.953+0000 7f6fbf2cf700 1 -- 172.21.15.3:0/2234199169 >> 172.21.15.3:0/2234199169 conn(0x7f6fb80b5810 msgr2=0x7f6fb80b5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:24:07.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.953+0000 7f6fbf2cf700 1 -- 172.21.15.3:0/2234199169 shutdown_connections 2024-05-08T21:24:07.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.953+0000 7f6fbf2cf700 1 -- 172.21.15.3:0/2234199169 wait complete. 2024-05-08T21:24:07.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.954+0000 7f6fbf2cf700 1 Processor -- start 2024-05-08T21:24:07.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.955+0000 7f6fbf2cf700 1 -- start start 2024-05-08T21:24:07.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.955+0000 7f6fbf2cf700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fb806b0e0 0x7f6fb8148a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:07.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.955+0000 7f6fbf2cf700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fb80d0f20 0x7f6fb81f72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:07.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.955+0000 7f6fbf2cf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fb815e830 0x7f6fb806e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:07.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.955+0000 7f6fbf2cf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb81f6e80 con 0x7f6fb815e830 2024-05-08T21:24:07.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.955+0000 7f6fbf2cf700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80c9cd0 con 0x7f6fb80d0f20 2024-05-08T21:24:07.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.955+0000 7f6fbf2cf700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb81f77e0 con 0x7f6fb806b0e0 2024-05-08T21:24:07.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.955+0000 7f6fbd86c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fb815e830 0x7f6fb806e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:07.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.955+0000 7f6fbd86c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fb815e830 0x7f6fb806e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57862/0 (socket says 172.21.15.3:57862) 2024-05-08T21:24:07.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.955+0000 7f6fbd86c700 1 -- 172.21.15.3:0/776628837 learned_addr learned my addr 172.21.15.3:0/776628837 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:24:07.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.956+0000 7f6fbc86a700 1 --2- 172.21.15.3:0/776628837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fb80d0f20 0x7f6fb81f72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:07.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.956+0000 7f6fbd06b700 1 --2- 172.21.15.3:0/776628837 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fb806b0e0 0x7f6fb8148a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:07.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.956+0000 7f6fbd86c700 1 -- 172.21.15.3:0/776628837 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fb806b0e0 msgr2=0x7f6fb8148a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:07.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.956+0000 7f6fbd86c700 1 --2- 172.21.15.3:0/776628837 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fb806b0e0 0x7f6fb8148a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:07.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.956+0000 7f6fbd86c700 1 -- 172.21.15.3:0/776628837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fb80d0f20 msgr2=0x7f6fb81f72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:07.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.956+0000 7f6fbd86c700 1 --2- 172.21.15.3:0/776628837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fb80d0f20 0x7f6fb81f72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:07.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.956+0000 7f6fbd86c700 1 -- 172.21.15.3:0/776628837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb0009a70 con 0x7f6fb815e830 2024-05-08T21:24:07.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.956+0000 7f6fbd86c700 1 --2- 172.21.15.3:0/776628837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fb815e830 0x7f6fb806e780 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80676a0 tx=0x7f6fb0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:24:07.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.956+0000 7f6fadffb700 1 -- 172.21.15.3:0/776628837 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0012430 con 0x7f6fb815e830 2024-05-08T21:24:07.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.956+0000 7f6fadffb700 1 -- 172.21.15.3:0/776628837 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb0020070 con 0x7f6fb815e830 2024-05-08T21:24:07.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.956+0000 7f6fbf2cf700 1 -- 172.21.15.3:0/776628837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb80c9710 con 0x7f6fb815e830 2024-05-08T21:24:07.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.956+0000 7f6fadffb700 1 -- 172.21.15.3:0/776628837 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0004280 con 0x7f6fb815e830 2024-05-08T21:24:07.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.956+0000 7f6fbf2cf700 1 -- 172.21.15.3:0/776628837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb81f6aa0 con 0x7f6fb815e830 2024-05-08T21:24:07.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.957+0000 7f6fbf2cf700 1 -- 172.21.15.3:0/776628837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb81f6900 con 0x7f6fb815e830 2024-05-08T21:24:07.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.958+0000 7f6fadffb700 1 -- 172.21.15.3:0/776628837 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6fb00043e0 con 0x7f6fb815e830 2024-05-08T21:24:07.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.958+0000 7f6fadffb700 1 --2- 172.21.15.3:0/776628837 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6f9c06ac90 0x7f6f9c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:07.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.958+0000 7f6fadffb700 1 -- 172.21.15.3:0/776628837 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f6fb0022070 con 0x7f6fb815e830 2024-05-08T21:24:07.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.959+0000 7f6fbd06b700 1 --2- 172.21.15.3:0/776628837 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6f9c06ac90 0x7f6f9c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:07.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.959+0000 7f6fbd06b700 1 --2- 172.21.15.3:0/776628837 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6f9c06ac90 0x7f6f9c06d150 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4006b20 tx=0x7f6fb4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:24:07.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:07.963+0000 7f6fadffb700 1 -- 172.21.15.3:0/776628837 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6fb81f6900 con 0x7f6fb815e830 2024-05-08T21:24:08.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.175+0000 7f6fbf2cf700 1 -- 172.21.15.3:0/776628837 --> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6fb81f6fc0 con 0x7f6f9c06ac90 2024-05-08T21:24:08.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.177+0000 7f6fadffb700 1 -- 172.21.15.3:0/776628837 <== mgr.4111 v2:172.21.15.186:6832/46682 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51014 (secure 0 0 0) 0x7f6fb81f6fc0 con 0x7f6f9c06ac90 2024-05-08T21:24:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:24:08.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.179+0000 7f6fbf2cf700 1 -- 172.21.15.3:0/776628837 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6f9c06ac90 msgr2=0x7f6f9c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:08.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.179+0000 7f6fbf2cf700 1 --2- 172.21.15.3:0/776628837 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6f9c06ac90 0x7f6f9c06d150 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4006b20 tx=0x7f6fb4008040 comp rx=0 tx=0).stop 2024-05-08T21:24:08.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.180+0000 7f6fbf2cf700 1 -- 172.21.15.3:0/776628837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fb815e830 msgr2=0x7f6fb806e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:08.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.180+0000 7f6fbf2cf700 1 --2- 172.21.15.3:0/776628837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fb815e830 0x7f6fb806e780 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80676a0 tx=0x7f6fb0004ad0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.180+0000 7f6fbf2cf700 1 -- 172.21.15.3:0/776628837 shutdown_connections 2024-05-08T21:24:08.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.180+0000 7f6fbf2cf700 1 --2- 172.21.15.3:0/776628837 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fb806b0e0 0x7f6fb8148a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.180+0000 7f6fbf2cf700 1 --2- 172.21.15.3:0/776628837 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f6f9c06ac90 0x7f6f9c06d150 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.180+0000 7f6fbf2cf700 1 --2- 172.21.15.3:0/776628837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fb815e830 0x7f6fb806e780 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.180+0000 7f6fbf2cf700 1 --2- 172.21.15.3:0/776628837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fb80d0f20 0x7f6fb81f72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.180+0000 7f6fbf2cf700 1 -- 172.21.15.3:0/776628837 >> 172.21.15.3:0/776628837 conn(0x7f6fb80b5810 msgr2=0x7f6fb80b4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:24:08.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.180+0000 7f6fbf2cf700 1 -- 172.21.15.3:0/776628837 shutdown_connections 2024-05-08T21:24:08.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.180+0000 7f6fbf2cf700 1 -- 172.21.15.3:0/776628837 wait complete. 2024-05-08T21:24:08.183 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-05-08T21:24:08.195 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-05-08T21:24:06.562762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":221584,"kb_used_data":5416,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753136,"statfs":{"total":773094113280,"available":772867211264,"internally_reserved":0,"allocated":5545984,"data_stored":2999128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002519"},"pg_stats":[{"pgid":"1.0","version":"46'119","reported_seq":202,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-08T21:23:47.734195+0000","last_change":"2024-05-08T21:23:47.734353+0000","last_active":"2024-05-08T21:23:47.734195+0000","last_peered":"2024-05-08T21:23:47.734195+0000","last_clean":"2024-05-08T21:23:47.734195+0000","last_became_active":"2024-05-08T21:21:42.332033+0000","last_became_peered":"2024-05-08T21:21:42.332033+0000","last_unstale":"2024-05-08T21:23:47.734195+0000","last_undegraded":"2024-05-08T21:23:47.734195+0000","last_fullsized":"2024-05-08T21:23:47.734195+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'119","last_scrub_stamp":"2024-05-08T21:23:47.734170+0000","last_deep_scrub":"46'119","last_deep_scrub_stamp":"2024-05-08T21:23:47.734170+0000","last_clean_scrub_stamp":"2024-05-08T21:23:47.734170+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:03:22.259461+0000","scrub_duration":0.0050278179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343712,"statfs":{"total":96636764160,"available":96607961088,"internally_reserved":0,"allocated":1138688,"data_stored":817667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.52100000000000002,"15min":0.51500000000000001},"min":{"1min":0.32800000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.48699999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.46300000000000002}]},{"osd":1,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.39500000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55200000000000005,"15min":0.55100000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.55900000000000005}]},{"osd":2,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.58399999999999996,"15min":0.57499999999999996},"min":{"1min":0.51000000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.63600000000000001}]},{"osd":3,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.38300000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.45200000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.58199999999999996}]},{"osd":4,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.52600000000000002,"15min":0.53300000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.628},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42399999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.55200000000000005}]},{"osd":5,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.502,"15min":0.503},"min":{"1min":0.36299999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.748,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.51900000000000002,"15min":0.52300000000000002},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71299999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.503}]},{"osd":6,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57599999999999996,"15min":0.57799999999999996},"min":{"1min":0.38700000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78200000000000003,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.52000000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27552,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":544768,"data_stored":227299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.47699999999999998,"15min":0.46200000000000002},"min":{"1min":0.38,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.45700000000000002}]},{"osd":1,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.51100000000000001,"15min":0.505},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.438},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53500000000000003,"15min":0.53000000000000003},"min":{"1min":0.39200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.60499999999999998}]},{"osd":2,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.35799999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.435,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.54000000000000004,"15min":0.53500000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.46000000000000002,"5min":0.376,"15min":0.376},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.55200000000000005}]},{"osd":4,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.496,"15min":0.49199999999999999},"min":{"1min":0.35199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.498},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.48999999999999999,"15min":0.48399999999999999},"min":{"1min":0.39400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.376}]},{"osd":5,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.47999999999999998,"15min":0.47199999999999998},"min":{"1min":0.41199999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.40400000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.54100000000000004}]},{"osd":7,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.76700000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.443,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.51300000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27552,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":544768,"data_stored":227299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.375,"15min":0.375},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.56999999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.42599999999999999,"15min":0.42499999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.85399999999999998,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.48399999999999999}]},{"osd":2,"last update":"Wed May 8 21:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.39400000000000002,"15min":0.39500000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.621,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.504},{"interface":"front","average":{"1min":0.44,"5min":0.46600000000000003,"15min":0.46999999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.622,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.45500000000000002}]},{"osd":3,"last update":"Wed May 8 21:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.46800000000000003,"15min":0.47299999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.59199999999999997,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.435,"15min":0.42899999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.53400000000000003}]},{"osd":4,"last update":"Wed May 8 21:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.64200000000000002,"15min":0.64400000000000002},"min":{"1min":0.46200000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.873}]},{"osd":5,"last update":"Wed May 8 21:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.82799999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.63900000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.78200000000000003}]},{"osd":6,"last update":"Wed May 8 21:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.66800000000000004,"15min":0.67200000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.88600000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.71799999999999997}]},{"osd":7,"last update":"Wed May 8 21:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.70299999999999996,"15min":0.70399999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65100000000000002,"15min":0.64500000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.70499999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343712,"statfs":{"total":96636764160,"available":96607961088,"internally_reserved":0,"allocated":1138688,"data_stored":817667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.49099999999999999,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.378,"15min":0.376},"min":{"1min":0.29599999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.40799999999999997}]},{"osd":2,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.39500000000000002,"15min":0.39300000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.5},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.40300000000000002,"15min":0.40100000000000002},"min":{"1min":0.313,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.59899999999999998,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.53800000000000003}]},{"osd":3,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.40500000000000003,"15min":0.41099999999999998},"min":{"1min":0.26100000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.57099999999999995,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68400000000000005,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.55500000000000005}]},{"osd":4,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.49199999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58699999999999997,"15min":0.58499999999999996},"min":{"1min":0.47399999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.60799999999999998}]},{"osd":5,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60499999999999998,"15min":0.60699999999999998},"min":{"1min":0.46600000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.64100000000000001,"15min":0.64200000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.82799999999999996,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.65700000000000003}]},{"osd":6,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66800000000000004,"15min":0.67200000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92000000000000004,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65100000000000002,"15min":0.64900000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.68700000000000006}]},{"osd":7,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.5,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66200000000000003,"15min":0.66400000000000003},"min":{"1min":0.52000000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.89700000000000002,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.63200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27560,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344280,"statfs":{"total":96636764160,"available":96608542720,"internally_reserved":0,"allocated":544768,"data_stored":227299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.41299999999999998,"15min":0.41599999999999998},"min":{"1min":0.26900000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.55700000000000005,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.439,"5min":0.49099999999999999,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63,"5min":1.083,"15min":1.083},"last":0.50600000000000001}]},{"osd":1,"last update":"Wed May 8 21:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.434,"15min":0.438},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.61899999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.442,"15min":0.45000000000000001},"min":{"1min":0.30299999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.58299999999999996,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.44500000000000001}]},{"osd":3,"last update":"Wed May 8 21:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.433,"15min":0.439},"min":{"1min":0.29999999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57999999999999996,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.5,"15min":0.50600000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.61499999999999999,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.58199999999999996}]},{"osd":4,"last update":"Wed May 8 21:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.65100000000000002,"15min":0.65700000000000003},"min":{"1min":0.46700000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79700000000000004,"5min":1.141,"15min":1.141},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.628},"min":{"1min":0.42899999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84299999999999997,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.78500000000000003}]},{"osd":5,"last update":"Wed May 8 21:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.68600000000000005,"15min":0.68799999999999994},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.89200000000000002,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.70499999999999996,"15min":0.71599999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.82399999999999995,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.76000000000000001}]},{"osd":6,"last update":"Wed May 8 21:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.91700000000000004,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69699999999999995,"15min":0.70399999999999996},"min":{"1min":0.53200000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.81000000000000005,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.61699999999999999}]},{"osd":7,"last update":"Wed May 8 21:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.72299999999999998,"15min":0.73199999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.85899999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.71799999999999997,"15min":0.72299999999999998},"min":{"1min":0.54400000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.94899999999999995,"5min":1.177,"15min":1.177},"last":0.74099999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27556,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344284,"statfs":{"total":96636764160,"available":96608546816,"internally_reserved":0,"allocated":544768,"data_stored":227299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.42899999999999999,"15min":0.435},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.59799999999999998,"5min":1.492,"15min":1.492},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.50600000000000001,"15min":0.52100000000000002},"min":{"1min":0.30099999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.68700000000000006,"5min":2.7919999999999998,"15min":2.7919999999999998},"last":0.36699999999999999}]},{"osd":1,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.46000000000000002,"15min":0.434},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.2370000000000001,"5min":4.2370000000000001,"15min":4.2370000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.436,"15min":0.441},"min":{"1min":0.31,"5min":0.308,"15min":0.308},"max":{"1min":0.54500000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.40999999999999998}]},{"osd":2,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.53400000000000003,"15min":0.55200000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.65000000000000002,"5min":2.9369999999999998,"15min":2.9369999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.496,"15min":0.46999999999999997},"min":{"1min":0.35299999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.2969999999999997,"5min":4.2969999999999997,"15min":4.2969999999999997},"last":0.42799999999999999}]},{"osd":4,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.623,"15min":0.622},"min":{"1min":0.48299999999999998,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.90300000000000002,"5min":1.173,"15min":1.173},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.45000000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.7490000000000001,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":0.66700000000000004}]},{"osd":5,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.76000000000000001,"15min":0.78600000000000003},"min":{"1min":0.47399999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.79900000000000004,"5min":2.9060000000000001,"15min":2.9060000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.874,"5min":0.70399999999999996,"15min":0.67600000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":5.1699999999999999,"5min":5.1699999999999999,"15min":5.1699999999999999},"last":0.64800000000000002}]},{"osd":6,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.68000000000000005,"15min":0.68400000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.76500000000000001,"15min":0.78900000000000003},"min":{"1min":0.50600000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.81799999999999995,"5min":2.8730000000000002,"15min":2.8730000000000002},"last":0.73999999999999999}]},{"osd":7,"last update":"Wed May 8 21:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.74299999999999999,"15min":0.71999999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":5.2030000000000003,"5min":5.2030000000000003,"15min":5.2030000000000003},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.70099999999999996,"15min":0.70699999999999996},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.1399999999999999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.78600000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27556,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344284,"statfs":{"total":96636764160,"available":96608546816,"internally_reserved":0,"allocated":544768,"data_stored":227299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.52700000000000002,"15min":0.52000000000000002},"min":{"1min":0.40400000000000003,"5min":0.312,"15min":0.312},"max":{"1min":0.748,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.39700000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.68500000000000005,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.73299999999999998}]},{"osd":1,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35199999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76000000000000001,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52300000000000002,"15min":0.51700000000000002},"min":{"1min":0.38200000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.64900000000000002}]},{"osd":2,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58999999999999997,"15min":0.58899999999999997},"min":{"1min":0.47399999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.747,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.71999999999999997,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.622}]},{"osd":3,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.36699999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.65800000000000003}]},{"osd":5,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81299999999999994,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.51000000000000001,"15min":0.505},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.74299999999999999,"5min":0.754,"15min":0.754},"last":0.63600000000000001}]},{"osd":6,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.38600000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.62,"15min":0.622},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85399999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.67700000000000005}]},{"osd":7,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79600000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.60699999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27552,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":544768,"data_stored":227299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.74399999999999999,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.83299999999999996,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.66400000000000003}]},{"osd":1,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56000000000000005,"15min":0.56200000000000006},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.78700000000000003,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.53000000000000003,"15min":0.52400000000000002},"min":{"1min":0.41599999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94599999999999995,"5min":1.147,"15min":1.147},"last":0.55600000000000005}]},{"osd":2,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.47999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.90100000000000002,"5min":1.282,"15min":1.282},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.96699999999999997,"5min":1.202,"15min":1.202},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.441,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.71599999999999997,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.69999999999999996}]},{"osd":4,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.502,"15min":0.499},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73199999999999998,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.372,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68700000000000006,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.50600000000000001}]},{"osd":6,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53500000000000003,"15min":0.53100000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.75600000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.63},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39800000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.77700000000000002,"5min":1.119,"15min":1.119},"last":0.54500000000000004}]},{"osd":7,"last update":"Wed May 8 21:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53200000000000003,"15min":0.53000000000000003},"min":{"1min":0.31900000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.79900000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.39300000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.85699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T21:24:08.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-05-08T21:24:08.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.330+0000 7f8572c72700 1 Processor -- start 2024-05-08T21:24:08.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.330+0000 7f8572c72700 1 -- start start 2024-05-08T21:24:08.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.331+0000 7f8572c72700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f856c0d1910 0x7f856c0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:08.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.331+0000 7f8572c72700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856c0cf560 con 0x7f856c15b4e0 2024-05-08T21:24:08.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.331+0000 7f8572c72700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856c0ccff0 con 0x7f856c15b8c0 2024-05-08T21:24:08.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.331+0000 7f8572c72700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856c0cefe0 con 0x7f856c0d1910 2024-05-08T21:24:08.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f8570a0e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f856c15b4e0 0x7f856c0b1200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53110/0 (socket says 172.21.15.3:53110) 2024-05-08T21:24:08.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f8570a0e700 1 -- 172.21.15.3:0/2168952539 learned_addr learned my addr 172.21.15.3:0/2168952539 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:24:08.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f856b7fe700 1 -- 172.21.15.3:0/2168952539 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950243279 0 0) 0x7f856c0cf560 con 0x7f856c15b4e0 2024-05-08T21:24:08.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f857120f700 1 --2- 172.21.15.3:0/2168952539 >> v2:172.21.15.186:3300/0 conn(0x7f856c0d1910 0x7f856c0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:08.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f856b7fe700 1 -- 172.21.15.3:0/2168952539 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85540035b0 con 0x7f856c15b4e0 2024-05-08T21:24:08.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f856b7fe700 1 -- 172.21.15.3:0/2168952539 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2479787885 0 0) 0x7f85540035b0 con 0x7f856c15b4e0 2024-05-08T21:24:08.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f856b7fe700 1 -- 172.21.15.3:0/2168952539 >> v2:172.21.15.186:3300/0 conn(0x7f856c0d1910 msgr2=0x7f856c0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:24:08.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f856b7fe700 1 --2- 172.21.15.3:0/2168952539 >> v2:172.21.15.186:3300/0 conn(0x7f856c0d1910 0x7f856c0cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f856b7fe700 1 -- 172.21.15.3:0/2168952539 >> v1:172.21.15.186:6789/0 conn(0x7f856c15b8c0 legacy=0x7f856c0d1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:08.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f856b7fe700 1 -- 172.21.15.3:0/2168952539 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c0cd2d0 con 0x7f856c15b4e0 2024-05-08T21:24:08.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f856b7fe700 1 -- 172.21.15.3:0/2168952539 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1034289180 0 0) 0x7f855c001670 con 0x7f856c15b4e0 2024-05-08T21:24:08.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f856b7fe700 1 -- 172.21.15.3:0/2168952539 >> v1:172.21.15.3:6789/0 conn(0x7f856c15b4e0 legacy=0x7f856c0b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:08.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f856b7fe700 1 --2- 172.21.15.3:0/2168952539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8554003a00 0x7f8554003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:08.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f856b7fe700 1 --2- 172.21.15.3:0/2168952539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8554004850 0x7f8554004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:08.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f856b7fe700 1 --2- 172.21.15.3:0/2168952539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8554005210 0x7f855400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:08.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f856b7fe700 1 -- 172.21.15.3:0/2168952539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0cefe0 con 0x7f8554004850 2024-05-08T21:24:08.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f856b7fe700 1 -- 172.21.15.3:0/2168952539 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855400c0b0 con 0x7f8554005210 2024-05-08T21:24:08.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f856b7fe700 1 -- 172.21.15.3:0/2168952539 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f855400c270 con 0x7f8554003a00 2024-05-08T21:24:08.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.332+0000 7f856bfff700 1 --2- 172.21.15.3:0/2168952539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8554004850 0x7f8554004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:08.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.333+0000 7f8570a0e700 1 --2- 172.21.15.3:0/2168952539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8554003a00 0x7f8554003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:08.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.333+0000 7f856bfff700 1 -- 172.21.15.3:0/2168952539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8554003a00 msgr2=0x7f8554003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:24:08.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.333+0000 7f856bfff700 1 --2- 172.21.15.3:0/2168952539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8554003a00 0x7f8554003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.333+0000 7f856bfff700 1 -- 172.21.15.3:0/2168952539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8554005210 msgr2=0x7f855400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:24:08.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.333+0000 7f856bfff700 1 --2- 172.21.15.3:0/2168952539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8554005210 0x7f855400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.333+0000 7f856bfff700 1 -- 172.21.15.3:0/2168952539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85540035b0 con 0x7f8554004850 2024-05-08T21:24:08.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.333+0000 7f857120f700 1 -- 172.21.15.3:0/2168952539 reap_dead start 2024-05-08T21:24:08.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.333+0000 7f856bfff700 1 --2- 172.21.15.3:0/2168952539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8554004850 0x7f8554004cd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f85600017c0 tx=0x7f8554005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f42091a33c14e01 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:24:08.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.333+0000 7f856b7fe700 1 -- 172.21.15.3:0/2168952539 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85600052d0 con 0x7f8554004850 2024-05-08T21:24:08.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.333+0000 7f856b7fe700 1 -- 172.21.15.3:0/2168952539 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8560005430 con 0x7f8554004850 2024-05-08T21:24:08.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.333+0000 7f856b7fe700 1 -- 172.21.15.3:0/2168952539 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85600038c0 con 0x7f8554004850 2024-05-08T21:24:08.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.334+0000 7f8572c72700 1 -- 172.21.15.3:0/2168952539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8554004850 msgr2=0x7f8554004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:08.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.334+0000 7f8572c72700 1 --2- 172.21.15.3:0/2168952539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8554004850 0x7f8554004cd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f85600017c0 tx=0x7f8554005880 comp rx=0 tx=0).stop 2024-05-08T21:24:08.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.334+0000 7f8572c72700 1 -- 172.21.15.3:0/2168952539 shutdown_connections 2024-05-08T21:24:08.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.334+0000 7f8572c72700 1 --2- 172.21.15.3:0/2168952539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8554004850 0x7f8554004cd0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.334+0000 7f8572c72700 1 -- 172.21.15.3:0/2168952539 >> 172.21.15.3:0/2168952539 conn(0x7f856c08afe0 msgr2=0x7f856c08b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:24:08.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.334+0000 7f8572c72700 1 -- 172.21.15.3:0/2168952539 shutdown_connections 2024-05-08T21:24:08.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.334+0000 7f8572c72700 1 -- 172.21.15.3:0/2168952539 wait complete. 2024-05-08T21:24:08.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.335+0000 7f8572c72700 1 Processor -- start 2024-05-08T21:24:08.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.335+0000 7f8572c72700 1 -- start start 2024-05-08T21:24:08.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.336+0000 7f8572c72700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8554004850 0x7f856c14fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:08.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.336+0000 7f8572c72700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f856c0c7870 0x7f856c0c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:08.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.336+0000 7f8572c72700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f856c0c5ce0 0x7f856c06cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:08.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.336+0000 7f8572c72700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0af5e0 con 0x7f856c0c7870 2024-05-08T21:24:08.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.336+0000 7f8572c72700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0ca990 con 0x7f8554004850 2024-05-08T21:24:08.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.336+0000 7f8572c72700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0aec40 con 0x7f856c0c5ce0 2024-05-08T21:24:08.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.336+0000 7f856bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f856c0c7870 0x7f856c0c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:08.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.336+0000 7f856bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f856c0c7870 0x7f856c0c57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57876/0 (socket says 172.21.15.3:57876) 2024-05-08T21:24:08.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.336+0000 7f856bfff700 1 -- 172.21.15.3:0/3319193099 learned_addr learned my addr 172.21.15.3:0/3319193099 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:24:08.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.336+0000 7f8570a0e700 1 --2- 172.21.15.3:0/3319193099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8554004850 0x7f856c14fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:08.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.337+0000 7f857120f700 1 --2- 172.21.15.3:0/3319193099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f856c0c5ce0 0x7f856c06cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:08.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.337+0000 7f856bfff700 1 -- 172.21.15.3:0/3319193099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f856c0c5ce0 msgr2=0x7f856c06cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:08.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.337+0000 7f856bfff700 1 --2- 172.21.15.3:0/3319193099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f856c0c5ce0 0x7f856c06cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.337+0000 7f856bfff700 1 -- 172.21.15.3:0/3319193099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8554004850 msgr2=0x7f856c14fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:08.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.337+0000 7f856bfff700 1 --2- 172.21.15.3:0/3319193099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8554004850 0x7f856c14fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.337+0000 7f856bfff700 1 -- 172.21.15.3:0/3319193099 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c0aef20 con 0x7f856c0c7870 2024-05-08T21:24:08.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.337+0000 7f8570a0e700 1 --2- 172.21.15.3:0/3319193099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8554004850 0x7f856c14fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-08T21:24:08.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.337+0000 7f856bfff700 1 --2- 172.21.15.3:0/3319193099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f856c0c7870 0x7f856c0c57a0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f8560005040 tx=0x7f8560001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:24:08.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.337+0000 7f85697fa700 1 -- 172.21.15.3:0/3319193099 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8560001290 con 0x7f856c0c7870 2024-05-08T21:24:08.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.337+0000 7f85697fa700 1 -- 172.21.15.3:0/3319193099 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8560007040 con 0x7f856c0c7870 2024-05-08T21:24:08.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.337+0000 7f85697fa700 1 -- 172.21.15.3:0/3319193099 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85600073b0 con 0x7f856c0c7870 2024-05-08T21:24:08.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.337+0000 7f8572c72700 1 -- 172.21.15.3:0/3319193099 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856c0af200 con 0x7f856c0c7870 2024-05-08T21:24:08.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.337+0000 7f8572c72700 1 -- 172.21.15.3:0/3319193099 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856c06d020 con 0x7f856c0c7870 2024-05-08T21:24:08.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.338+0000 7f8572c72700 1 -- 172.21.15.3:0/3319193099 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856c0af720 con 0x7f856c0c7870 2024-05-08T21:24:08.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.338+0000 7f85697fa700 1 -- 172.21.15.3:0/3319193099 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8560007510 con 0x7f856c0c7870 2024-05-08T21:24:08.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.339+0000 7f85697fa700 1 --2- 172.21.15.3:0/3319193099 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f855806acc0 0x7f855806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:08.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.339+0000 7f85697fa700 1 -- 172.21.15.3:0/3319193099 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f8560089980 con 0x7f856c0c7870 2024-05-08T21:24:08.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.339+0000 7f8570a0e700 1 --2- 172.21.15.3:0/3319193099 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f855806acc0 0x7f855806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:08.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.340+0000 7f8570a0e700 1 --2- 172.21.15.3:0/3319193099 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f855806acc0 0x7f855806d180 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f855c007cf0 tx=0x7f855c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:24:08.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.344+0000 7f85697fa700 1 -- 172.21.15.3:0/3319193099 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f856c0af720 con 0x7f856c0c7870 2024-05-08T21:24:08.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.561+0000 7f8572c72700 1 -- 172.21.15.3:0/3319193099 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f856c0af720 con 0x7f856c0c7870 2024-05-08T21:24:08.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.564+0000 7f85697fa700 1 -- 172.21.15.3:0/3319193099 <== mon.0 v2:172.21.15.3:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f8560054fb0 con 0x7f856c0c7870 2024-05-08T21:24:08.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.565+0000 7f85697fa700 1 -- 172.21.15.3:0/3319193099 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v14)=0 v14) v1 ==== 123+0+0 (secure 0 0 0) 0x7f856c0af720 con 0x7f856c0c7870 2024-05-08T21:24:08.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.568+0000 7f8572c72700 1 -- 172.21.15.3:0/3319193099 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f855806acc0 msgr2=0x7f855806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:08.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.568+0000 7f8572c72700 1 --2- 172.21.15.3:0/3319193099 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f855806acc0 0x7f855806d180 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f855c007cf0 tx=0x7f855c009040 comp rx=0 tx=0).stop 2024-05-08T21:24:08.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.568+0000 7f8572c72700 1 -- 172.21.15.3:0/3319193099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f856c0c7870 msgr2=0x7f856c0c57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:08.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.568+0000 7f8572c72700 1 --2- 172.21.15.3:0/3319193099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f856c0c7870 0x7f856c0c57a0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f8560005040 tx=0x7f8560001120 comp rx=0 tx=0).stop 2024-05-08T21:24:08.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.568+0000 7f8572c72700 1 -- 172.21.15.3:0/3319193099 shutdown_connections 2024-05-08T21:24:08.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.568+0000 7f8572c72700 1 --2- 172.21.15.3:0/3319193099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f856c0c5ce0 0x7f856c06cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.568+0000 7f8572c72700 1 --2- 172.21.15.3:0/3319193099 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7f855806acc0 0x7f855806d180 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.568+0000 7f8572c72700 1 --2- 172.21.15.3:0/3319193099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8554004850 0x7f856c14fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.568+0000 7f8572c72700 1 --2- 172.21.15.3:0/3319193099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f856c0c7870 0x7f856c0c57a0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.568+0000 7f8572c72700 1 -- 172.21.15.3:0/3319193099 >> 172.21.15.3:0/3319193099 conn(0x7f856c08afe0 msgr2=0x7f856c06ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:24:08.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.568+0000 7f8572c72700 1 -- 172.21.15.3:0/3319193099 shutdown_connections 2024-05-08T21:24:08.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.568+0000 7f8572c72700 1 -- 172.21.15.3:0/3319193099 wait complete. 2024-05-08T21:24:08.584 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-08T21:24:08.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T21:24:08.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.719+0000 7fd18e1dc700 1 Processor -- start 2024-05-08T21:24:08.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.719+0000 7fd18e1dc700 1 -- start start 2024-05-08T21:24:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.721+0000 7fd18e1dc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd18813e5f0 0x7fd18815fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.721+0000 7fd18e1dc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd1880ce8d0 0x7fd18806a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.721+0000 7fd18e1dc700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1880cf900 con 0x7fd18806aea0 2024-05-08T21:24:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.721+0000 7fd18e1dc700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd18815baf0 con 0x7fd1880ce8d0 2024-05-08T21:24:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.721+0000 7fd18e1dc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18815bd70 con 0x7fd18813e5f0 2024-05-08T21:24:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.721+0000 7fd186ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd1880ce8d0 0x7fd18806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.721+0000 7fd187fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd18806aea0 0x7fd18815c200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53120/0 (socket says 172.21.15.3:53120) 2024-05-08T21:24:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.721+0000 7fd187fff700 1 -- 172.21.15.3:0/3955191983 learned_addr learned my addr 172.21.15.3:0/3955191983 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-05-08T21:24:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.721+0000 7fd1877fe700 1 --2- 172.21.15.3:0/3955191983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd18813e5f0 0x7fd18815fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.721+0000 7fd186ffd700 1 -- 172.21.15.3:0/3955191983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd18813e5f0 msgr2=0x7fd18815fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-08T21:24:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.721+0000 7fd186ffd700 1 --2- 172.21.15.3:0/3955191983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd18813e5f0 0x7fd18815fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.721+0000 7fd186ffd700 1 -- 172.21.15.3:0/3955191983 >> v1:172.21.15.3:6789/0 conn(0x7fd18806aea0 legacy=0x7fd18815c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.721+0000 7fd1867fc700 1 -- 172.21.15.3:0/3955191983 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162768687 0 0) 0x7fd1880cf900 con 0x7fd18806aea0 2024-05-08T21:24:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.721+0000 7fd186ffd700 1 -- 172.21.15.3:0/3955191983 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1880cf060 con 0x7fd1880ce8d0 2024-05-08T21:24:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.721+0000 7fd1877fe700 1 --2- 172.21.15.3:0/3955191983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd18813e5f0 0x7fd18815fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-08T21:24:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.721+0000 7fd186ffd700 1 --2- 172.21.15.3:0/3955191983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1880ce8d0 0x7fd18806a960 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009840 tx=0x7fd17c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d862985b007a09c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:24:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.722+0000 7fd1867fc700 1 -- 172.21.15.3:0/3955191983 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c011830 con 0x7fd1880ce8d0 2024-05-08T21:24:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.722+0000 7fd1867fc700 1 -- 172.21.15.3:0/3955191983 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd17c011990 con 0x7fd1880ce8d0 2024-05-08T21:24:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.722+0000 7fd1867fc700 1 -- 172.21.15.3:0/3955191983 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c011d40 con 0x7fd1880ce8d0 2024-05-08T21:24:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.722+0000 7fd18e1dc700 1 -- 172.21.15.3:0/3955191983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1880ce8d0 msgr2=0x7fd18806a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.722+0000 7fd18e1dc700 1 --2- 172.21.15.3:0/3955191983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1880ce8d0 0x7fd18806a960 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009840 tx=0x7fd17c009b50 comp rx=0 tx=0).stop 2024-05-08T21:24:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.722+0000 7fd18e1dc700 1 -- 172.21.15.3:0/3955191983 shutdown_connections 2024-05-08T21:24:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.722+0000 7fd18e1dc700 1 --2- 172.21.15.3:0/3955191983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd18813e5f0 0x7fd18815fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.722+0000 7fd18e1dc700 1 --2- 172.21.15.3:0/3955191983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1880ce8d0 0x7fd18806a960 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.722+0000 7fd18e1dc700 1 -- 172.21.15.3:0/3955191983 >> 172.21.15.3:0/3955191983 conn(0x7fd1880cfce0 msgr2=0x7fd1880d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:24:08.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.722+0000 7fd18e1dc700 1 -- 172.21.15.3:0/3955191983 shutdown_connections 2024-05-08T21:24:08.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.722+0000 7fd18e1dc700 1 -- 172.21.15.3:0/3955191983 wait complete. 2024-05-08T21:24:08.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.724+0000 7fd18e1dc700 1 Processor -- start 2024-05-08T21:24:08.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.724+0000 7fd18e1dc700 1 -- start start 2024-05-08T21:24:08.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.725+0000 7fd18e1dc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd18806aea0 0x7fd188141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:08.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.725+0000 7fd18e1dc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1880ce8d0 0x7fd188158b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:08.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.725+0000 7fd18e1dc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18813e5f0 0x7fd1881f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:08.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.725+0000 7fd18e1dc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1881f2080 con 0x7fd18813e5f0 2024-05-08T21:24:08.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.725+0000 7fd18e1dc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18813ec80 con 0x7fd1880ce8d0 2024-05-08T21:24:08.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.725+0000 7fd18e1dc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18813ef60 con 0x7fd18806aea0 2024-05-08T21:24:08.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.725+0000 7fd187fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18813e5f0 0x7fd1881f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.725+0000 7fd187fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18813e5f0 0x7fd1881f1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57906/0 (socket says 172.21.15.3:57906) 2024-05-08T21:24:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.725+0000 7fd187fff700 1 -- 172.21.15.3:0/2333456636 learned_addr learned my addr 172.21.15.3:0/2333456636 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-05-08T21:24:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.726+0000 7fd1877fe700 1 --2- 172.21.15.3:0/2333456636 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd18806aea0 0x7fd188141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.726+0000 7fd186ffd700 1 --2- 172.21.15.3:0/2333456636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1880ce8d0 0x7fd188158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.726+0000 7fd187fff700 1 -- 172.21.15.3:0/2333456636 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd18806aea0 msgr2=0x7fd188141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.726+0000 7fd187fff700 1 --2- 172.21.15.3:0/2333456636 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd18806aea0 0x7fd188141150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.726+0000 7fd187fff700 1 -- 172.21.15.3:0/2333456636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1880ce8d0 msgr2=0x7fd188158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.726+0000 7fd187fff700 1 --2- 172.21.15.3:0/2333456636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1880ce8d0 0x7fd188158b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.726+0000 7fd187fff700 1 -- 172.21.15.3:0/2333456636 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c0094a0 con 0x7fd18813e5f0 2024-05-08T21:24:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.726+0000 7fd187fff700 1 --2- 172.21.15.3:0/2333456636 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18813e5f0 0x7fd1881f1b40 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd180002230 tx=0x7fd18000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:24:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.726+0000 7fd16ffff700 1 -- 172.21.15.3:0/2333456636 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18000aac0 con 0x7fd18813e5f0 2024-05-08T21:24:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.726+0000 7fd16ffff700 1 -- 172.21.15.3:0/2333456636 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd18000ac40 con 0x7fd18813e5f0 2024-05-08T21:24:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.726+0000 7fd18e1dc700 1 -- 172.21.15.3:0/2333456636 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18813f240 con 0x7fd18813e5f0 2024-05-08T21:24:08.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.726+0000 7fd16ffff700 1 -- 172.21.15.3:0/2333456636 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1800070d0 con 0x7fd18813e5f0 2024-05-08T21:24:08.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.726+0000 7fd18e1dc700 1 -- 172.21.15.3:0/2333456636 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18813f800 con 0x7fd18813e5f0 2024-05-08T21:24:08.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.727+0000 7fd18e1dc700 1 -- 172.21.15.3:0/2333456636 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd18813f9f0 con 0x7fd18813e5f0 2024-05-08T21:24:08.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.728+0000 7fd16ffff700 1 -- 172.21.15.3:0/2333456636 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd180007230 con 0x7fd18813e5f0 2024-05-08T21:24:08.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.728+0000 7fd16ffff700 1 --2- 172.21.15.3:0/2333456636 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd17406ac90 0x7fd17406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-08T21:24:08.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.728+0000 7fd16ffff700 1 -- 172.21.15.3:0/2333456636 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd180088cb0 con 0x7fd18813e5f0 2024-05-08T21:24:08.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.729+0000 7fd1877fe700 1 --2- 172.21.15.3:0/2333456636 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd17406ac90 0x7fd17406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-08T21:24:08.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.729+0000 7fd1877fe700 1 --2- 172.21.15.3:0/2333456636 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd17406ac90 0x7fd17406d150 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd1881f4610 tx=0x7fd178008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-08T21:24:08.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.733+0000 7fd16ffff700 1 -- 172.21.15.3:0/2333456636 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd1800540b0 con 0x7fd18813e5f0 2024-05-08T21:24:09.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.999+0000 7fd18e1dc700 1 -- 172.21.15.3:0/2333456636 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd188068000 con 0x7fd18813e5f0 2024-05-08T21:24:09.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:08.999+0000 7fd16ffff700 1 -- 172.21.15.3:0/2333456636 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+3526 (secure 0 0 0) 0x7fd180018020 con 0x7fd18813e5f0 2024-05-08T21:24:09.002 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-05-08T21:24:09.002 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":30,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6833/2104267343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2104267343},{"type":"v1","addr":"172.21.15.3:6833","nonce":2104267343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/208321708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":208321708},{"type":"v1","addr":"172.21.15.3:6835","nonce":208321708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/1591386131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1591386131},{"type":"v1","addr":"172.21.15.186:6836","nonce":1591386131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5662,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3716101115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3716101115},{"type":"v1","addr":"172.21.15.186:6837","nonce":3716101115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[]} 2024-05-08T21:24:09.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:09.002+0000 7fd18e1dc700 1 -- 172.21.15.3:0/2333456636 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd17406ac90 msgr2=0x7fd17406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:09.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:09.002+0000 7fd18e1dc700 1 --2- 172.21.15.3:0/2333456636 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd17406ac90 0x7fd17406d150 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd1881f4610 tx=0x7fd178008040 comp rx=0 tx=0).stop 2024-05-08T21:24:09.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:09.002+0000 7fd18e1dc700 1 -- 172.21.15.3:0/2333456636 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18813e5f0 msgr2=0x7fd1881f1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-08T21:24:09.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:09.002+0000 7fd18e1dc700 1 --2- 172.21.15.3:0/2333456636 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18813e5f0 0x7fd1881f1b40 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd180002230 tx=0x7fd18000cf50 comp rx=0 tx=0).stop 2024-05-08T21:24:09.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:09.002+0000 7fd18e1dc700 1 -- 172.21.15.3:0/2333456636 shutdown_connections 2024-05-08T21:24:09.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:09.002+0000 7fd18e1dc700 1 --2- 172.21.15.3:0/2333456636 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd18806aea0 0x7fd188141150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:09.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:09.002+0000 7fd18e1dc700 1 --2- 172.21.15.3:0/2333456636 >> [v2:172.21.15.186:6832/46682,v1:172.21.15.186:6833/46682] conn(0x7fd17406ac90 0x7fd17406d150 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:09.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:09.002+0000 7fd18e1dc700 1 --2- 172.21.15.3:0/2333456636 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18813e5f0 0x7fd1881f1b40 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:09.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:09.002+0000 7fd18e1dc700 1 --2- 172.21.15.3:0/2333456636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1880ce8d0 0x7fd188158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-08T21:24:09.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:09.002+0000 7fd18e1dc700 1 -- 172.21.15.3:0/2333456636 >> 172.21.15.3:0/2333456636 conn(0x7fd1880cfce0 msgr2=0x7fd188159760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-08T21:24:09.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:09.002+0000 7fd18e1dc700 1 -- 172.21.15.3:0/2333456636 shutdown_connections 2024-05-08T21:24:09.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-05-08T21:24:09.002+0000 7fd18e1dc700 1 -- 172.21.15.3:0/2333456636 wait complete. 2024-05-08T21:24:09.006 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 30 2024-05-08T21:24:09.018 DEBUG:tasks.cephfs.filesystem:already dead... 2024-05-08T21:24:09.018 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-08T21:24:09.018 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-08T21:24:09.018 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.028 INFO:tasks.ceph.mds.a:Stopped 2024-05-08T21:24:09.028 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-08T21:24:09.028 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.079 INFO:tasks.ceph.mds.c:Stopped 2024-05-08T21:24:09.079 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-08T21:24:09.079 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.090 INFO:tasks.ceph.mds.b:Stopped 2024-05-08T21:24:09.090 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-08T21:24:09.090 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.141 INFO:tasks.ceph.mds.d:Stopped 2024-05-08T21:24:09.141 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-08T21:24:09.141 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-08T21:24:09.141 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.169 INFO:tasks.ceph.osd.0:Stopped 2024-05-08T21:24:09.170 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-08T21:24:09.170 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.241 INFO:tasks.ceph.osd.1:Stopped 2024-05-08T21:24:09.242 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-08T21:24:09.242 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.310 INFO:tasks.ceph.osd.2:Stopped 2024-05-08T21:24:09.311 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-08T21:24:09.311 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.379 INFO:tasks.ceph.osd.3:Stopped 2024-05-08T21:24:09.379 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-08T21:24:09.379 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.408 INFO:tasks.ceph.osd.4:Stopped 2024-05-08T21:24:09.408 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-08T21:24:09.408 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.477 INFO:tasks.ceph.osd.5:Stopped 2024-05-08T21:24:09.477 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-05-08T21:24:09.477 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.547 INFO:tasks.ceph.osd.6:Stopped 2024-05-08T21:24:09.548 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-05-08T21:24:09.548 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.619 INFO:tasks.ceph.osd.7:Stopped 2024-05-08T21:24:09.620 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-08T21:24:09.620 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-05-08T21:24:09.620 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.647 INFO:tasks.ceph.mgr.y:Stopped 2024-05-08T21:24:09.648 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-08T21:24:09.648 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.687 INFO:tasks.ceph.mgr.x:Stopped 2024-05-08T21:24:09.687 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-08T21:24:09.688 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-08T21:24:09.688 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.702 INFO:tasks.ceph.mon.a:Stopped 2024-05-08T21:24:09.702 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-08T21:24:09.702 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.740 INFO:tasks.ceph.mon.b:Stopped 2024-05-08T21:24:09.740 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-08T21:24:09.740 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T21:24:09.792 INFO:tasks.ceph.mon.c:Stopped 2024-05-08T21:24:09.792 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-08T21:24:09.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'bad backtrace' | egrep -v 'object missing on disk' | egrep -v 'error reading table object' | egrep -v 'error reading sessionmap' | egrep -v 'unmatched fragstat' | egrep -v 'unmatched rstat' | egrep -v 'was unreadable, recreating it now' | egrep -v 'Scrub error on inode' | egrep -v 'Metadata damage detected' | egrep -v MDS_FAILED | egrep -v MDS_DAMAGE | head -n 1 2024-05-08T21:24:09.843 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi003.front.sepia.ceph.com 2024-05-08T21:24:09.843 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-08T21:24:10.010 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi003.front.sepia.ceph.com 2024-05-08T21:24:10.010 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-08T21:24:10.113 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi003.front.sepia.ceph.com 2024-05-08T21:24:10.113 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-08T21:24:10.220 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi003.front.sepia.ceph.com 2024-05-08T21:24:10.220 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-08T21:24:10.307 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi186.front.sepia.ceph.com 2024-05-08T21:24:10.307 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-05-08T21:24:10.471 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi186.front.sepia.ceph.com 2024-05-08T21:24:10.471 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-05-08T21:24:10.569 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi186.front.sepia.ceph.com 2024-05-08T21:24:10.569 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-05-08T21:24:10.632 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi186.front.sepia.ceph.com 2024-05-08T21:24:10.632 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-05-08T21:24:10.695 INFO:tasks.ceph:Archiving mon data... 2024-05-08T21:24:10.696 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7698047/data/mon.a.tgz 2024-05-08T21:24:10.697 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-05-08T21:24:10.718 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.q1KOkDIubi 2024-05-08T21:24:10.719 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.q1KOkDIubi 2024-05-08T21:24:10.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.q1KOkDIubi 2024-05-08T21:24:10.992 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.q1KOkDIubi is 312KB 2024-05-08T21:24:11.065 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.q1KOkDIubi 2024-05-08T21:24:11.087 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7698047/data/mon.b.tgz 2024-05-08T21:24:11.087 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-05-08T21:24:11.106 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.Rh7mvVm28q 2024-05-08T21:24:11.106 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Rh7mvVm28q 2024-05-08T21:24:11.263 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.Rh7mvVm28q 2024-05-08T21:24:11.350 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.Rh7mvVm28q is 415KB 2024-05-08T21:24:11.446 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.Rh7mvVm28q 2024-05-08T21:24:11.463 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7698047/data/mon.c.tgz 2024-05-08T21:24:11.463 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-05-08T21:24:11.519 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.GpdoCeXTa1 2024-05-08T21:24:11.519 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.GpdoCeXTa1 2024-05-08T21:24:11.659 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.GpdoCeXTa1 2024-05-08T21:24:11.746 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.GpdoCeXTa1 is 415KB 2024-05-08T21:24:11.843 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.GpdoCeXTa1 2024-05-08T21:24:11.859 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-08T21:24:11.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-08T21:24:11.862 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-08T21:24:11.926 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-05-08T21:24:11.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-05-08T21:24:11.938 DEBUG:teuthology.orchestra.run.smithi186:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-05-08T21:24:11.971 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-05-08T21:24:11.993 INFO:teuthology.orchestra.run.smithi186.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-05-08T21:24:11.999 INFO:tasks.ceph:Archiving crash dumps... 2024-05-08T21:24:12.001 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7698047/remote/smithi003/crash 2024-05-08T21:24:12.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-08T21:24:12.053 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7698047/remote/smithi186/crash 2024-05-08T21:24:12.054 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-08T21:24:12.080 INFO:tasks.ceph:Shutting down logrotate 2024-05-08T21:24:12.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-08T21:24:12.098 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-08T21:24:12.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-05-08T21:24:12.207 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-05-08T21:24:12.234 INFO:tasks.ceph:Compressing logs... 2024-05-08T21:24:12.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-08T21:24:12.250 DEBUG:teuthology.orchestra.run.smithi186:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-08T21:24:13.281 INFO:tasks.ceph:Archiving logs... 2024-05-08T21:24:13.281 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7698047/remote/smithi003/log 2024-05-08T21:24:13.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-08T21:24:13.624 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7698047/remote/smithi186/log 2024-05-08T21:24:13.625 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-08T21:24:13.773 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-08T21:24:13.798 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-08T21:24:13.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-08T21:24:13.801 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-08T21:24:13.875 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-08T21:24:13.884 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-08T21:24:13.884 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-08T21:24:13.888 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-08T21:24:13.903 INFO:teuthology.orchestra.run.smithi186.stderr:bash: ntpq: command not found 2024-05-08T21:24:13.905 INFO:teuthology.orchestra.run.smithi003.stderr:bash: ntpq: command not found 2024-05-08T21:24:13.907 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-08T21:24:13.907 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-05-08T21:24:13.907 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv01.front.sepia.ceph.com 3 6 377 61 +1673us[+1674us] +/- 58ms 2024-05-08T21:24:13.907 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 61 -1632us[-1631us] +/- 64ms 2024-05-08T21:24:13.908 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 -252us[ -252us] +/- 83ms 2024-05-08T21:24:13.908 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T21:24:13.911 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-08T21:24:13.911 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-05-08T21:24:13.911 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv01.front.sepia.ceph.com 3 6 377 61 +1652us[+1647us] +/- 58ms 2024-05-08T21:24:13.911 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 60 -1636us[-1636us] +/- 64ms 2024-05-08T21:24:13.911 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 -274us[ -274us] +/- 83ms 2024-05-08T21:24:13.911 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T21:24:13.912 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-08T21:24:13.922 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-08T21:24:13.923 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-08T21:24:13.932 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-08T21:24:14.037 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-08T21:24:14.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-08T21:24:14.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T21:24:14.140 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-08T21:24:14.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T21:24:14.170 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-08T21:24:14.181 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-08T21:24:14.190 INFO:teuthology.task.internal:Duration was 1041.993421 seconds 2024-05-08T21:24:14.190 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-08T21:24:14.207 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-08T21:24:14.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-08T21:24:14.210 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-08T21:24:14.257 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-08T21:24:14.268 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-08T21:24:14.435 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-08T21:24:14.435 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2024-05-08T21:24:14.436 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-08T21:24:14.465 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-05-08T21:24:14.465 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-08T21:24:14.487 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-08T21:24:14.487 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-08T21:24:14.507 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-08T21:24:14.571 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-08T21:24:14.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-08T21:24:14.574 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-08T21:24:14.706 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-08T21:24:14.716 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-08T21:24:14.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-08T21:24:14.748 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-08T21:24:14.776 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-08T21:24:14.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-08T21:24:14.792 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-08T21:24:14.823 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2024-05-08T21:24:14.842 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-05-08T21:24:14.856 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-08T21:24:14.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T21:24:14.903 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-08T21:24:14.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T21:24:14.919 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-08T21:24:14.929 INFO:teuthology.task.internal:Transferring archived files... 2024-05-08T21:24:14.930 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7698047/remote/smithi003 2024-05-08T21:24:14.930 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-08T21:24:14.990 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7698047/remote/smithi186 2024-05-08T21:24:14.990 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-08T21:24:15.028 INFO:teuthology.task.internal:Removing archive directory... 2024-05-08T21:24:15.028 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-08T21:24:15.031 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-08T21:24:15.079 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-08T21:24:15.090 INFO:teuthology.task.internal:Not uploading archives. 2024-05-08T21:24:15.090 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-08T21:24:15.099 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-08T21:24:15.099 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-08T21:24:15.102 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-08T21:24:15.121 INFO:teuthology.orchestra.run.smithi003.stdout: 262175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 8 21:24 /home/ubuntu/cephtest 2024-05-08T21:24:15.135 INFO:teuthology.orchestra.run.smithi186.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 8 21:24 /home/ubuntu/cephtest 2024-05-08T21:24:15.137 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-08T21:24:15.146 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-08T21:24:15.175 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} duration: 1041.9934208393097 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-05-08T21:24:15.176 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-08T21:24:15.263 INFO:teuthology.run:pass